Recently @jrb posted on Discord link to Xgboost article about Vector-Leaf model and I thought it was worth trying. What is Vector-Leaf model? So with Xgboost 3.4.1 you have an option to have one tree with multiple outputs. You can train several targets at once and each tree is shared across targets, not one target one tree.
I did a quick experiment:
- Ender60 vs
- Tyler60 vs
- ensemble(Ender60, Tyler60) 50/50 vs
- Vector-Leaf(Ender60, Tyler60).
I picked Tyler60 because it is the least correlated with Ender60.
Setup. v5.3, medium features (780) training eras 1-574. Embargo 575-586. Validation 587-1221. 500 rounds, lr 0.05, depth 5, seed 42. No tuning. XGBoost 3.4.1, CPU.
Results are mean per era
| Model | Trees | Split nodes | CORR60 | BMC60 |
|---|---|---|---|---|
| Scalar Ender | 500 | 15,500 | 0.028379 | 0.001126 |
| Scalar Tyler | 500 | 15,500 | 0.022761 | -0.001367 |
| Ender + Tyler 50/50 rank blend | 1,000 | 31,000 | 0.028103 | -0.000039 |
| Vector Ender + Tyler | 500 | 15,500 | 0.028705 | 0.001142 |
Note the BMC. The 50/50 blend keeps the CORR but loses all the BMC. The Vector-Leaf model uses the same two targets and does not. Mixing two finished predictions is not the same as letting two targets pick the splits. The Vector-Leaf model has the same 500 trees and 15,500 split nodes as the single Ender model. The blend has 31,000 and scores worst.
I think Vector-Leaf Xgboost is worth further exploring with more targets and better tuning. In Neural Networks it is easy to add multi head output, and now we can use it in GBM models too! Here is full writeup: XGBoost Vector Leaf: Multi-Output Regression Explained