Very interesting post, thanks for sharing! This got my attention and I started investigating vector leaf myself, so here is what I found.
My setup
- v5.3 data with a subset of 500 top features. Fewer features means much faster iterations, which I needed for the amount of runs.
- Walk-forward: train from era 200 up to a cutoff, 12-era gap, predict the next 52 eras, retrain and repeat. First window 732–1010 (279 eras), then 1016–1227 (212 eras) as a later check, the last 56 of those (1172–1227) being my final holdout.
- A standard XGBoost 3.4.1 setting, no tuning:
multi_strategy="multi_output_tree", lr 0.001, 2000 trees, depth 6, min_child_weight 500, subsample 0.8, colsample 0.7, lambda 10, one seed.
- Every vector head compared against a scalar model of the same target: same rows, same params, same seed.
- Everything scored on target_ender_60: CORR, and BMC against the v5.3 lgbm ender60 benchmark.
I trained on GPU because it was way faster: one 40-target model on 2.7M rows took 34 min on an A100 vs 4.4 hours on my CPU (7.8x). CPU and GPU predictions correlate 0.993, close but not identical, so I never mixed backends inside a comparison.
Then I went all in with the heads:
- all the 60D targets (19) in one model, on 30% of the rows of each era: about 9 min per fold on an A100
- the whole 40 targets (20D + 60D) on all rows, which were slow: 34 to 63 min per fold on an A100
Ender head vs scalar Ender (60D model)
| Eras |
vector CORR |
scalar CORR |
vector BMC |
scalar BMC |
| 732–1010 |
0.0373 |
0.0363 |
0.0062 |
0.0056 |
| 1016–1227 |
0.0236 |
0.0238 |
0.0036 |
0.0039 |
| 1172–1227 |
0.0173 |
0.0193 |
0.0056 |
0.0076 |
Same story with the 40-target model. In the first window I see what you saw, a small edge for the vector head, but it is not significant (paired per-era bootstrap). In the later eras it flips and the scalar model is ahead, more clearly in the last 56. So for the main target I would not call it a win.
The interesting bit: the worse targets got better. Across all 59 heads, the vector heads beat their scalar twin on 1016–1227 (CORR +0.0018, better in 77% of the eras). But the gain is in the weak targets. In the 40-target model, the 10 weakest targets (mostly 20D) gained +0.0036 CORR and +0.0012 BMC, while the 10 strongest (60D, Ender included) only +0.0006 CORR and −0.0004 BMC. Shared trees seem to pull every head toward a common signal: good for the weak ones, not for the best one.
Ensembling the heads did not help as I expected. The average of all heads was worse than the best single head (60D model: BMC 0.0063 vs 0.0071; 40-target model: 0.0047 vs 0.0065). And averaging the 40 vector heads ended up about the same as averaging 40 separate scalar models. Your Ender + Tyler blend result also shows up on my side: a 50/50 blend of scalar Ender and scalar Tyler lost BMC vs Ender alone (0.0020 vs 0.0031 on 1016–1227).
They are not the same models though. The vector Ender head and the scalar Ender model correlate around 0.90 per era (0.77 to 0.95 across targets). That is about as different as a scalar model trained on 30% of the rows vs all rows. Meanwhile the heads inside one vector model are almost copies of each other: the Tyler head correlates 0.99 with the Ender head, more than with a Tyler-only model (0.93).
However, blending scalar and vector heads did not show anything interesting. 50/50 vector Ender + scalar Ender lands in between the two (BMC 0.0028, vector 0.0025, scalar 0.0031), no gain over the scalar model.
A few practical notes
- No free compute: the 40-target model took 63 min vs about 74 min for 40 scalar models on the same A100, so the cost grows with the number of targets. You do get one model file instead of 40.
- Every row needs every target, so rows with a missing target get dropped (the jeremy targets start later, about 1.2% of rows).
- 30% of the rows sampled inside each era was close to full rows for Ender (a bit less CORR in the first window, same or better BMC), a cheap way to iterate.
- This post was mainly written by Opus 5.5, I just vomited the data and ideas hehe.
My take so far: vector leaf gives you a different model, not a better one, at least untuned and on the main target. I really wanted to find something genuinely better, but it requires more grinding. A few more things that I can add:
- Claudia beat Ender at predicting Ender. In both vector models, the Claudia-60 head scored better on Ender than the Ender head itself (first window, BMC 0.0071 vs 0.0062). It’s a nice curiosity.
- Tuning. We could add one line saying tuning matters a lot here, without numbers. Conceptually, It could allow for the models to extract better information during training (I am exploring this right now).
- Weighting targets. Multiplying each centered target by √w makes the tree splits weight that target by exactly w. This trick did not come from me, actually was Astra, and it did make a difference.
- Features. Maybe including more features, can also make a difference.
