# (Way too early) Comparison of legacy & new models **URL:** \ **Category:** Tournament\ **Created:** [December 9, 2021, 12:53am UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595 "2021-12-09T00:53:29Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![yxbot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/yxbot/32/2906_2.png) [@yxbot](https://forum.numer.ai/u/yxbot)\ **Post date:** [December 9, 2021, 12:53am UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/1 "2021-12-09T00:53:29Z")
Hello all: How are you guys doing with your new models trained from the super massive dataset? or are you diamond handing your good old legacy models? Now that the new dataset had been around for almost 4 months, and some of the new models have up to 10 resolved rounds under their belt, I thought it would be interesting to make some overall comparison. Here is the comparison view of all of my legacy models v.s. new models - the first two are corr, and the last two are corr percentile **corr (upper/lower: legacy/new)** ![image](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/7/72a41726d1cd21b27f2fff2a5fdd6781ab1a70e4.png) **corr percentile (upper/lower: legacy/new)** ![image](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/c/cc3bddfd26fa829803ae09621d3c4c811bfe1bc7.png) **To be clear:** I think it is far too early to draw conclusions on any model’s performance until they have more than 20 resolved rounds but still find this interesting 🙂 what stands out for me for now is model performance divergency - my legacy models do tend to go up and down together, some are more stable than others, but they more or less bundle together. the new models however seem to be behaving quite differently in this aspect. For instance, for round289 my new models have more or less covered the whole spectrum, have not seen model performance spread quite so widely from my legacy bunch… I am more or less using the same data pre-processing steps, similar algorithms, and not quite different validation setup. My guess is that the wider choice of features, and the newly available alternative targets are contributing quite heavily to this divergence. Are you guys seeing the same phenomena? May the burn be with you! 😃 ---
**Author:** ![restrading](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/restrading/32/2928_2.png) [@restrading](https://forum.numer.ai/u/restrading)\ **Post date:** [December 9, 2021, 1:09am UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/2 "2021-12-09T01:09:18Z")
Could the difference in performance (legacy vs new) be due to the difference in targets? 20D1L vs 20D2L nomi? ---
**Author:** ![restrading](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/restrading/32/2928_2.png) [@restrading](https://forum.numer.ai/u/restrading)\ **Post date:** [December 9, 2021, 1:10am UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/3 "2021-12-09T01:10:08Z")
> [@yxbot](#): > > May the burn be with you! Lol this was not how @arbitrage used to say it ---
**Author:** ![yxbot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/yxbot/32/2906_2.png) [@yxbot](https://forum.numer.ai/u/yxbot)\ **Post date:** [December 9, 2021, 1:22am UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/4 "2021-12-09T01:22:49Z")
I thought both legacy and new model are 20D1L up till now? ---
**Author:** ![restrading](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/restrading/32/2928_2.png) [@restrading](https://forum.numer.ai/u/restrading)\ **Post date:** [December 9, 2021, 1:26am UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/5 "2021-12-09T01:26:45Z")
If I understand correctly, the training target for new data is already 20D2L, but we are currently scored on 20D1L untill soon ---
**Author:** ![gammarat](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gammarat/32/3281_2.png) [@gammarat](https://forum.numer.ai/u/gammarat)\ **Post date:** [December 9, 2021, 2:51am UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/6 "2021-12-09T02:51:28Z")
> [@yxbot](#): > > How are you guys doing with your new models trained from the super massive dataset? > or are you diamond handing your good old legacy models? I’m quite happy with the new data and have been phasing out my legacy models; the last of those I submitted 3 weeks or so ago. Part of that is due to the size of the new data forcing me to rethink my approach, so with round 281 I took advantage of the extension to 50 models from 30 and introduced a somewhat different algorithm as well. The cumulative results are shown below: ![image](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/5/5c14fcedf4e0343f538dd917d19353b9007967a5.jpeg) All but one are grey, that’s just because of the way Numerai has the colour tables set up. I have 2 parameters that govern this model, one that can take one of 4 discreet integer values, the other can take 5, which results in the clustering of the tracks. These are GammaRat 31 through 50. Around round 285 (iirc, I don’t really keep track) I took the model above and introduced a new parameter, and replaced GammaRat 11 to 30 with that. (Those had previously used the legacy model). About half have really improved, and the rest not. But that’s ok, because it’s giving me a decent view of how these parameters interact. So I’m using that info to once again redesign the underlying algorithm. Fun and games, to be sure 😆 ![image](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/f/f7c154eb2759a4ba82b28d22070e5834b9c92a2b.jpeg) GammaRat 1 to 10, my last legacy models, got dropped three weeks ago and replaced with a similar algorithm to the ones above. They don’t look great, but right now it’s too early to say for sure. ---
**Author:** ![yxbot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/yxbot/32/2906_2.png) [@yxbot](https://forum.numer.ai/u/yxbot)\ **Post date:** [December 9, 2021, 7:46am UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/7 "2021-12-09T07:46:09Z")
or shall I say “may the burn be against you” 😂 ---
**Author:** ![restrading](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/restrading/32/2928_2.png) [@restrading](https://forum.numer.ai/u/restrading)\ **Post date:** [December 9, 2021, 7:57am UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/8 "2021-12-09T07:57:11Z")
“May the burn be in your favor™” 🙂 ---
**Author:** ![yxbot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/yxbot/32/2906_2.png) [@yxbot](https://forum.numer.ai/u/yxbot)\ **Post date:** [December 9, 2021, 8:18am UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/9 "2021-12-09T08:18:10Z")
oh haha I see, appropriately trademarked ---
**Author:** ![yxbot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/yxbot/32/2906_2.png) [@yxbot](https://forum.numer.ai/u/yxbot)\ **Post date:** [December 9, 2021, 8:21am UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/10 "2021-12-09T08:21:40Z")
nice, interesting to see how this developed. I haven’t fully replicated all my modelling methods on the new dataset yet - with the newer data update coming, probably I will do more after December. Nevertheless, after a few tough rounds, most of the legacy models seem to have recovered - some of them never suffered in the first place - so I am just happy seeing them running. I would definitely keep most of my new models at least for 20 rounds to see how they play out longer terms 🙂 ---
**Author:** ![wigglemuse](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/wigglemuse/32/3094_2.png) [@wigglemuse](https://forum.numer.ai/u/wigglemuse)\ **Post date:** [December 9, 2021, 2:13pm UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/11 "2021-12-09T14:13:14Z")
I think a possibly underrated subject of the new data is all the new targets. I think they’ve been maybe more helpful to me than the new data itself. ---
**Author:** ![platemort](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/platemort/32/1143_2.png) [@platemort](https://forum.numer.ai/u/platemort)\ **Post date:** [December 9, 2021, 2:48pm UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/12 "2021-12-09T14:48:12Z")
Do you have the cumulative scores plotted? I can’t make much out of these plots. ---
**Author:** ![platemort](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/platemort/32/1143_2.png) [@platemort](https://forum.numer.ai/u/platemort)\ **Post date:** [December 9, 2021, 2:54pm UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/13 "2021-12-09T14:54:26Z")
My experience is that my legacy workflow has beat a similar workflow on the new data. I don’t have all the comparison data together to show, but yesterday my legacy model returned 4.8% and was 95 percentile on corr and mmc. My co-modeler burned -0.4% and was 40 percentile on corr and mmc. I’m not excited about being forced down the new data route. ---
**Author:** ![yxbot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/yxbot/32/2906_2.png) [@yxbot](https://forum.numer.ai/u/yxbot)\ **Post date:** [December 9, 2021, 2:56pm UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/14 "2021-12-09T14:56:28Z")
No not yet, good idea though, I will add that when I come to further work in my dashboard ---
**Author:** ![yxbot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/yxbot/32/2906_2.png) [@yxbot](https://forum.numer.ai/u/yxbot)\ **Post date:** [December 9, 2021, 2:59pm UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/15 "2021-12-09T14:59:10Z")
Yes that is one of my observations at the moment, from my point of view seems some of the new targets are more volatile than Nomi Not sure how much they help though, probably need a longer runway to see. For now my legacy models are out performing the new ones simply by being more stable ---
**Author:** ![yxbot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/yxbot/32/2906_2.png) [@yxbot](https://forum.numer.ai/u/yxbot)\ **Post date:** [December 9, 2021, 3:00pm UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/16 "2021-12-09T15:00:59Z")
It is my recent rounds observation too, although I think more rounds are needed to draw any conclusions. I don’t like being pushed to use new dataset neither. Have you tried to 300+ features they said are closely related to the legacy features? ---
**Author:** ![platemort](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/platemort/32/1143_2.png) [@platemort](https://forum.numer.ai/u/platemort)\ **Post date:** [December 9, 2021, 3:05pm UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/17 "2021-12-09T15:05:56Z")
I have not tried only using those. I didn’t realize that there was a list of features that were “closely related” to legacy features. I remember the question being asked about which features are the old features and the answer being that none are the same because of the timing differences. Is that list published or maybe I just missed it in the original announcement? ---
**Author:** ![platemort](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/platemort/32/1143_2.png) [@platemort](https://forum.numer.ai/u/platemort)\ **Post date:** [December 9, 2021, 3:25pm UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/18 "2021-12-09T15:25:33Z")
Needs fact checking but from memory the tournament 3 month avg was around 15-20% when the new data came out and is now at 7.7%. I was around 25% when it came out and now I’m at 49%. I’m assuming that the meta model is currently dominated by models on the new data set, while I have been playing it safe and sticking with my legacy model while I see how the new data performs. Green is staked legacy, Orange is unstaked legacy experiments, Cyan is unstaked super massive. ![image](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/3/33be5c94554ba87b23269f7903a293bbdfb684db.png) ---
**Author:** ![yxbot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/yxbot/32/2906_2.png) [@yxbot](https://forum.numer.ai/u/yxbot)\ **Post date:** [December 9, 2021, 5:15pm UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/19 "2021-12-09T17:15:14Z")
from the team’s [October Updates](http://forum.numer.ai/t/october-2021-updates/4384) under **New Feature Metadata** - the “legacy” set > **Legacy** : 304 of the original 310 features that were carried over to the new dataset. You can use this set to achieve nearly the same model as the legacy data. ---
**Author:** ![platemort](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/platemort/32/1143_2.png) [@platemort](https://forum.numer.ai/u/platemort)\ **Post date:** [December 9, 2021, 6:04pm UTC](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595/20 "2021-12-09T18:04:00Z")
Are we to assume that it is the first 304 features? [Next page](https://forum.numer.ai/t/way-too-early-comparison-of-legacy-new-models/4595.md?page=2)