How are you guys doing with your new models trained from the super massive dataset?
or are you diamond handing your good old legacy models?
Now that the new dataset had been around for almost 4 months, and some of the new models have up to 10 resolved rounds under their belt, I thought it would be interesting to make some overall comparison.
Here is the comparison view of all of my legacy models v.s. new models - the first two are corr, and the last two are corr percentile
corr (upper/lower: legacy/new)
corr percentile (upper/lower: legacy/new)
To be clear: I think it is far too early to draw conclusions on any model’s performance until they have more than 20 resolved rounds but still find this interesting
what stands out for me for now is model performance divergency - my legacy models do tend to go up and down together, some are more stable than others, but they more or less bundle together. the new models however seem to be behaving quite differently in this aspect. For instance, for round289 my new models have more or less covered the whole spectrum, have not seen model performance spread quite so widely from my legacy bunch…
I am more or less using the same data pre-processing steps, similar algorithms, and not quite different validation setup. My guess is that the wider choice of features, and the newly available alternative targets are contributing quite heavily to this divergence.
Are you guys seeing the same phenomena?
May the burn be with you!