As an experiment, I’ve been submitting random numbers between 0 and 1 for all the available tickers. The model is called ‘Jersey Devil’ and is currently ranked 71 for corr 20 as of round 278 with one more day to go.
I’m not staking on this model (just the minimum .01 to see returns). I use it solely as a baseline for my other models. If they can’t beat random numbers, they die. Yes it hurts when you put a lot of time and effort into a model and it doesn’t do better than random numbers .
I was pretty surprised this has positive corr at all but we’ll have to see if that continues in the future. Here is a colab notebook with the exact code.
An interesting experiment, thanks for sharing! This is not surprising on second thoughts. A randomly selected portfolio of stocks often performs quite well, especially in long bull runs. From memory there are some studies showing that it handily beats most fund managers. Whether a random portfolio could also beat a hedge fund with great data… I suspect no self-respecting hedge fund would submit to that test.
Doesn’t mean anything really unless it consistently beats everything for a long time. It would be quite expected to have both good and bad runs but be close to zero when averaged over many rounds (but many more than you might intuitively think is necessary).
Yes. Although, as I joked above, neutralization of random numbers would actually add a tiny bit of structure to the randomness (because that which it is neutralized against is not random). Probably not enough to notice scorewise, and if it did turn random numbers into a consistently positively (or negatively) scoring model, then that would probably indicate a bug somewhere.
if the random number is able to beat the submission, it probably means the market is moving towards certain direction at the end. and it could be driven, not by the individual company performance, but the the overall trend or human race behavior, like inflation, innovation, corporatization, etc
Is anyone able to explain why its corr and mmc track each other so closely? (Of course better corr usually equals better mmc but my models do not display this behaviour)
Could it be that if it randomly does well one round, it is likely to get high mmc as other models are unlikely to have submittted predictions similar to randomness?
@testorganisation Because a random submission will have near-zero correlation with the metamodel and so not much if any of the prediction signal is residualized away in the MMC process. (Generally, correlation with metamodel sends MMC towards zero.) So basically the reason they track the same is because they are pretty much the same.
You’ll see this behavior for everybody’s models if the metamodel itself is at zero – when the metamodel essentially is giving no (actually predictive) signal, then each component signal will have an mmc near equal to its main predictions.
I haven’t thought about random for signals - but my thoughts on random in classic is that as we are predicting very-close-to-random anyway (correlation strengths of 0.03), it is pretty easy for random to perform well.
You could have 10 different random submissions, and see how well they cluster around 0, or that they indeed swing collectively above/below zero as well.
In different tournaments, random might never perform well - say, if the tournament is “Landing rovers on Mars”, a random rocket construction and route-plotting program is never going to get close to performing well :-).
Yes, these tournaments are symmetrically designed so that random is at zero in the center. It is exactly as difficult to score -0.1 as it is to score +0.1. But yeah, since even good score magnitudes are so low most of the time it is natural that randomness can be competitively positive for any given round. But it should over time be negative about as much. (But we also know from random walks that once it is on one side of the line, it can manage to stay there for quite a while as far as cumulative performance.)
Maybe I am crazy, but how is 0.011 35% return, when we start with 0.01? Should it not be 0.013? I think the return calculation is just wrong? When we only look at NMR, we can see that it is random because it’s still at around 0.01.
Or wait, today minus 3 months equals to 15.09, when the model went down to 0.08 NMR. The difference to 0.011 NMR it is around a 35% increase, haha. So I guess we are not losing (yet) against a random model.
.
