Are there example models that many people are being submitted on? Very high frequency of exact same score.
Some people buy predictions from other people on numerbay
1 Like
correlations could be analyzed to find repetitions of the same model. I think those models should probably be penalized, because they do not contribute new information to the overall meta prediction.
Yeah there are example predictions as well as example models from the tutorial notebooks which have a lot of identical scores.
They split MMC among them if they are equal predictions, this is a penalization. If they have the same corr but different predictions, there’s no reason to penalize.
Why penalize them? I can stack N NMR to one model or N/2 to two models using the same prediction. The second option is better for me if in future I want to use two differents models. Stacking all to one model is less flexible.
MM is built taking into account the overall NMR staked to each prediction, so where is the problem? They CONTRIBUTE with new information, the more important information, someone is confident enough to put the skin in the game.
This would be the case if the MMC was computed with leave one out strategy, but that is very expensive in CPUs. MMC is computed with your model vs the MM (which include your model), so MMC is independent of your NMR is all-in or shared by 2 o 10 models.