Facing the dilemma of “COR vs MMC staking”, I looked closely at round performance of my (and other users’) models. I simply plotted COR vs MMC scores of my models and also included performance of current top500 users.
Each scatter-plot shows the results of all models in one round (rounds 212-209 haven’t resolved yet). Each dot represents one user model with X coordinate is correlation score and Y is MMC score (blue triangles are my own models). At each scatter-plot, I added the indifference line: ABOVE the line model is better of staking at MMC score, bellow that line user better to stake on correlation.
At “bad” rounds (like 195-198), MMC is a good way to improve returns for most of the models.
At “good” rounds like 203-209 COR staking bits MMC staking for most of the users’ models. So in general MMC staking strategy provides some kind of protection against performance volatility.
I’ve done my own simple theoretical research of COR vs MMC staking using validation data. As the rough approximation of the MMC payout I’ve used the equation: 2 * (my model’s correlation - example predictions correlation * 0.5) per era in validation set. 0.5 was chosen because my predictions have 0.5 average correlation with metamodel for the last 4 rounds. The results are 0.0268 average payout (sharpe 0.9) using MMC staking and 0.0278 average payout (sharpe 1.7) using COR staking. Current live results (4 unresolved rounds) are: 0.0243 average calculated payout (sharpe 1.1) using MMC staking and 0.0268 average payout (sharpe 1.4) using COR staking.
Probably all that just mean that my model is not really useful for the metamodel
I’ve been checking models returns COR vs MMC, and I think it makes no sense moving from COR to MMC. Checking returns of big stakers that have been moved to MMC…
I haven’t explored other models but I have a couple of models with low MMC correlation but MMC score is heavily dependant on COR performance. When I have COR < 0, even being on top percentiles (less negative than the rest) I get a bad MMC because my model is contributing on a negative original way. I understand numerai wanting fresh approaches, that’s why I see MMC as a leaderboard/originality bonus replacement, “you do well, you get paid. you do well and your model is doing something original, you get paid a bit more”.
I agree with this idea - rewards originality, and avoids punishing that originality when/if the MMC correlation changes. If I understand correctly, the actions of other participants will affect the payout/burn (unlike with COR - which is determined by the market).
What does switching the method look like? If I wanted to try MMC for a week and then go back to CORR the following week, could I? Is the 4 week time frame relevant in any way for switching?
We should be able to switch for each round – wouldn’t make sense otherwise. So if you are set on MMC at the start of a round (meaning Thursday when any other stake changes would take place), then the payment for that round 4 weeks later will be on MMC. (Remember you only get paid for a round at the end of it now.) But presumably, you could change it back to CORR the next week, but it would apply to the next round. So you can’t change CORR->MMC or MMC->CORR on a round that’s already started, but you could alternate round to round if you wanted.
Gotta admit, not crazy about MMC either. To me it seems like a way to reduce risk of volatile or bad time periods at the cost of introducing risk involving what other people do, and what random subset of them is chosen.
Actually… If you really want to make MMC a bonus substitute, staking should just be general, and the model should just get paid on the better of the 2 choices automatically.
I suggested that, but Mike pointed out that would be vulnerable to a p/1-p attack because you could have opposite accounts both getting the best return (corr vs mmc) automatically which would guarantee a net positive result.
So basically any form of anything that looks like compensation will get destroyed by this attack… The only way to earn anything is gambling on how predictable the stock market is, with the best loaded dice we can make. A P-1 model or anti model really just says the market wont be predictable in this era (based on whatever predictability looks like to the modler, and whatever these features are).
Choosing MMC seems super arbitrary since most of the data we can go on is for models that aren’t in prod now, assuming people have begun using the new validation data a few eras back.
