tl;dr Numerai will no longer make payouts based on CORR or TC, all payouts will be on MMC only on a new upcoming target called Teager starting the end of this year.
Numerai has made payouts in a number of different ways over the years as this meme from RocketChat shows. I mentioned in the Fireside chat, we need to make changes to payouts to discourage poor performing models from staking. I went through different ideas but I think we’ve settled on something. Thanks @murkyautomata for your help.
On the one hand, it seems natural to pay Numerai participants based on their predictions’ correlation to the target Numerai defines. On the other hand, Numerai already has benchmark models which already have very good correlation with the targets see: Numerai.
Another payout mechanism which seems natural is True Contribution (TC): how much your staked predictions improve the post-optimization portfolio returns for Numerai. However, TC has some weaknesses in that it is blackbox and also tied to certain optimizer settings. The optimizer settings for Numerai One and Supreme are different from each other and change from time to time. To have TC stay “True” the whole time would require constant alterations to it — even the size of the funds influence TC. So TC is challenging to maintain without becoming even more blackbox and mysterious.
MMC (MMC2 Announcement) was the previous way we solved the problem of incentivizing orthogonal signals that are actually contributing to the Meta Model. We take the Meta Model predictions and ask: does a small weight on your signal when added to the Stake Weighted Meta Model improve or hurt its correlation with the target? (This is equivalent to the residual-MMC discussed in the old MMC2 post).
In the past, MMC had some weaknesses. It used to be computed against old targets without feature penalization or without liquidity adjustments (many small stocks need a liquidity adjustment to reflect that a decent size hedge fund can’t buy them in large size). Because MMC on old targets had these weaknesses, we developed TC. However, we are now almost ready with a new target (called Teager) which does almost all the important transformations that the optimizer does within the target making MMC on this target a great measurement of contribution. In other words, MMC on this target is quite close to having TC under new minimal optimizer settings Numerai intends to move to. Also important to note: Numerai now gives out the Meta Model signal so MMC is not a blackbox any more and can be computed locally.
So why change to MMC only payouts? Why can’t it be optional? Why can’t you keep CORR or TC?
In the past MMC and TC were optional to stake on. Many users would simply stake CORR with a large stake and not stake MMC or TC at all. The problem is many large stakers this year have had persistently negative TC & far worse TC than benchmark models. This would be fine if these models were being burned away but if they weren’t staking TC this year, they would hardly burn at all if their CORR was okay or flat. The point of payouts is to get feedback into the stakes so that the Stake Weighted Meta Model can improve. Users persistently hurting the Meta Model but doing okay on CORR shouldn’t be able to earn a positive return on their stake; they should earn a strongly and persistently negative return on their stake.
How much payout multiplier will be available on MMC?
We think MMCx2 is probably makes sense. Whatever multiplier we choose there will only be one multiplier. For example, if we choose MMCx2 everyone who stakes will have to stake with that multiplier and there will be no option for MMCx0.5 or MMCx3. This puts all staked NMR on a level playing field for earnings.
Aren’t you worried this will cause many users to unstake because earning MMC is too hard?
We are not worried in fact we sincerely hope fewer users stake on Numerai after this change. Clearly, unlike any web service I can think of, Numerai can actually benefit from losing users if they aren’t good. Of course, we are hurt badly if we lose the best data scientists i.e. the data scientists who will be the best going forward at MMC. But by making the tournament harder and getting users who don’t even believe they have any MMC to stop staking, we are going to get to a world where the payout factor is far higher for the top data scientists who remain. And of course, users who don’t think they have MMC can still submit unstaked predictions to Numerai until they feel confident enough that they have MMC that they are prepared to stake on it.
To put it even more simply: Numerai data scientists can currently earn NMR even though they hurt the fund performance (they had bad TC but didn’t stake it) and performed far worse than Benchmark Models (on CORR or any other metric we display). Does it make sense for Numerai to reward models like this? What for? It’s a bad incentive. You can see the problem by noting the average return on stakes on Numerai is 30.65% over a year where the Meta Model and fund performance has been poor. Clearly, payouts are too generous in aggregate at the moment to make any sense for us or you in the long term. We were happy to have generous payouts to seed the data science community but going into next season, we need to have much closer alignment. Producing a contributive model to the Meta Model is very difficult; earning payouts on Numerai has to be exactly as difficult.
What about Signals?
There are no changes planned for Signals just yet. Numerai Signals already has payouts on the stricter FNC rather than CORR. Though it is possible the TC payouts on Signals could change to something more MMC-like in the future as well.
More details to follow and we plan to release the new Teager target by the end of the month.