The latest announcement is 0.5 Corr + 3x MMC, although I’m not sure you should bank on that either. (And MMC was corrected on the website earlier in the week from an erroneous version that was previously displayed. So if you had looked at your models a week ago thinking about MMC, check again.)
Thanks @wigglemuse, yes I’m aware of the new MMC calculation. It’s hard to follow all the threads.
That really puts a lot of emphasis on MMC. I hope it correlates with fund performance, I’m not sure we can survive another drawdown like we recently experienced.
Where is described the bug and the correct way to compute MMC?
Is Murky’s version correct as said before?
Is there a R implementation of the function to do it?
The code released is correct, but the website displayed version (before sometime last week) was still using the old Nomi target (leftover from original MMC), not the Cyrus target as it should have been. (Seems like if that was the case, the code was probably wrong too, but whatever – supposedly it is all correct now. I leave it to others to verify that as it seems like many are recreating locally. Do it all match up now people?)
1 Like
The most up-to-date function is always located here: numerai-tools/numerai_tools/scoring.py at master · numerai/numerai-tools · GitHub
You can check what has been fixed in the commits history: History for numerai_tools/scoring.py - numerai/numerai-tools · GitHub
As you can see, the latest fixes were pushed 6 hours ago. 
So it might be a good idea to subscribe to repo changes.
2 Likes
Thanks for sharing this. I have some (probably basic) questions about how the MMC and the payout is calculated.
- Why in the correlation_contribution function we don’t raise the gaussian to the 1.5 power as in the numerai_corr?
- Am I oversimplifying if I see the payout (excluding clipping and payout factor) as 0.5xCORR(P, T) + 3xCORR(P⊥M, T)?
Where:
CORR → Correlation (as calculated in numerai_corr)
P → User predictions
M → Meta Model predictions
T → Cyrus target
P⊥M → The user predictions component independent of (orthogonal to) the meta model predictions
Thank you in advance
There’s also the Gaussianization step for both the meta model and the user predictions that happens before 1.5 power and corr (for corr) and neutralization (for mmc).
Thanks @andralienware, yes this is what I meant with “as calculated in numerai_corr”.
It just surprised me that the steps to calculate CORR (the score) seem to be different from those needed to calculate MMC and not just the neutralization but also the 1.5 power.
I was trying to find a simple way to tie together CORR and MMC like in 0.5xCORR(P, T)+3xCORR(P⊥M, T) but it doesn’t match the code I see in the repository so I guess there is no easy way to represent it.
CORR and MMC are fundamentally different metrics and thus cannot be calculated the same way. They are different both mathematically but also different in their intent. We wanted CORR to capture performance in the tails of your prediction distribution (hence the pow 1.5) whereas MMC cares about how your predictions improve the entire Meta Model (hence no accentuation of the tails with a pow 1.5). As others have stated, we do similar pre-processing in both CORR and MMC, so they aren’t completely dissimilar.
1 Like
Isn’t the reason that pow 1.5 is employed to capture the leptokurtosis of real returns? It seems that if there are issues in the tail of the meta model (which is probably what drives most trading losses and gains), and a model’s predictions correct them, there should be a payout increase corresponding to that. I understand the idea of getting rid of predictions’ components made up of each’s projection onto the meta model, but that does not preclude the idea of using Gaussian-ized and then pow 1.5’ed predictions from being projected. I agree that Spearman correlation does not make sense, but I agree with sirbradflies’s idea that the 1.5 pow makes sense. We all know market returns are not actually Gaussian, and if the pow 1.5 pre-processing step makes sense for CORR for its effects on tails, then it should make sense as something to do before neutralization. Run it by @murkyautomata, and I’m sure you’ll get agreement that pow 1.5 (or whichever transformation you think matches the tails) would be a good idea in MMC.
1 Like
Shouldn’t the Meta Model be deprived of the model contribution before we can use it to compute MMC?
Firstly using a model to compute the Meta Model and asking in a second step how much a model can still improve the Meta Model (MMC), means that MMC disregards the contribution that a model gave in building the Meta Model in the first place.
9 Likes
I’m very interested too in this topic. Computing MMC as taori says have a big computational cost (a MM should be computed with a LOO strategy).
Is there other way to adjust the ‘size’ efect of a model in the MMC?
Perhaps computing MM with LOO only for big staked predictions, those that the difference in MMC computed as now and with LOO have a significant difference. I would like Numerai Team does some test with top 100 stacked models and check the impact in MMC with/without LOO MM build.
1 Like
Thanks to @PTR for clarifying my question on discord. You are totally right.
I forgot that this is also how TC worked. Both MMC and TC try to estimate how much more (or less) of a model is needed in the Meta Model to improve its performance.
So MMC serves to optimize the weights of the models within the SWMM.
From the hedge fund that seems reasonable, but from the user perspective it is totally unfair. The users would like to be paid on how much their models contributed to the performance of the SWMM. That part is the 0.5xCORR, but I believe that is too low for the risk associated with NMR.
Using the stake, and hence payout, as a mechanism to optimize the MM is in conflict with how the users see the stake and the payout and I hope this part of the tournament changes in the future.
1 Like
A models CORR is not how much it contributed to the MM - this is just a measure of how predictive of the target it is. MMC is the models contribution to the MMs CORR. Here are some facts about MMC that you may be missing:
-
If you have positive CORR but 0 MMC then you are providing no uniquely additive signal - you just have a model that we (and other data scientists) already know about.
-
We orthogonalize predictions wrt the MM and most predictions have very small stake so there is virtually no difference between raw MMC and bagged (LOO) MMC.
-
Calculating MMC in a bagged/LOO formulation makes the calculation more opaque to data scientists because you all don’t have access to other models’ raw predictions, thus you can’t optimize for it locally. Local optimization is a key characteristic.
1 Like
If the MM was so good that it can perfectly predict the target, then all models would have MMC 0 and not rewarded.
However MM doesn’t come for free, so you should also reward the models that created the MM in the first place and the risk the users take in staking those models. Currently this part is rewarded 0.5xCORR, which is too low.
if you have positive CORR but 0 MMC then you are providing no uniquely additive signal - you just have a model that we (and other data scientists) already know about.
This is where I disagree with you. You believe you don’t have to pay for the MM, but it has a cost for the users and I am saying you need to reward that
6 Likes
Hello, thanks for your posts I’ve been enjoying thinking about what you are saying.
Just to play devils advocate on this point, and help myself understand a little bit, is it unreasonable to say they have already paid for the MM with historical payouts? If MMC is 0, doesn’t that mean the is no reason to incorporate the user’s signal into the next version of the MM and the MM would remain unchanged if this were the only contributed signal?
Don’t get me wrong, I’m devastated that Corr multiplier is 0.5, and that MMC is 4x higher, it might be the death of my participation, as I was just getting my Classic models to produce some stable signals, and don’t want to “start over”. But, I don’t see the fundamental problem with the team having “bought” the current state of the meta-model with the payouts that have run the tournament over the past years, and now consider this model the bare minimum to be able to improve upon be able to continue in the tournament.
I don’t think it’s a user-friendly way to move forward, and in fact I hate it; it says to me “we’re using your predictions only for as long as we need you,” which is something I feared before I started participating. Just trying to understand if there’s something I’m missing with respect to fairness towards us.
Best,
TDD
1 Like
is it unreasonable to say they have already paid for the MM with historical payouts?
I believe it is unreasonable. The MM has to be built at every round, so Numerai continuously needs the users’ models. But without reward the users would stop submitting.
Just to be clear, I like the current tournament and MMC is much better than TC. I don’t even want the models to be rewarded for predictions that are either useless or detrimental to the hedge fund (if the fund doesn’t go well, everybody lose).
My only concern is that MMC and TC do not take into consideration the effort and risk (!) of building the MM and Numerai doesn’t seem to care much about that. Numerai even considered going for 0 CORR multiplier. That is what worries me.
1 Like
the switch to MMC will cause an outflow of bad performers which will significantly increase the payout factor
But decrease the payout.
What Numerai neglects to acknowledge is that when the outflow of (likely good) performers happens, they cash out. When they cash out, they sell the crypto and as such, influence the market.
In other words, if you extract a benchmark (e.g. short top cryptos or just ETH and buy NMR), you’ll be at a loss. Because the monetary payout “factor” has decreased, not increased due to the NMR impact on payouts.
1 Like
After carefully observing both MMC and CORA20V2 results, it seems that they are closely aligned. If this observation holds true, it prompts the question: why maintain two separate indicators for scoring? A similar pattern is also evident in the signals.
I have been thinking about it for a while and I believe BMC is a better metric to base the payout on.
@numerark have you (the numerai team) considered to base the payout on BMC instead of MMC? Would you share the pros and cons that you come up with?