True Contribution for dummies

I don’t understand the scoring at all and how the current process is deemed “fair” when my models out perform most all top 10 models.

What metric are you looking at when you say that your models outperform the top 10? Correlation?

Maybe im wrong and it was supposed to say top 100… and yes correlation. maybe Im just too simple and don’t understand the complex way in which the leader board is determined. But if models that have been burning NMR for 2 months are considered good then I have it way backwards as to what the goal is.

Burns depend on what you stake on and how much round-to-round – lots of choices there. Leaderboard position is score based only – it used to be corr, now it is TC. (20 round moving average with the current live rounds gaining in weight each day while the 4 least recent of 20 lose weight each day – at least that’s how it was done on corr). Still, I only see people with a lot of green (earns) on top of the leaderboard so don’t know what you’re talking about there really.

Whether or not you see a lot of earns really doesn’t matter as I am obviously not speaking on those models. Just those that have a lot of burns. It seems they are being rewarded for the stake and not the work put into building the model. I see just as many with a lot of burns, which my models dont burn so i was just wondering. No offense to you if one of the models are yours. I mean no harm just want to understand.
Thanks for the info on how TC is calculated.

Can you give an example of a model that’s been burning NMR the last two months that is also ranked high?

This has been already acknowledged in here, Numerai knows it works like that, but they dismissed the problem as a theoretical one that doesn’t happen in practice. I would be happy if they ran some proper simulations and proved the users that they don’t have to worry, but they simply got rid of the matter with just one questionable explanation. But that is a fundamental property of TC that requires more attention.

At the same time, users keep seeing that TC doesn’t correlate well with model metrics, so there would be good reasons to investigate more.

Numerai has properly tested the effects of the TC mechanics on their fund (it has been reported multiple times how the performance has improved with TC,how many simulations they ran, etc), so we know it has been a great change for them, but why don’t they do a thorough analysis on the payout scheme too (the user perspective of TC)? I mean, even if there was a problem in the payout, that could be improved without getting rid of the benefits that TC brings to their fund. I don’t see why there is no discussion on this topic. Maybe I am just wrong.

I just would like to see evidence that it is not a real concern and I would be happy.

No I won’t give any examples. I’ve done that before and ended up offending someone.l.

I think anyone can go through the models and compare their standing to others. How does a model with a consistent negative corr have 100 percentile TC? Thats all I want to know, and until I can answer that question I will only hold NMR.

I put so much work into learning this that at one point I had to see a doctor for a neck strain.

I knew nothing about data science a year ago, now I can build models in both python and R. I can also build compute nodes, and that alone made Numerai worth it.

I’ll be back once I have an understanding of the scoring.

Good luck to all

You didn’t offend me, if I’m who you are referring to. I was just wondering what you were talking about, same as @joakim. Because the top of the leaderboard (which I am not on I assure you) doesn’t contain a bunch of models that are doing a bunch of burning. It just seemed a weird thing to say.

2 Likes

Whats even weirder is that you’re so focused on whether or not there are a bunch burns in the models. Listen this post is about TC, and my question is about TC, yes i’m a dummy. I’m sure you’re like a genius or something, but If you cant answer the question then move on.

Let me reiterate, why are there models with no correlation ranking at the top of TC? Said models would burn NMR under the previous scoring system. Im sure my question doesn’t apply to all models and i’m sure you haven’t checked all models.

Sorry I offended you or your model.

I am unhappy about the current scoring system as you are and I can understand the inconsistency of the leaderboard you are referring to. I believe there are good reasons to consider the current scoring system unfair, although I think that TC is a good mechanics for the fund and should be kept while fixing the payout. I wish Numerai could provide more data, tests and explanations on why they believe the current tournament is fair.Maybe it is just a matter of seeing things from the right perspective.

Hello, Though my English is very limited,
I have tried to make an explanation for your question.

This is just a schematic of my personal understanding,
and there may be incorrect things.

I would be happy if it helps. If not, please offend me too.

5 Likes

@annon I like you explanation, very intuitive. That explains why a model with negative correlation might be required by the Numerai’s fund and for that reason it has positive TC. All good, but what about the payment based on TC alone? Could you tell us your thoughts?

I believe you cannot pay models on TC alone. The meta model itself is built from all the models that are indeed highly correlated with it. They contribute for the majority of the predictions and they need to be paid for the computation and their stake at risk, although the gradient will give them TC~0. A fair payment logic would include not only TC but also the part of the predictions correlated with the Meta Model.

2 Likes

Thank you for your kind reply.

I agree with you, I think the payout system that includes Corr is better than only TC.

The reason I think so is that the source of TC is finite.

I’ll try to write some more intuitive things.

Below is an intuitive diagram.

As the metamodel improves, the sources of TC will decrease.
So TC could be more difficult to find out.

Also, if the sources of TC decrease, The signal/noise ratio will then get worse, and the volatility of TC would increase.

(Regarding why TC is noisy. Besides the guess that extreme prediction would get high TC scores, I think one reason is that the signal of TC is relatively small.)

I am currently staking on 3xCorr and trying to make adjustments for TC, but if the TC difficulty increases in the future, I may back to Corr.

4 Likes

When we use CORR to evaluate models, users are rewarded if the model performs well, and punished if the model performs poorly.

When it comes to TC, it’s not the case. TC is not a metric to evaluate the quality of the model itself, it reflects whether a model can improve the overall return when it working with other models.

A model may get punished for not working well with other models, even the model may be a good model itself. (For example, models with positive CORR MMC and FNC get a negative TC.) This situation is unacceptable for model developers. Therefore, TC is a big risk to me.

Doesn’t it make more sense if TC is just for rewards and no punishments?

2 Likes

Sounds like building a model to align with market performance, vs building a model to help numerai fund to get better performance, are two different goals. For first goal, we are competing independently, between designer and market. Under the second situation, the result was heavily impacted by those unknown teammates.

Then, the followup question is, whether the quant access those data - their teammates, to help them purposely improve the model, in order to improve the TC.

I will try to understand the logic below using one example i saw from Signal. Round 329. CORR 6.5%, IC: 0.7%. TC: 95.5%. I don’t quite understand the logic and removing and add in that signal will help dramatically improve the portfolio performance, considering CORR/IC has such low score.

1 Like

First I’d like to award you my “Salt Crown”, I’ve owned it far too long. Go forth and do great things.

Secondly, there are a LOT of really helpful people on the forum and in chat, Wiggle is one of the few who can quickly boil down the really complex into simple terms we can all understand. We hold him in high regard: getting cranky with him could leave you on the outs with the rest of the community and will certainly discourage him from commenting on your queries. I’ve been there, it’s a crappy place to be. Food for thought.

Finally, “why are there models with no correlation ranking at the top of TC…” because TC isn’t correlation. You don’t have to be a genius to understand they are completely different things, you just need to accept that they are and that TC is important to Numerai.

Competing in the tournament is completely optional.
Staking on your submissions is also completely optional.
So TC really doesn’t pose any risk until you’re certain you have a handle on it.

“The obstaCle is the way.”

2 Likes

Its even funnier that you haven’t posted in 7 months and you came back just to reply to me…lol. Thanks Numerai… you’re not obvious at all lmao.
Im only asking questions that FINRA would ask. In order for the platform to grow integrity must be maintained.

Thanks for this information

@crownholder @objectscience is also an active member of the community and posts every once in a while in RochetChat. Checkout https://rocketchat.numer.ai/ if you haven’t.