Sounds like building a model to align with market performance, vs building a model to help numerai fund to get better performance, are two different goals. For first goal, we are competing independently, between designer and market. Under the second situation, the result was heavily impacted by those unknown teammates.
Then, the followup question is, whether the quant access those data - their teammates, to help them purposely improve the model, in order to improve the TC.
I will try to understand the logic below using one example i saw from Signal. Round 329. CORR 6.5%, IC: 0.7%. TC: 95.5%. I donât quite understand the logic and removing and add in that signal will help dramatically improve the portfolio performance, considering CORR/IC has such low score.
First Iâd like to award you my âSalt Crownâ, Iâve owned it far too long. Go forth and do great things.
Secondly, there are a LOT of really helpful people on the forum and in chat, Wiggle is one of the few who can quickly boil down the really complex into simple terms we can all understand. We hold him in high regard: getting cranky with him could leave you on the outs with the rest of the community and will certainly discourage him from commenting on your queries. Iâve been there, itâs a crappy place to be. Food for thought.
Finally, âwhy are there models with no correlation ranking at the top of TCâŚâ because TC isnât correlation. You donât have to be a genius to understand they are completely different things, you just need to accept that they are and that TC is important to Numerai.
Competing in the tournament is completely optional.
Staking on your submissions is also completely optional.
So TC really doesnât pose any risk until youâre certain you have a handle on it.
âThe obstaCle is the way.â
Its even funnier that you havenât posted in 7 months and you came back just to reply to meâŚlol. Thanks Numerai⌠youâre not obvious at all lmao.
Im only asking questions that FINRA would ask. In order for the platform to grow integrity must be maintained.
Thanks for this information
@crownholder @objectscience is also an active member of the community and posts every once in a while in RochetChat. Checkout https://rocketchat.numer.ai/ if you havenât.
Letâs roll back to where we were before things became emotional and personal, keep this thread focused on TC.
It would be interesting to know how the other users perceive the payout based only on TC, is it fair or not? At the moment we have the choice of being paid on correlation too, which is fair. However I am afraid that if we keep accepting the idea that TC is the true contribution, while in reality it is just a part of it, then there will be a time when payout based on correlation will not be possible anymore and we will be stuck on an unfair tournament. And even if payout on TC was fair under a certain logic, there would still be the issues brought up by @sunkay and @annon
I wish this thread stimulate more users to give their feedback so that we can have an overview of what is the general feeling of the community around payout based on TC.
I think Numerai would like to remove staking on CORR. They probably will not do so to avoid a mass exodus and NMR crashing event.
Iâm fine with TC only. However, I admit that my personal models suck at TC and I have been forced to buy good TC models on Numerbay to stake on (thanks Numerbay!).
My initial intuition about CORR was that it needed to remain in order to give the metamodel a foundational center. With TC-only we could see the metamodel âmove aroundâ for no other reason than people are trying to have their own models be less correlated to it. (i.e. it might oscillate like a pendulum back and forth over the same spots for no other reason besides individuals are incentivized by TC to not have their models be like other individuals.) Whether that is actually what would happen I donât know. Surprisingly, Richard at one point a few months ago said pretty much the exact same thing â we need to have corr in order to having something stable. So there doesnât seem to be a big push on Numeraiâs end to get rid of corr. Which is wise, because corr is what got them this far.
However, with the effect of the payout factor and corr being limited to 1x staking, corr is becoming less and less viable a vehicle to make much return. (Iâm personally trying to get away from corr as primary for just that reason.) Still, it has been interesting that from what I can tell (just looking at the corr w/ metamodel for models that have been running a long time) the metamodel has changed remarkably little since the introduction of TC.
This does not necessarily mean that people just arenât staking on TC or changing up their models trying to get TC. It is quite possible (Iâve pointed this out before) that even if most everybody changes up their models to maximize TC and neglects corr that the metamodel that it all creates will turn out to be more-or-less the same as it would have been under a corr-only system. Different route, different results for different individuals, but when ensembled together very close to the same result. Just a huge no op. Thatâs definitely a possibility.
But it is also a possibility that TC just really hasnât had much impact yet â even if large stakers adopt it you could say it doesnât really count as âtransformationalâ if they didnât specifically gear their models to TC in the first place, i.e. for pre-TC existing models stakers may enable TC staking on particular models that seem to get TC, but not on others that donât and they continue to stake corr-only on those (and without adjusting amounts too much). And I think we do see this with the biggest stakers â theyâll take some percentage of TC if it looks reliable enough, otherwise not. But no big wholesale model revolution trying to get TC. The staking feedback by itself (if it doesnât create a impetus to make new and different models with significant stakes) will be fairly slow in reallocating stakes to higher TC models (if stakers â particularly big stakers â even allow that reallocation).
Itâs already explained in the docs. Do you expect him to tutor you in linear algebra until you understand it?
I donât expect they will make the payout entirely TC based because TC appears to be zero-sum or at least very nearly so. Neither Numerai nor the participants benefit from the tournament becoming a zero-sum game. Numerai wants data scientists to have an incentive to participate, and itâs not as if they gain anything from burns.
The question as I see it is how much incentive does there need to be, or in other words how much of a positive sum does the game need? With too much the payout factor diminishes, people farm NMR just by submitting sample predictions, and the metamodel is saturated with these lazier submissions. With too little there is no reason for the lower performing half to participate, they drop off and the next highest group becomes the lower half and they start burning and drop off and so on until no one is left.
The solution seems pretty straight forward: adjust TC multipliers in line with payout factor but keep the corr multiplier constant. Eventually an equilibrium will be found where the corr payout is just enough to maintain the current total stake sum.
I would suggest though that staking on TC probably shouldnât be optional. As long as its possible to farm yield just by submitting sample notebooks the payout factor is likely to continue to decline.
Wiggle do you think the Wobble could be stabilized âin-houseâ by them using their own CORR based models, and then relying on the crowd to provide TC? It seems to me TC only staking would greatly extend the life of the treasury and generate much higher value for NMR spent. If Iâm not mistaken current payouts arenât sustainable, and at some point payouts need to fall into some reasonable percentage of fees earned.
I have a question In point #4:
And if we applied this process (gradient computation and stake update) multiple times we could find the optimal stake values for that round, the one that produces the portfolio with higher returns. That would be overfitting though, so the stakes are updated only once per round.
Why Numerai doesnât use optimal stake values to calculate TC (ie: apply the process multiple times )? I can only think of one reason which is because thereâs no single optimal solution, is that it or there are other reasons?
Thanks for this post.
IIRC, TC is the gradient of portfolio return over stake multiplied by a constant. There might be an optimal point for stake allocations each round, but itâs unlikely to be stationary over time. That constant muiltplier would control how greedily the meta model chases the optimal point, similar to the learning rate in gradient descent. Users can also control how greedily they chase the optimal allocation of their models by setting their own TC multiplers.
What I am getting after reading your reply and reading the post again is that when calculating TC of a model for a new round, my previous TCâs will be added to my current stake, did I get this right?
@smokh no, the TC for any round is the gradient of the portfolio return with regard to the modelâs stake for âthatâ round (multiplied by a constant). I.e. how much the overall portfolio âwould haveâ improved/degrade if you had 1 more NMR staked for that round. TC value doesnât carry across rounds.
Okay good, I understand this part so my question remains:
That constant muiltplier would control how greedily the meta model chases the optimal point
Whatâs wrong with greedily chasing the optimal point?
@smokh thereâs nothing wrong with greedily chasing the optimal point. I use the word âgreedilyâ to refer to the heuristic instead of using it as a judgement.
UmmâŚmaybe. But not as well as not doing that (i.e. not as well as it happens now). And then they would also have to decide how much weight to give those internal models. Technically possible but I have a feeling theyâd reject that idea as unNumerian.
In mass online collaborations (my experience being MITâs Theory U, Aarhus Universityâs Quantum Moves, Stanford Universityâs Eterna and now Numerai), each person becomes a node in the network. Personal identities are subsumed into and become superfluous for this new organism and its mission. Within this network/organism, as each node fires, other nodes respond to weed out noise and connect signals to associated messages and nodes to formulate new meaning and organism movement. The quicker, more frictionless and the higher the node awareness is of this process, the faster and efficiently the organism develops meaning and capacity to evolve further.
