It seems that low CORR W/METAMODEL model tent to get high TC.
Top TC models on the leaderboard have very low CORR W/METAMODEL.
This could makes sense, as a random model will have in average 0 CORR with response and 0 CORR with metamodel if you give more importance in TC to uniqueness than correlation the balance will be positive.
This is a sign of overfitting in the TC computation if random noise added to the metamodel improves it.
2 Likes
There are random models and then there are random predictions. It is not weird at all that randomness will have good rounds â but it should also have equally bad rounds. And if it is a random MODEL, meaning that it was randomly created but once created remains the same model, then it is also not weird that it would exhibit streaks of goodness or badness because there is high auto-correlation between round results (at least much of the time). Any random model will settle somewhere over time â most likely right around zero, but a randomly-made model can also be randomly good or bad (but again, not likely to be strongly one thing or another unless it isnât as random as you thought â some ârandomâ methods make very good models). (And do we have any information about what this example actually is?)
If there is no model and there are just random PREDICTIONS created fresh each round, then the first thing holds but there should be less streakiness involved because now the predictions would have no auto-correlation with themselves. But streaks appear randomly too, and those streaks can be much longer than your intuition would think.
As far as being less correlated with the metamodel leading to greater TC, of course it does, no surprise there and the team has pointed that out repeatedly and has been encouraging more original models. However, this uncorrelation BY ITSELF wonât give you TC riches, but it is more like it is increasing the potential TC capacity of your model, i.e. it still has to be a good model â your potential for high negative TC grows with your potential for high positive TC as you become less correlated with everybody else.
4 Likes
Iâve been doing random predictions for signals for awhile: Numerai
Like wigglemuse said, there are good runs and bad runs. Overall itâs down about 30%
2 Likes
Rightly or wrongly, these tournaments are beginning to look deeply suspect, like so many enterprises based on crypto coins.
Some time back, I was getting really poor 1% on TC. The correlation was more stable but not much better and my Numeraires were getting burnt off at an alarming rate. So, partly as an experiment, I cancelled the stake. As soon as I had done that, my TC shot up to 95%+. So I turned the staking back on and now TC has again fallen back to around 1%. I am not talking about one-off week here but several weeks running in each case. Deliberate or not? I guess we will never know. It is certainly in no way transparent.
3 Likes
Mean reversion. If your NMR is burning at an âalarmingâ rate you are staking beyond your risk tolerance (and possibly beyond what math would sensibly indicate). When bad periods inevitably arrive, you shouldnât be panicked â itâs a bad period and it will pass. If you have staked too much (or at too high a multiple) that those periods are going to freak you out and make you pull stakes, then also inevitably you are going to miss the following upswing. This is pretty much a universal pattern in betting/gambling. Strong emotions (up or down) attached to particular outcomes in short time-frames shouldnât happen â they are a sign of over-staking (psychologically and probably mathematically) â and they lead to hasty emotional decisions.
But Iâd just point out this isnât a casino â there is no motive to rig the game. Numerai has nothing to gain by you losing. They do well when we do well â by extension when you do well. When you burn, do they gain anything? Nope, that means they are suffering also. SoâŚTC is a black box to us, itâs true. (Hopefully that wonât be true forever.) Bugs or incorrect calculations are a possibility (just from the fact that we canât independently verify). ButâŚâdeeply suspect, like so many enterprises based on crypto coinsââŚthat sounds like an implication of deliberate tampering with results. Again, there is no motive, it is contrary to their interests to do that.
4 Likes
Except my money, of course.
It is only alarming in as much as it may take me out of playing in these tournaments.
I am not that stupid to consider NMR as some kind of an investment.
Anyway, the degree of my alarm is not really the main point here.
Having participated for 75 rounds and reviewed the performance against CORR and TC, I now believe the tournament has essentially shot itself in the foot by moving to an unpredictable target. When looked at in combination with the ever decreasing payout factor, the volatility of the token in relation to fiat, and changes to the structure which sometimes need manual recoding to stay âcompetitiveâ, it becomes apparent that there is no point in spending time continually improving a model when the reward is not directly linked to its individual performance. That is simply gambling, and there are many easier ways to do that.
5 Likes
NMR burns go to the burn address, not Numeraiâs treasury. They donât gain from your burning.
1 Like
They do not get your money, thatâs my point.
If you want to criticize TC for being terrible (like @magic101 above) and a mistake and a bad idea, etc, I wonât argue with you. Thatâs a legit argument. But when you imply they are running a scam or stealing from you by manipulating scores, there is nothing to support that and again zero reason to do it. They want the fund to to well â to drive people away making good models by tinkering with their scores to no benefit to themselves (and losing the benefit of your good model) doesnât make any sense at all.
5 Likes
The decreasing payout factor is a necessary consequence of CORR staking. Anyone who can copy a sample notebook can farm NMR staking on CORR, so naturally total CORR stakes increase. Unless you expect numerai to provide exponentially larger payouts over time, CORR staking must require a diminishing payout factor.
TC being more of a competition does not have this problem, but for the same reason it is more difficult. TC is not random. We know that it correlates with CORR at around something like 0.25, and if you submit a model that outperforms the rest of the meta-model, a positive TC is a certainty. Reasoning about what creates a positive TC may be more theoretical than what many data scientists are used to, but it can be done and many on the leaderboard have managed a fairly consistent return on TC.
3 Likes
Well, let us hope so. My main argument is that with a âblack boxâ, with obfuscated inputs and obfuscated outputs, it can never be more than just a hope.
The advantage of smart contracts is that everyone can see what they will do. The smart contract, which dictates what happens to burned NMR, is not a black box.
1 Like
Nowadays, I am getting corr20 over 90% and corr with metamodel negative. Are these not the circumstances, from which the metamodel should be learning and therefore TC ought to be high? Yet, I am still getting poor TC. Could you please explain to me what is going on?
Someone did a post (I think 2 months ago?) where it can be seen that low corr with meta model will also increase TC volatility up to a point where your mean TC has to be really high otherwise you have basically random TC.
Another thing that I found during my experimentation (I am still waiting a few weeks to have more definive results) that TC seems to be more related to ranking metrics than pure correlation. Imagine having an onlineshop where your search will lead to 5000 products that might be of related to your query. Nobody is interested if the products somewhere on page 50 are accurately ordered by your preferences, but only the top results are important. In case of Numerai, it seems that both top and bottom results are equally important, especially for TC. So TC is a really different metric, more so than other metrics.
Another thing is that TC is a gradient, not a global optimum, which can be quite confusing, at least it was (still kinda is) like that for me. It means increasing your weight in the meta model ensemble just ever so slightly while also considering risk constraints makes the MM worse, eventhough your model alone might be better in this round. There still can be a local performance minimum in the direction of your model weight. eventhough the maximum behind it might be higher than in its current configuration.
1 Like
you probably need to separate out the different types of models that have low corr with meta model for that analysis to be right. the two types being an actual model that just happens to have low corr and a random seed that by definition has low corr
1 Like
While they did show some stats about mean and corr w/ meta model (CWMM) that would be consistent with that conclusion based on those two things alone, nevertheless that conclusion is not warranted, at all. It only tells you that lowering your CWMM will not automatically give you a higher TC mean, but it will probably give you higher volatility. (And this is not really news.)
3 Likes
Whatever the actual mysteries of TC are, the ranking kind of makes sense. That is why I referred above to percentages of my corr20.
Could it be that my actual model nonetheless produces random results? Hmmm. Something to ponder.