-0.0854 is your TC score, 2.2 is TC Percentile
1 Like
And this is other model:
having
5.1 PCT CORR
6.6 PCT MMC
7.2 PCT FNC
38.1 PCT FNCV3
The magic of TC: 0.1646 with PCT 99.99
It’s a JOKE
4 Likes
@mdo are you really confident in gradient is measuring what we supposed?
Could you please do a fast and clean test? Check if the gradient is working well.
Compute the metamodel without this last model (with 0.1646 of TC) and see how much
the metamodel is worst.
Yeah markets are quite volatile at the moment and a metric based on portfolio returns (i.e. the returns of a subset of the stocks) is going to be much choppier than correlation to a normalized target across all stocks. With lots of stocks making double digit moves such things are to be expected.
7 Likes
ditto.
__ __ _ ____ _ __
__ / /__ / / ___ ___ __ __ | | /| / / / (_)__ ___ ___ / /_
/ // / _ \/ _ \/ _ \/ _ \/ // / | |/ |/ / _ \/ / _ \/ _ \/ -_) __/
\___/\___/_//_/_//_/_//_/\_, / |__/|__/_//_/_/ .__/ .__/\__/\__/
/___/ /_/ /_/
Surely, the actual predictive performance (correlation) is all that should matter.
I can see the usefulness of these additional arbitrary measures for the building of the ‘metamodel’ internally but not for the individual predictors.
I agree that it seems that way… but only Numerai sees the “actual predictive performance”, by taking market positions. And based on what they see, they adjust the incentive package. And re-adjust, about half a dozen times already.
Still though, you can choose to ignore TC and stake only on CORR.
I have no idea, how I could aim my model design toward high TC. But if my model somehow hits positive TC, I’ll respond to Numerai’s incentive signal and stake on it.
2 Likes
Ensembles only thrive with diverse components. TC will certainly incentivize more diversity. It is just a question of how much accuracy is given up (component-wise) to get it. First, let’s assume that a large portion of modellers switch to striving primarily for TC (which may not even happen). Even if they do, it is quite possible an ensemble based (largely) on TC feedback will turn out to be about the same as the one that was based on CORR/MMC feedback (even though the components will be quite different than before). Or it could be a lot better, or modestly better, or worse. We’ll just have to see. Given the way TC is made, worse seems unlikely so this is probably a good bet on Numerai’s behalf. But nothing is guaranteed, and if the users hate it then maybe it won’t work out even though it technically should. I imagine with the huge magnitudes TC is capable of paying compared to CORR that it would be a good incentive, but then again the burns can be just as big. In a bad round, you’ll be thankful for a low payoff factor if you are betting on 2x TC. (Your earn/burn is capped by the 0.25 round payoff/burn limit * payoff factor, so if payoff factor is 0.45, what could have been a 25% burn will only be 0.25*0.45 = .1125 which is bad enough.)
On the question of “can you optimize for TC?” In the sense of can you just put TC into a loss function, then no you can’t do that, but that doesn’t mean you are 100% in the dark. You can certainly make educated guesses about the types of methods and niches you could explore that you could reasonably expect not many others to be exploring, i.e. maybe don’t make a vanilla xgboost model if you are shooting for high TC. Although even if you do, you’ll probably be at least positive on TC over time (the integration_test models both have positive TC) – a “normal” straightforward model with fairly high metamodel correlation getting decent CORR scores probably won’t lose (on average) betting 0.5x or 1x TC along with CORR. But to really excel on TC you’re gonna have to do something weirder (and be ok with more volatility in results). If you must absolutely have a definite function to optimize on, FNC3 looks like the one (or make a custom one that is similar). Some high TC models are doing very bad on FNC3 (and CORR), but very few high FNC3 models are getting negative TC so that seems fairly safe. Could be a moving target though…
6 Likes
Totally agree with @wigglemuse , I got 2 models in the top 100 (one currently at nr5), and I remember I put those models out there as a shot in the dark, like just try something bananas 
Guess it works a bit for now, that said it can also be over in a heartbeat, I had another model also top30 TC, and within 1-2 weeks its totally gone (as in bad TC performance on all rounds suddenly). At least I got a bit lucky this time, lol.
2 Likes
To me it seems models that are very sure to not have negative correlations regardless of how miniscule the mean correlation is (< 0.01), tend to get high TC values. My guess is to not optimise for a high mean of correlation, but rather the probability for the correlation to be positive across eras will lead to a high TC value.
1 Like
Looks like a totally random model does quite well with TC, wonder if it’s a fluke or if it’ll keep going.
https://numer.ai/totally_random
8 Likes
It seems that low CORR W/METAMODEL model tent to get high TC.
Top TC models on the leaderboard have very low CORR W/METAMODEL.
This could makes sense, as a random model will have in average 0 CORR with response and 0 CORR with metamodel if you give more importance in TC to uniqueness than correlation the balance will be positive.
This is a sign of overfitting in the TC computation if random noise added to the metamodel improves it.
2 Likes
There are random models and then there are random predictions. It is not weird at all that randomness will have good rounds – but it should also have equally bad rounds. And if it is a random MODEL, meaning that it was randomly created but once created remains the same model, then it is also not weird that it would exhibit streaks of goodness or badness because there is high auto-correlation between round results (at least much of the time). Any random model will settle somewhere over time – most likely right around zero, but a randomly-made model can also be randomly good or bad (but again, not likely to be strongly one thing or another unless it isn’t as random as you thought – some “random” methods make very good models). (And do we have any information about what this example actually is?)
If there is no model and there are just random PREDICTIONS created fresh each round, then the first thing holds but there should be less streakiness involved because now the predictions would have no auto-correlation with themselves. But streaks appear randomly too, and those streaks can be much longer than your intuition would think.
As far as being less correlated with the metamodel leading to greater TC, of course it does, no surprise there and the team has pointed that out repeatedly and has been encouraging more original models. However, this uncorrelation BY ITSELF won’t give you TC riches, but it is more like it is increasing the potential TC capacity of your model, i.e. it still has to be a good model – your potential for high negative TC grows with your potential for high positive TC as you become less correlated with everybody else.
4 Likes
I’ve been doing random predictions for signals for awhile: Numerai
Like wigglemuse said, there are good runs and bad runs. Overall it’s down about 30%
2 Likes
Rightly or wrongly, these tournaments are beginning to look deeply suspect, like so many enterprises based on crypto coins.
Some time back, I was getting really poor 1% on TC. The correlation was more stable but not much better and my Numeraires were getting burnt off at an alarming rate. So, partly as an experiment, I cancelled the stake. As soon as I had done that, my TC shot up to 95%+. So I turned the staking back on and now TC has again fallen back to around 1%. I am not talking about one-off week here but several weeks running in each case. Deliberate or not? I guess we will never know. It is certainly in no way transparent.
3 Likes
Mean reversion. If your NMR is burning at an “alarming” rate you are staking beyond your risk tolerance (and possibly beyond what math would sensibly indicate). When bad periods inevitably arrive, you shouldn’t be panicked – it’s a bad period and it will pass. If you have staked too much (or at too high a multiple) that those periods are going to freak you out and make you pull stakes, then also inevitably you are going to miss the following upswing. This is pretty much a universal pattern in betting/gambling. Strong emotions (up or down) attached to particular outcomes in short time-frames shouldn’t happen – they are a sign of over-staking (psychologically and probably mathematically) – and they lead to hasty emotional decisions.
But I’d just point out this isn’t a casino – there is no motive to rig the game. Numerai has nothing to gain by you losing. They do well when we do well – by extension when you do well. When you burn, do they gain anything? Nope, that means they are suffering also. So…TC is a black box to us, it’s true. (Hopefully that won’t be true forever.) Bugs or incorrect calculations are a possibility (just from the fact that we can’t independently verify). But…“deeply suspect, like so many enterprises based on crypto coins”…that sounds like an implication of deliberate tampering with results. Again, there is no motive, it is contrary to their interests to do that.
4 Likes
Except my money, of course.
It is only alarming in as much as it may take me out of playing in these tournaments.
I am not that stupid to consider NMR as some kind of an investment.
Anyway, the degree of my alarm is not really the main point here.