Yes. Couple of gold medals then 1st percentile for TC. simultaneously, some of my secret no-hoper models have gone from mid-range TC performance to massive TC scores. I’m buying some dice and a coin-flipping machine to help with predictions for the up-coming rounds.
I have another possible explanation: The massive TC drop correlates quite well with the time when TC staking was enabled. So probably there was a big change in the models people stake on, because they shifted their stake to their high TC models. This resulted in a big change of the meta model. Since TC is calculated against the meta model, it would make sense that your TC changed a lot.
Yeah markets are quite volatile at the moment and a metric based on portfolio returns (i.e. the returns of a subset of the stocks) is going to be much choppier than correlation to a normalized target across all stocks. With lots of stocks making double digit moves such things are to be expected.
Surely, the actual predictive performance (correlation) is all that should matter.
I can see the usefulness of these additional arbitrary measures for the building of the ‘metamodel’ internally but not for the individual predictors.
I agree that it seems that way… but only Numerai sees the “actual predictive performance”, by taking market positions. And based on what they see, they adjust the incentive package. And re-adjust, about half a dozen times already.
Still though, you can choose to ignore TC and stake only on CORR.
I have no idea, how I could aim my model design toward high TC. But if my model somehow hits positive TC, I’ll respond to Numerai’s incentive signal and stake on it.
Ensembles only thrive with diverse components. TC will certainly incentivize more diversity. It is just a question of how much accuracy is given up (component-wise) to get it. First, let’s assume that a large portion of modellers switch to striving primarily for TC (which may not even happen). Even if they do, it is quite possible an ensemble based (largely) on TC feedback will turn out to be about the same as the one that was based on CORR/MMC feedback (even though the components will be quite different than before). Or it could be a lot better, or modestly better, or worse. We’ll just have to see. Given the way TC is made, worse seems unlikely so this is probably a good bet on Numerai’s behalf. But nothing is guaranteed, and if the users hate it then maybe it won’t work out even though it technically should. I imagine with the huge magnitudes TC is capable of paying compared to CORR that it would be a good incentive, but then again the burns can be just as big. In a bad round, you’ll be thankful for a low payoff factor if you are betting on 2x TC. (Your earn/burn is capped by the 0.25 round payoff/burn limit * payoff factor, so if payoff factor is 0.45, what could have been a 25% burn will only be 0.25*0.45 = .1125 which is bad enough.)
On the question of “can you optimize for TC?” In the sense of can you just put TC into a loss function, then no you can’t do that, but that doesn’t mean you are 100% in the dark. You can certainly make educated guesses about the types of methods and niches you could explore that you could reasonably expect not many others to be exploring, i.e. maybe don’t make a vanilla xgboost model if you are shooting for high TC. Although even if you do, you’ll probably be at least positive on TC over time (the integration_test models both have positive TC) – a “normal” straightforward model with fairly high metamodel correlation getting decent CORR scores probably won’t lose (on average) betting 0.5x or 1x TC along with CORR. But to really excel on TC you’re gonna have to do something weirder (and be ok with more volatility in results). If you must absolutely have a definite function to optimize on, FNC3 looks like the one (or make a custom one that is similar). Some high TC models are doing very bad on FNC3 (and CORR), but very few high FNC3 models are getting negative TC so that seems fairly safe. Could be a moving target though…
Totally agree with @wigglemuse , I got 2 models in the top 100 (one currently at nr5), and I remember I put those models out there as a shot in the dark, like just try something bananas Guess it works a bit for now, that said it can also be over in a heartbeat, I had another model also top30 TC, and within 1-2 weeks its totally gone (as in bad TC performance on all rounds suddenly). At least I got a bit lucky this time, lol.
To me it seems models that are very sure to not have negative correlations regardless of how miniscule the mean correlation is (< 0.01), tend to get high TC values. My guess is to not optimise for a high mean of correlation, but rather the probability for the correlation to be positive across eras will lead to a high TC value.