I have a V3 model performing very well a weeks ago respect to TC.
In last three round its TC is lower than percentile 5.
But is strange as the model itself seems not bad:
These are the percentiles:
CORR 87.5
MMC 81.7
FNC 92.3
FNCv3 90.5
And the magic of TC: -0.0854 (Percentile 2.2).
Very dissapointed with TC as seems this is like a lottery.
Someone with similar experience?
Yes. Couple of gold medals then 1st percentile for TC. simultaneously, some of my secret no-hoper models have gone from mid-range TC performance to massive TC scores. Iâm buying some dice and a coin-flipping machine to help with predictions for the up-coming rounds.
I am in the same situation. I posted the same information today ( Why are my TC scores worse after V4 data? Is volatility with a high TC score good for participants? ). If you have time, please take a look through it. It is the exact same situation.
To be honest, I too am disappointed at the TC score. I donât think a score that fluctuates so much is a legitimate metric.
I have another possible explanation: The massive TC drop correlates quite well with the time when TC staking was enabled. So probably there was a big change in the models people stake on, because they shifted their stake to their high TC models. This resulted in a big change of the meta model. Since TC is calculated against the meta model, it would make sense that your TC changed a lot.
and yet other models, from a quick perusal, would not seem to have suffered the same fate.
It seems eleven_sigmaâs model is good. Hard to imagine it having a negative contribution.
No negative, Percentile 2.2 of TC.
-0.0854 is your TC score, 2.2 is TC Percentile
And this is other model:
having
5.1 PCT CORR
6.6 PCT MMC
7.2 PCT FNC
38.1 PCT FNCV3
The magic of TC: 0.1646 with PCT 99.99
Itâs a JOKE
@mdo are you really confident in gradient is measuring what we supposed?
Could you please do a fast and clean test? Check if the gradient is working well.
Compute the metamodel without this last model (with 0.1646 of TC) and see how much
the metamodel is worst.
Yeah markets are quite volatile at the moment and a metric based on portfolio returns (i.e. the returns of a subset of the stocks) is going to be much choppier than correlation to a normalized target across all stocks. With lots of stocks making double digit moves such things are to be expected.
ditto.
__ __ _ ____ _ __
__ / /__ / / ___ ___ __ __ | | /| / / / (_)__ ___ ___ / /_
/ // / _ \/ _ \/ _ \/ _ \/ // / | |/ |/ / _ \/ / _ \/ _ \/ -_) __/
\___/\___/_//_/_//_/_//_/\_, / |__/|__/_//_/_/ .__/ .__/\__/\__/
/___/ /_/ /_/
Surely, the actual predictive performance (correlation) is all that should matter.
I can see the usefulness of these additional arbitrary measures for the building of the âmetamodelâ internally but not for the individual predictors.
I agree that it seems that way⌠but only Numerai sees the âactual predictive performanceâ, by taking market positions. And based on what they see, they adjust the incentive package. And re-adjust, about half a dozen times already.
Still though, you can choose to ignore TC and stake only on CORR.
I have no idea, how I could aim my model design toward high TC. But if my model somehow hits positive TC, Iâll respond to Numeraiâs incentive signal and stake on it.
Ensembles only thrive with diverse components. TC will certainly incentivize more diversity. It is just a question of how much accuracy is given up (component-wise) to get it. First, letâs assume that a large portion of modellers switch to striving primarily for TC (which may not even happen). Even if they do, it is quite possible an ensemble based (largely) on TC feedback will turn out to be about the same as the one that was based on CORR/MMC feedback (even though the components will be quite different than before). Or it could be a lot better, or modestly better, or worse. Weâll just have to see. Given the way TC is made, worse seems unlikely so this is probably a good bet on Numeraiâs behalf. But nothing is guaranteed, and if the users hate it then maybe it wonât work out even though it technically should. I imagine with the huge magnitudes TC is capable of paying compared to CORR that it would be a good incentive, but then again the burns can be just as big. In a bad round, youâll be thankful for a low payoff factor if you are betting on 2x TC. (Your earn/burn is capped by the 0.25 round payoff/burn limit * payoff factor, so if payoff factor is 0.45, what could have been a 25% burn will only be 0.25*0.45 = .1125 which is bad enough.)
On the question of âcan you optimize for TC?â In the sense of can you just put TC into a loss function, then no you canât do that, but that doesnât mean you are 100% in the dark. You can certainly make educated guesses about the types of methods and niches you could explore that you could reasonably expect not many others to be exploring, i.e. maybe donât make a vanilla xgboost model if you are shooting for high TC. Although even if you do, youâll probably be at least positive on TC over time (the integration_test models both have positive TC) â a ânormalâ straightforward model with fairly high metamodel correlation getting decent CORR scores probably wonât lose (on average) betting 0.5x or 1x TC along with CORR. But to really excel on TC youâre gonna have to do something weirder (and be ok with more volatility in results). If you must absolutely have a definite function to optimize on, FNC3 looks like the one (or make a custom one that is similar). Some high TC models are doing very bad on FNC3 (and CORR), but very few high FNC3 models are getting negative TC so that seems fairly safe. Could be a moving target thoughâŚ
Totally agree with @wigglemuse , I got 2 models in the top 100 (one currently at nr5), and I remember I put those models out there as a shot in the dark, like just try something bananas 
Guess it works a bit for now, that said it can also be over in a heartbeat, I had another model also top30 TC, and within 1-2 weeks its totally gone (as in bad TC performance on all rounds suddenly). At least I got a bit lucky this time, lol.
To me it seems models that are very sure to not have negative correlations regardless of how miniscule the mean correlation is (< 0.01), tend to get high TC values. My guess is to not optimise for a high mean of correlation, but rather the probability for the correlation to be positive across eras will lead to a high TC value.
Looks like a totally random model does quite well with TC, wonder if itâs a fluke or if itâll keep going.
https://numer.ai/totally_random