I have just realized that since the introduction of TC my desire to experiment and try new things out is slowing down.
I have still many ideas, e.g. training on multiple targets, but knowing that I cannot compute TC for my models makes me carefully ponder every new development because I cannot access its quality (being good on corr it’s not enough for me anymore). When I want to evaluate a new model I need to create a tournament test entry and wait few months before being able to get a sense of the TC performance. Not only that is boring and slow, it is also wrong to access the TC on few recent entries only.
In the long run, I believe numerai will move away from corr and will focus only on a metric that make sense for their portfolio. That metric could be TC or something else, but they need to allow users to evaluate that metric during the R&D phase. They are probably already thinking at a solution, otherwise the research of new ideas will be negatively impacted.
Why am I writing this? Just for fun and because I am curious to hear what other users think.
It is not slowing me down from burning holes in my cpus running new things all the time as usual, but it is slowing me down from actually staking on them yeah.
It slows me down because it takes 2wks to see the first live TC and much longer needed to tell confidence, due to the lack of TC for late subs and lack of diagnostic tools.
I’ve mentioned in RocketChat: the only ones who can research methodically for TC (with good confidence) are the ones who already have models with good TC because they have long history of backfill TCs and just need to improve on those models. For the majority who are not so lucky including myself, it’s pretty much a shot in the dark and there’s no way to tell if a new model is good on TC until it has a (very) long history. So I only stake 0.5-1xTC on very few new models without much confidence.
TC has actually spurred me to try many more crazy experiments. I was only using 1 model slot before TC, now I am at 34.
Instead of grinding huge ensembles trying to get more bits of CORR, I’m now rapidly rolling out the most diverse set of models that I can think of in the hopes of getting slices of the TC pie. It’s like panning for gold I suppose.
But yeah, I definitely agree with you that chasing recent TC performance is a big problem.
its basically like finding needles in a haystack, but before with corr you could use a metal detector (metrics for validation) to find them more quickly. Basically either we need that metal detector again or we need to increase the number of needles, so give us more slots pls ;~)
No slow down. Just ignore and don’t stake on it.
100 %, I am at least trying to optimize the metrics that are suppose to be correlated with good TC, but
it is only getting worse. My best model at TC is the stupidest one that is trained on 1/5 of the training dataset.
Totally agree. After TC I have a lot of ideas parked. If I haven’t any way to check myself if a model is better or no I have no incentive to research.
In my opinion, TC has some problems that is need to change, so I don’t want to study it at the moment too.
I second @qeintelligence , it is more the running out of model slots that is slowing me down - ideally, I want to have at least 500 slots - and probably run 5-10 variant on each base model that I can come up with