To me the main problem with this TC is that it makes the participants responsible for something we are not supposed to be responsible for.
Currently we build predictive models for the stock market and numer.ai is responsible for using these predictions to create a profitable trading strategy. This seems a logical separation of responsibilities to me.
TC would make us responsible both for the prediction of the stock market and then on top of that on how they are going to trade those predictions. I don’t think this is something we can reasonable be asked to be responsible for. If for example you look at the outcome of round 295 you can see that NOPAIXX took some of the top spots:
Yet he ended up with negative TCs implying NOPAIXX did not do well/help. I don’t think this is a correct interpretation of what happened. NOPAIXX did ridiculously well, unfortunately it seems that numer.ai’s trading strategy generator did not work well. Then it seems to me that it’s the responsibility of numer.ai to improve their trading strategy, not on NOPAIXX to change his models such that he thinks numer.ai would make better use of them. Which as @degerhan noted might be an impossible feat due to the lack of observability a participant has.
In general I do think numer.ai might have found an important problem. Because as you can see in for example round 295, Numerai, is that the correlation between CORR and TC or MMC and TC seems to be remarkably low. So this does seem something that needs solving, but I don’t believe in the current solutions. I think one of the following directions makes more sense:
- Improve the trading strategy generator
- Change the metric to something we can reasonably optimize for (e.g. classification of top 200/bottom 200 stocks).
- Possibly check if your TC calculations are wrong, if they are there actually might not be a problem.
P.S. I also don’t per se like the way TC and MMC want to measure contribution, by adding a model to the meta model and seeing if that helps. As they believe in using Shapley values for feature selection, (Feature Selection with BorutaShap), why don’t they believe in it for measuring our contributions (exactly where shapley values are intended for Shapley value - Wikipedia).