I’m afraid it’s still unclear to me where the advantage of unique (or strictly ordered) predictions lies. Perhaps I don’t quite understand what we are trying to predict.
To try to flesh out my question, assuming the
numerai_score is indeed calculated as:
rank_pred = y_pred.groupby(eras).apply(lambda x: x.rank(pct=True, method="first"))
We are trying to find the pearson product-moment correlation coefficients between the target and our prediction. To look at a concrete example. In the numerai example data, 400 rows of target in the first era look like this:
and the prediction of a linear regression looks like this:
which after ranking becomes:
We then calculate the correlation.
Why wouldn’t a perfect discrete prediction that matches exactly the target work best? Let’s take a toy example where my model is god-like and predicts the output perfectly:
t = numpy.array([0,0.25,0.5,0.25,0.25,0.75,1]) # target
pre = numpy.array([0,0.25,0.5,0.25,0.25,0.75,1]) # prediction
If I rank and calculate correlations with:
pre_rank = pandas.DataFrame(pre).apply(lambda x: x.rank(pct=True, method="first"))
I get a
numerai_score of 0.95:
If I instead construct a prediction which is directionally correct (goes up and down when the target does), but has repeated values (has ties) I still get the same score:
If I remove the ties in the prediction, nothing changes in the score. For example, using
pre = numpy.array([0,0.2, 0.4, 0.3, 0.35, 0.6, 0.8])
Equally, introducing errors in the 3 cases above (perfect prediction, monotonically correct prediction with ties, monotonically correct prediction without ties) seems to give equal scores.
With that little exploration, I’m back to my original question: Why can’t the submission look like the target, … in other words, a column with just the 5 possible values between 0 and 1? It seems the
numerai_score in the documentation would allow it.
I would appreciate any insights you may have to help me what I am missing. Thank you.