The more independent bets in a portfolio the better.
A casino in Las Vegas doesn’t need many Roulette wheels to almost never have a losing month because each wheel spin is independent and has an edge that favors the house. But often quant funds all go down in the same month. Clearly the bets of the quant funds are not nearly as independent as the casinos but independence is a very good goal because there is a relationship between your information coefficient (your correlation with the target), the number of independent bets you take and your Sharpe ratio.
Given this relationship, I think broadly people might be too focussed on improving their information coefficient instead of improving the degree of independence in their bets.
How do you measure whether your model trained on Numerai’s data is producing independent bets?
I was thinking about this last year and wrote this note that I wanted to share:
*What I mean by independence is that if you knew the outcome of 99 of the top 100 predictions of your model, would you be able able to say something more about the 100th stock? For example, if you had a predicted score of 0.7 and this was your highest prediction, and you knew that your 99 other scores did very well that era, would you want to bump up your prediction of 0.7 to say 0.75 or would knowing the targets not change your original prediction at all? If you wouldn’t want to change your 0.7 prediction at all then your predictions are independent.
I was thinking about ways to formalize this and came up with:
We call the error vector of a prediction set: (targets - prediction)**2
Richard’s Dependence = cor(1-error, era_score)
If you think of your performance on a prediction as 1-error, then Richard’s Dependence says the more correlation your era_score has to your prediction performance, the more dependent each prediction is on the outcome of your other predictions that month.*
I’ve found this simple measure of independence to be quite useful. For example, it can be used to determine when to stop training a model (when dependence is minimized). It also seems to be a better measure of the riskiness of a model that the standard deviation of correlation scores. By targeting independence when training a model, the model will tend to learn to reduce feature exposure automatically (big feature exposure typically means not a lot of independence). Just like the formula from Grinold & Kahn above, high independence tends to lead to higher Sharpe.
Nevertheless, I’m sure there are better, more statistical approaches to calculate the independence of model predictions. Can anyone propose something better than this with a more statistical basis? Perhaps we can formulate an independence measure we can include in diagnostics on Numerai and Numerai Signals.
Here’s a code sample that prints the dependence of example predictions to the validation targets
sub = pd.read_csv("s3://numerai-public-datasets/latest_numerai_example_predictions_data.csv.xz", index_col=0)
val_data = pd.read_csv("s3://numerai-public-datasets/latest_numerai_validation_data.csv.xz", index_col=0)
val_targets = val_data[['era', 'target']]
df = val_targets.join(sub)
def richards_dependence(df, target_col, era_col, prediction_col):
scores_by_era = df.groupby(era_col).apply(
lambda d: d[[prediction_col, target_col]].corr()[target_col][0]
)
# these need to be ranked within era so "error" makes sense
df[prediction_col] = df.groupby(era_col)[prediction_col].rank(pct=True)
df[target_col] = df.groupby(era_col)[target_col].rank(pct=True)
df["era_score"] = df[era_col].map(scores_by_era)
df["error"] = (df[target_col] - df[prediction_col]) ** 2
df["1-error"] = 1 - df["error"]
# Returns the correlation of the 1-error with the era_score
# i.e. how dependent/correlated each prediction is with its era_score
return df[["1-error", "era_score"]].corr()["era_score"][0]
print(richards_dependence(df, 'target', 'era', 'prediction'))
# 0.022152739462416336
