# Independence and Sharpe

**URL:** <https://forum.numer.ai/t/independence-and-sharpe/2560>\
**Category:** Data Science\
**Created:** [March 27, 2021, 9:03pm UTC](https://forum.numer.ai/t/independence-and-sharpe/2560 "2021-03-27T21:03:02Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![richai](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/richai/32/2493_2.png) [@richai](https://forum.numer.ai/u/richai)\
**Post date:** [March 27, 2021, 9:03pm UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/1 "2021-03-27T21:03:02Z")

</div>

The more independent bets in a portfolio the better.

A casino in Las Vegas doesn’t need many Roulette wheels to almost never have a losing month because each wheel spin is independent and has an edge that favors the house. But often quant funds all go down in the same month. Clearly the bets of the quant funds are not nearly as independent as the casinos but independence is a very good goal because there is a relationship between your information coefficient (your correlation with the target), the number of independent bets you take and your Sharpe ratio.

 ![Screen Shot 2021-03-27 at 1.32.07 PM](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/1X/43c2e0cae672dea91330ec07383f52002ae48090.png)

Given this relationship, I think broadly people might be too focussed on improving their information coefficient instead of improving the degree of independence in their bets.

How do you measure whether your model trained on Numerai’s data is producing independent bets?

I was thinking about this last year and wrote this note that I wanted to share:

\*What I mean by independence is that if you knew the outcome of 99 of the top 100 predictions of your model, would you be able able to say something more about the 100th stock? For example, if you had a predicted score of 0.7 and this was your highest prediction, and you knew that your 99 other scores did very well that era, would you want to bump up your prediction of 0.7 to say 0.75 or would knowing the targets not change your original prediction at all? If you wouldn’t want to change your 0.7 prediction at all then your predictions are independent.

I was thinking about ways to formalize this and came up with:

We call the error vector of a prediction set: (targets - prediction)\*\*2

Richard’s Dependence = cor(1-error, era\_score)

If you think of your performance on a prediction as 1-error, then Richard’s Dependence says the more correlation your era\_score has to your prediction performance, the more dependent each prediction is on the outcome of your other predictions that month.\*

I’ve found this simple measure of independence to be quite useful. For example, it can be used to determine when to stop training a model (when dependence is minimized). It also seems to be a better measure of the riskiness of a model that the standard deviation of correlation scores. By targeting independence when training a model, the model will tend to learn to reduce feature exposure automatically (big feature exposure typically means not a lot of independence). Just like the formula from Grinold & Kahn above, high independence tends to lead to higher Sharpe.

Nevertheless, I’m sure there are better, more statistical approaches to calculate the independence of model predictions. Can anyone propose something better than this with a more statistical basis? Perhaps we can formulate an independence measure we can include in diagnostics on Numerai and Numerai Signals.

Here’s a code sample that prints the dependence of example predictions to the validation targets

```auto
sub = pd.read_csv("s3://numerai-public-datasets/latest_numerai_example_predictions_data.csv.xz", index_col=0)
val_data = pd.read_csv("s3://numerai-public-datasets/latest_numerai_validation_data.csv.xz", index_col=0)

val_targets = val_data[['era', 'target']]
df = val_targets.join(sub)

def richards_dependence(df, target_col, era_col, prediction_col):    
    scores_by_era = df.groupby(era_col).apply(
        lambda d: d[[prediction_col, target_col]].corr()[target_col][0]
    )

    # these need to be ranked within era so "error" makes sense
    df[prediction_col] = df.groupby(era_col)[prediction_col].rank(pct=True)
    df[target_col] = df.groupby(era_col)[target_col].rank(pct=True)

    df["era_score"] = df[era_col].map(scores_by_era)

    df["error"] = (df[target_col] - df[prediction_col]) ** 2
    df["1-error"] = 1 - df["error"]

    # Returns the correlation of the 1-error with the era_score
    # i.e. how dependent/correlated each prediction is with its era_score
    return df[["1-error", "era_score"]].corr()["era_score"][0]

print(richards_dependence(df, 'target', 'era', 'prediction'))
# 0.022152739462416336

```

---

<div class="post-metadata">

**Author:** ![joakim](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/joakim/32/3442_2.png) [@joakim](https://forum.numer.ai/u/joakim)\
**Post date:** [March 27, 2021, 10:13pm UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/2 "2021-03-27T22:13:02Z")

</div>

In short, if I have a tiny edge I’d like to place as many independent bets as possible.

Grinold & Kahn last year released a supplemental version to their classic: [https://www.amazon.com/Advances-Active-Portfolio-Management-Econometrics/dp/1260453715](https://www.amazon.com/Advances-Active-Portfolio-Management-Econometrics/dp/1260453715)

I’m still going through it…

---

<div class="post-metadata">

**Author:** ![gbrecht](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gbrecht/32/3509_2.png) [@gbrecht](https://forum.numer.ai/u/gbrecht)\
**Post date:** [March 27, 2021, 10:54pm UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/3 "2021-03-27T22:54:52Z")

</div>

quick question to the code, specifically the `'DATE'` and `mm` keys in the input DataFrame and era scores:  
Does `mm` stand for metamodel?  
Does the `'DATE'` refer to predictions done in the rounds before or the month `scores_by_era[i]`?

---

<div class="post-metadata">

**Author:** ![richai](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/richai/32/2493_2.png) [@richai](https://forum.numer.ai/u/richai)\
**Post date:** [March 27, 2021, 11:46pm UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/4 "2021-03-27T23:46:07Z")

</div>

good question – edited the code to say “era”. this is from some internal code where we used “DATE” instead and “mm” maybe did mean a meta model I was testing… I think you could replace that with `prediction_col` and it should work so I made that edit now too. @_liamhz says he’ll make a cleaner version if that helps.

---

<div class="post-metadata">

**Author:** ![\_liamhz](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/_liamhz/32/1368_2.png) [@\_liamhz](https://forum.numer.ai/u/_liamhz)\
**Post date:** [March 28, 2021, 12:38am UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/5 "2021-03-28T00:38:20Z")

</div>

I’ve edited Richard’s post with a more clear standalone code sample

---

<div class="post-metadata">

**Author:** ![\_liamhz](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/_liamhz/32/1368_2.png) [@\_liamhz](https://forum.numer.ai/u/_liamhz)\
**Post date:** [March 28, 2021, 3:52am UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/6 "2021-03-28T03:52:06Z")

</div>

Here’s the source of the Sharpe ratio image

> **[Muller\_Proprietary\_trading.pdf](https://faculty.fuqua.duke.edu/~charvey/Teaching/BA453_2004/Muller_Proprietary_trading.pdf)**
>
> 928.05 KB

---

<div class="post-metadata">

**Author:** ![jorijnsmit](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/jorijnsmit/32/2305_2.png) [@jorijnsmit](https://forum.numer.ai/u/jorijnsmit)\
**Post date:** [March 28, 2021, 8:52am UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/7 "2021-03-28T08:52:41Z")

</div>

You write:

> We call the error vector of a prediction set: (targets - prediction)\*2

You code:

```auto
df["error"] = (df[target_col] - df[prediction_col])
df["error"] = df["error"] * df["error"]

```

I believe the \*2 should be \*\*2? Are you squaring the error to get rid of signage? What effect would square rooting the error afterwards have (basically implementing RMSE)? Or are you doing this on purpose to punish bigger errors stronger?

---

<div class="post-metadata">

**Author:** ![gbrecht](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gbrecht/32/3509_2.png) [@gbrecht](https://forum.numer.ai/u/gbrecht)\
**Post date:** [March 28, 2021, 9:19am UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/8 "2021-03-28T09:19:19Z")

</div>

Thank you very much.  
As a “scholar of @arbitrage” (regTM) I find the part about digging deep vs. searching wide interesting, because of its translation to our classic tournament efforts: It means that it is better to look for one good learner than to ensemble weak learners. The reasons is that the correlation may not be as low as we think. Ensembles work very well in the classic tournament. Does that mean  
a) @mdo put together such a good target that it is easy for us to find uncorrelated assets?  
b) the bets of the fund are so long term and wide that the correlation transports well?  
c) there is a mismatch between live corr and fund performance (hopefully not!)?

---

<div class="post-metadata">

**Author:** ![gbrecht](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gbrecht/32/3509_2.png) [@gbrecht](https://forum.numer.ai/u/gbrecht)\
**Post date:** [March 28, 2021, 10:17am UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/9 "2021-03-28T10:17:02Z")

</div>

You mean instead of:

> [@jorijnsmit](#):
>
> `df["error"] = df["error"] * df["error"]`

it should read:  
`df["error"] = df["error"] ** df["error"]`

that would translate error^2 to error^(error)

---

<div class="post-metadata">

**Author:** ![jorijnsmit](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/jorijnsmit/32/2305_2.png) [@jorijnsmit](https://forum.numer.ai/u/jorijnsmit)\
**Post date:** [March 28, 2021, 12:01pm UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/10 "2021-03-28T12:01:20Z")

</div>

No I believe the code is correct, it’s just that in the initial quote it is written as

> (targets - prediction)\*2

Anyway that I believe is just a typo.

My real question is why not square root the error afterwards again?

---

<div class="post-metadata">

**Author:** ![orbitalteapot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/orbitalteapot/32/2403_2.png) [@orbitalteapot](https://forum.numer.ai/u/orbitalteapot)\
**Post date:** [March 28, 2021, 2:08pm UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/11 "2021-03-28T14:08:33Z")

</div>

> lambda d: d[[prediction\_col, target\_col]].corr(method=‘spearman’)[target\_col][0]

In this line, need to add method=‘spearman’ to recreate Richard’s results.

If the function is applied several times on the same df, it doesn’t matter due to the mutation to ranks later on in the function.

---

<div class="post-metadata">

**Author:** ![gbrecht](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gbrecht/32/3509_2.png) [@gbrecht](https://forum.numer.ai/u/gbrecht)\
**Post date:** [March 28, 2021, 2:40pm UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/12 "2021-03-28T14:40:25Z")

</div>

With other metrics like MSE and RMSE for regression, the difference is that without rooting outliers are penalized harder.

---

<div class="post-metadata">

**Author:** ![richai](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/richai/32/2493_2.png) [@richai](https://forum.numer.ai/u/richai)\
**Post date:** [March 28, 2021, 6:23pm UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/13 "2021-03-28T18:23:28Z")

</div>

yes when thinking about targets we do look at how much dependence there is. if you remember our old target “Bernie”, it had much higher dependence and models didn’t generalize nearly as well on live data because of that.

---

<div class="post-metadata">

**Author:** ![richai](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/richai/32/2493_2.png) [@richai](https://forum.numer.ai/u/richai)\
**Post date:** [March 28, 2021, 6:38pm UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/14 "2021-03-28T18:38:20Z")

</div>

@MikeP was talking to me last night about giving an example of a model which has low Standard Deviation of correlations on the validation set (SD) but actually has high Dependence — such an example would show how this really is a different measure of risk. I think there is probably some feature that if you submit the feature as your prediction it has a good Sharpe (low standard deviation) but has very high dependence, which is a nice warning sign that even if it has high Sharpe on validation it’s probably not likely to work well out of sample because of it’s high dependence. Would be nice if someone could find such a feature and post the result here 🙂

---

<div class="post-metadata">

**Author:** ![jorijnsmit](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/jorijnsmit/32/2305_2.png) [@jorijnsmit](https://forum.numer.ai/u/jorijnsmit)\
**Post date:** [March 31, 2021, 8:33am UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/15 "2021-03-31T08:33:59Z")

</div>

Just to deepen my understanding, why does adding `method='spearman'` matter? We already ranked `prediction_col` and `target_col` in order to calculate `error` properly, so it shouldn’t matter right? I do still get a very small difference (maybe just rounding?):

```auto
0.021525909978845296
0.021525909978845223

```

Oh and I found one more typo, `df.era.map(scores_by_era)` should be `df['era_col'].map(scores_by_era)`.

---

<div class="post-metadata">

**Author:** ![profricecake](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/profricecake/32/2528_2.png) [@profricecake](https://forum.numer.ai/u/profricecake)\
**Post date:** [April 1, 2021, 7:34am UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/16 "2021-04-01T07:34:35Z")

</div>

> [@richai](#):
>
> I think there is probably some feature that if you submit the feature as your prediction it has a good Sharpe (low standard deviation) but has very high dependence, which is a nice warning sign that even if it has high Sharpe on validation it’s probably not likely to work well out of sample because of it’s high dependence. Would be nice if someone could find such a feature and post the result here 🙂

I ran through the full feature list but didn’t find any that I’d think of as having a particularly high Sharpe.

For reference, the best I found was `feature_charisma74` with a validation Sharpe of `.47`.

I tried inverting the features too, just for kicks. The best performer was `1.0 - feature_constitution100` with a validation Sharpe of `.52`.

Do you consider either of those high enough to be representative of what you were discussing, @richai ?

---

<div class="post-metadata">

**Author:** ![orbitalteapot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/orbitalteapot/32/2403_2.png) [@orbitalteapot](https://forum.numer.ai/u/orbitalteapot)\
**Post date:** [April 2, 2021, 2:08pm UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/17 "2021-04-02T14:08:58Z")

</div>

The ranking takes place after the score\_by\_era calculation.

In order to recreate the result (0.022152739462416336) in OP, ranking needs to be performed before the score\_by\_era calculation or spearman needs to be used.

When performing the richards\_dependence function twice, you’d get the OP’s result the second time. The results I get on two consecutive runs.

```
0.021851763462089693
0.022152739462416336

```

The reason behind this is because pandas is pass-by-reference and not pass-by-value. Thus these two lines overwrite the original values of prediction\_col and target\_col.

```
df[prediction_col] = df.groupby(era_col)[prediction_col].rank(pct=True)
df[target_col] = df.groupby(era_col)[target_col].rank(pct=True)
```

---

<div class="post-metadata">

**Author:** ![richai](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/richai/32/2493_2.png) [@richai](https://forum.numer.ai/u/richai)\
**Post date:** [April 5, 2021, 10:01pm UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/18 "2021-04-05T22:01:08Z")

</div>

Definitely high enough. And did you try calculate their dependence?

---

<div class="post-metadata">

**Author:** ![profricecake](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/profricecake/32/2528_2.png) [@profricecake](https://forum.numer.ai/u/profricecake)\
**Post date:** [April 6, 2021, 3:08am UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/19 "2021-04-06T03:08:26Z")

</div>

@richai Here are the validation Sharpes and “Richard’s dependence” values for both features:

```
feature_charisma74 vsharpe: 0.47 rdep: 0.0158
1.0 - feature_constitution100 vsharpe: 0.52 rdep: 0.0100
```

---

<div class="post-metadata">

**Author:** ![thekizoch](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/thekizoch/32/1403_2.png) [@thekizoch](https://forum.numer.ai/u/thekizoch)\
**Post date:** [November 22, 2021, 7:02am UTC](https://forum.numer.ai/t/independence-and-sharpe/2560/20 "2021-11-22T07:02:55Z")

</div>

What was the end result of this discussion? Does a finalized form of measuring independence get returned in the validation\_metrics() function in the (basic) example script or is this still up to the data scientist to code Richard’s dependence into his/her code?

Thanks in advance

[Next page](https://forum.numer.ai/t/independence-and-sharpe/2560.md?page=2)