# Optimizing for FNC and TB scores

**URL:** <https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132>\
**Category:** Tournament\
**Created:** [March 22, 2022, 10:21pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132 "2022-03-22T22:21:59Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![mdo](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/mdo/32/3398_2.png) [@mdo](https://forum.numer.ai/u/mdo)\
**Post date:** [March 22, 2022, 10:21pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/1 "2022-03-22T22:21:59Z")

</div>

With the advent of TC many users may wonder how to optimize for metrics beyond correlation and mean-squared error. Here we will show how to directly optimize for metrics like TB200 and FNC. This is intended to be a proof of concept and source of inspiration, not a set of instructions.

A previous [forum post](http://forum.numer.ai/t/differentiable-spearman-in-pytorch-optimize-for-corr-directly/2287) demonstrated how to optimize for Spearman correlation directly. This work can be extended fairly simply to allow for the optimization of a top/bottom correlation (e.g. TB200) where only the most extreme values of the prediction are used in the correlation function.

```auto
import torch
import pandas as pd
import numpy as np
import torchsort
from torch.distributions import Normal
from torch.functional import F
import torch.optim as optim
from torch import nn

normal = Normal(0,1)

def numerair_tb(pred, target, tb=None, gaussianize=False, regularization_strength=.0001):
    # Computes and returns a differentiable Numerai score with option to use only 
    # the top and bottom tb values. Use the gaussianize option to perform Gauss-rank 
    # instead of just rank transform on predictions
    
    pred = pred.reshape(1, -1)
    target = target.reshape(1, -1)
    
    # get sorted indicies
    rr = torchsort.soft_rank(pred, regularization_strength=regularization_strength)
    
    # change pred to uniform distribution
    pred = (rr - .5)/rr.shape[1]
    
    # convert uniform to gaussian distribution
    if gaussianize:
        pred = normal.icdf(pred)
        
    # select top/bottom indices
    if tb is not None:
        tbidx = torch.bitwise_xor(rr<=tb, rr > (rr.shape[1]-tb))
        pred = pred[tbidx]
        target = target[tbidx]
    
    # Pearson correlation
    pred = pred - pred.mean()
    pred = pred / pred.norm()
    target = target - target.mean()
    target = target / target.norm()
    return (pred * target).sum()

```

If we want to control feature exposure of the top/bottom part of the signal, it can be helpful to have the correlation function return this exposure as well so it can be incorporated into the overall cost function. A modified version of the above to return the total feature exposure:

```auto
import torch
import pandas as pd
import numpy as np
import torchsort
from torch.distributions import Normal
from torch.functional import F
import torch.optim as optim
from torch import nn

normal = Normal(0,1)

def numerai_r_tb_exposure(pred, target, features, tb=None, gaussianize=False, regularization_strength=.0001):
    # Computes and returns a Numerai score and feature exposure
    
    pred = pred.reshape(1, -1)
    target = target.reshape(1, -1)
    
    # get sorted indicies
    rr = torchsort.soft_rank(pred, regularization_strength=regularization_strength)
    # change pred to uniform distribution
    pred = (rr - .5)/rr.shape[1]
    
    # convert uniform to gaussian distribution
    if gaussianize:
        pred = normal.icdf(pred)
        
    # select top/bottom indicies
    if tb is not None:
        tbidx = torch.bitwise_xor(rr<=tb, rr > (rr.shape[1]-tb))
        pred = pred[tbidx]
        target = target[tbidx]
        features = features[tbidx[0]]
    
    # Pearson correlation
    pred = pred - pred.mean()
    pred = pred / pred.norm()
    target = target - target.mean()
    target = target / target.norm()
    
    return (pred * target).sum(), ((pred @ features)**2).sum()

```

We can use the above cost functions to compute CORR and TB scores as well as feature penalty terms. The inclusion of a differentiable version of the psudoinverse in Pytorch, means we can feature-neutralize a model’s predictions and directly optimize for FNC as well. Now we will show how to train a simple neural network on a cost function optimizing for FNC, FNC TB500, CORR, while penalizing feature exposure in the prediction and the top/bottom 500 of the neutralized prediction. (We’ve found TB500 a bit more stable to use for optimization as TB200 tends to overfit easily.) We initialize a simple neural network like:

```auto
class Net(nn.Module):
    def __init__ (self):
        super(Net, self). __init__ ()
        self.lin1 = nn.Linear(1050, 100)
        self.lin2 = nn.Linear(100, 30)
        self.lin3 = nn.Linear(30, 1)
        self.bn = nn.BatchNorm1d(1)
        self.do1 = nn.Dropout(0.5)
        self.do2 = nn.Dropout(0.5)

    def forward(self, x):
        x = self.lin1(x)
        x = self.do1(F.mish(x))
        x = self.lin2(x)
        x = self.do2(F.mish(x))
        output = self.bn(self.lin3(x))
        return output

```

We can then set up a training loop as follows to optimize for this multi-part cost function.

```auto
for epoch in range(epochs):
    np.random.shuffle(era_list)
    for ii, era in enumerate(era_list):
        # get features and target from data and put in tensors
        features = torch.tensor(training_data[training_data.era == era].filter(like='feature').values) - .5
        target = torch.tensor(training_data[training_data.era == era]['target'])

        # zero gradient buffer and get model output
        optimizer.zero_grad()
        model.train()
        output = model(features)

        # neutralize model output
        b = features.pinverse(rcond=1e-6) @ output
        linear_pred = features @ b
        neutralized_output = output - linear_pred

        
        neut_tb_loss, neut_tb_exp = numerai_r_tb_exposure(neutralized_output, target, features, tb=500)
        neut_loss = numerair_tb(neutralized_output, target)
        orig_loss, orig_exp = numerai_r_tb_exposure(output, target, features)
        
        # loss = -tb500 corr for neutralized output - corr for neutralized output - corr + tb500 exposure + exposure
        loss = -neut_tb_loss - neut_loss - orig_loss \
                + neut_tb_exp/1e3 + orig_exp/1e4

        loss.backward()
        optimizer.step()

```

We’ve trained a model using this code and have submitted it [here](https://numer.ai/covid19). The validation statistics for this model are here. Again this is far from optimized and is meant only to show what is possible, but it seems fairly decent already. Cheers and good luck!

 ![image](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/0/0c763b3a4d05e29f8518f38a2072e8e06e561fe1.png)  
 ![image](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/6/626e4dfa703a68596ab1ec704fea5c4e3e054248.png)

---

<div class="post-metadata">

**Author:** ![perfect\_fit](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/perfect_fit/32/2527_2.png) [@perfect\_fit](https://forum.numer.ai/u/perfect_fit)\
**Post date:** [March 23, 2022, 12:03pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/2 "2022-03-23T12:03:23Z")

</div>

Awesome! I suppose the same process can be applied when optimizing for FNCv3? Do I understand correctly that the only difference is the feature set we are neutralizing against?

---

<div class="post-metadata">

**Author:** ![mdo](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/mdo/32/3398_2.png) [@mdo](https://forum.numer.ai/u/mdo)\
**Post date:** [March 23, 2022, 5:16pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/3 "2022-03-23T17:16:33Z")

</div>

Yup that is correct. The old FNC was using the old 310 features.

---

<div class="post-metadata">

**Author:** ![perfect\_fit](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/perfect_fit/32/2527_2.png) [@perfect\_fit](https://forum.numer.ai/u/perfect_fit)\
**Post date:** [March 23, 2022, 7:08pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/4 "2022-03-23T19:08:02Z")

</div>

That’s clear, thank you!

---

<div class="post-metadata">

**Author:** ![bguberfain](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/bguberfain/32/3050_2.png) [@bguberfain](https://forum.numer.ai/u/bguberfain)\
**Post date:** [March 24, 2022, 3:04pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/5 "2022-03-24T15:04:18Z")

</div>

> [@mdo](#):
>
> `for ii, era in enumerate(era_list):`

Thanks for sharing! I was wondering if there is any special meaning on training per era, like if the loss functions only make sense when used this way.

Do you think that a random batch or more than one era per batch would penalize the convergence of the model?

---

<div class="post-metadata">

**Author:** ![mdo](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/mdo/32/3398_2.png) [@mdo](https://forum.numer.ai/u/mdo)\
**Post date:** [March 24, 2022, 3:38pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/6 "2022-03-24T15:38:06Z")

</div>

Feature neutralization makes the most sense on a per-era basis.

---

<div class="post-metadata">

**Author:** ![olivepossum](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/olivepossum/32/2259_2.png) [@olivepossum](https://forum.numer.ai/u/olivepossum)\
**Post date:** [March 24, 2022, 9:44pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/7 "2022-03-24T21:44:00Z")

</div>

What number of features does include FNCv3? And v2 and v1?

Thanks!

---

<div class="post-metadata">

**Author:** ![gbrecht](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gbrecht/32/3509_2.png) [@gbrecht](https://forum.numer.ai/u/gbrecht)\
**Post date:** [March 25, 2022, 7:15am UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/8 "2022-03-25T07:15:48Z")

</div>

FNCv3 is the 420 features of the “medium” featureset they released.  
On RC they announced that code for FNCv3 will be released soon

---

<div class="post-metadata">

**Author:** ![gbrecht](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gbrecht/32/3509_2.png) [@gbrecht](https://forum.numer.ai/u/gbrecht)\
**Post date:** [March 25, 2022, 7:17am UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/9 "2022-03-25T07:17:35Z")

</div>

@mdo What is the purpose of the gaussianize switch? What effect would it have to make the uniform distribution of the prediction a gaussian distribution?

---

<div class="post-metadata">

**Author:** ![olivepossum](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/olivepossum/32/2259_2.png) [@olivepossum](https://forum.numer.ai/u/olivepossum)\
**Post date:** [March 25, 2022, 3:53pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/10 "2022-03-25T15:53:50Z")

</div>

When using validation data for early stopping, does it make sense to do use eras as batches or shouldn’t make a difference there?  
Thanks!

---

<div class="post-metadata">

**Author:** ![gbrecht](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gbrecht/32/3509_2.png) [@gbrecht](https://forum.numer.ai/u/gbrecht)\
**Post date:** [March 25, 2022, 6:36pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/11 "2022-03-25T18:36:10Z")

</div>

You do check for early stopping at the end of an epoch, right?  
If so, I say it is a good idea to use era-batches for training.

---

<div class="post-metadata">

**Author:** ![olivepossum](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/olivepossum/32/2259_2.png) [@olivepossum](https://forum.numer.ai/u/olivepossum)\
**Post date:** [March 25, 2022, 7:25pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/12 "2022-03-25T19:25:36Z")

</div>

> You do check for early stopping at the end of an epoch, right?  
> Yes, I do. At the end of each epoch I use validation data to check for early stopping. My doubt is if I should calculate the total loss on the validation data by using batches per era there or it does not really matter (I would use per era batches with the train data but not for the validation\_data used for early stopping).

---

<div class="post-metadata">

**Author:** ![gbrecht](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gbrecht/32/3509_2.png) [@gbrecht](https://forum.numer.ai/u/gbrecht)\
**Post date:** [March 25, 2022, 7:47pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/13 "2022-03-25T19:47:32Z")

</div>

you calculate the corr score per era but I do not see a reason why you should predict in batches.

---

<div class="post-metadata">

**Author:** ![olivepossum](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/olivepossum/32/2259_2.png) [@olivepossum](https://forum.numer.ai/u/olivepossum)\
**Post date:** [March 25, 2022, 11:49pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/14 "2022-03-25T23:49:03Z")

</div>

My doubt is if at the end of each training epoch, it makes sense to do an early stopping check using validation data with validation eras like this:

```auto

def validation_early_stopping(val_data, model):
    model.eval()
    era_list = eras_validation.unique()
    np.random.shuffle(era_list)
    batch_count = 0
    acc_loss_val = 10000
    
    with torch.no_grad(): 
      for era in era_list:
          batch_count += 1
          # get features and target from data and put in tensors
          features = torch.tensor(val_data[val_data.erano == era].filter(items=feature_names).values) - .5
          target = torch.tensor(val_data[val_data.erano == era]['target'])
          features = features.cuda()
          target = target.cuda()

          output = model(features)
          
          # neutralize model output
          b = features.pinverse(rcond=1e-6) @ output
          linear_pred = features @ b
          neutralized_output = output - linear_pred

          neut_tb_loss, neut_tb_exp = numerai_r_tb_exposure(neutralized_output, target, features, tb=500)
          neut_loss = numerair_tb(neutralized_output, target)
          orig_loss, orig_exp = numerai_r_tb_exposure(output, target, features)
          
          loss = -neut_tb_loss - neut_loss - orig_loss + neut_tb_exp/1e3 + orig_exp/1e4
          
          acc_loss_val += loss

      loss_val = acc_loss_val / batch_count
      return loss_val.item()

```

As we are using TB500, I’m not sure if the size or the composition of the validation batches matters here of if it’s even conceptually correct to check early stopping like this in this case.

---

<div class="post-metadata">

**Author:** ![dzheng1887](https://avatars.discourse-cdn.com/v4/letter/d/f475e1/32.png) [@dzheng1887](https://forum.numer.ai/u/dzheng1887)\
**Post date:** [March 26, 2022, 9:56pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/15 "2022-03-26T21:56:49Z")

</div>

Would there be an update soon for the numerai tournament? I wasn’t sure if something had already changed but I did not notice. I am not sure where to go for big notices like that.

---

<div class="post-metadata">

**Author:** ![pumplerod](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/pumplerod/32/3553_2.png) [@pumplerod](https://forum.numer.ai/u/pumplerod)\
**Post date:** [March 28, 2022, 12:25am UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/16 "2022-03-28T00:25:27Z")

</div>

Perhaps I misunderstand the meaning for TB500. I believe that to reference the Top/Bottom 500 prediction values. This is indeed a smaller subset than the full era, however in the code it looks like the TB500 samples are being used in their neutralized form as an addition to the full set of sample losses. Effectively adding extra pressure to the top and bottom 500 to improve performance.

Unless I’m reading this code incorrectly ( very possible), the `neut_loss` ( effectively the corr for the entire set of neutralized predictions) and the `orig_loss` ( corr for the entire set of raw predictions) are being maximized due to the “-” when they are included in the final `loss` calculation. This is also where the additional loss from the tb500 are included.

If I’m reading this wrong however, I would love a clear breakdown of the process.

---

<div class="post-metadata">

**Author:** ![mdo](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/mdo/32/3398_2.png) [@mdo](https://forum.numer.ai/u/mdo)\
**Post date:** [March 28, 2022, 7:11pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/17 "2022-03-28T19:11:31Z")

</div>

Sounds like you’ve got it!

---

<div class="post-metadata">

**Author:** ![pumplerod](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/pumplerod/32/3553_2.png) [@pumplerod](https://forum.numer.ai/u/pumplerod)\
**Post date:** [April 5, 2022, 4:42pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/18 "2022-04-05T16:42:21Z")

</div>

@mdo if Numerai is performing Feature Neutralization on our predictions before TC calculations, would it not help to know which features the team is using to neutralize with? As Numerai is not aware of which features we may have used to generate predictions, how have they determined the best features to use in neutralization? Or do they use them all?

I see above a mention:

> FNCv3 is the 420 features of the “medium” featureset they released.  
> On RC they announced that code for FNCv3 will be released soon

is this information public somewhere? I don’t find it on ‘Neutralization’ section of the Docs.

---

<div class="post-metadata">

**Author:** ![mdo](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/mdo/32/3398_2.png) [@mdo](https://forum.numer.ai/u/mdo)\
**Post date:** [April 6, 2022, 11:55pm UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/19 "2022-04-06T23:55:31Z")

</div>

Predictions are **not** neutralized before TC calculations, just Gauss-rank transformed.

---

<div class="post-metadata">

**Author:** ![wigglemuse](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/wigglemuse/32/3094_2.png) [@wigglemuse](https://forum.numer.ai/u/wigglemuse)\
**Post date:** [April 7, 2022, 3:36am UTC](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132/20 "2022-04-07T03:36:04Z")

</div>

Isn’t neutralization part of the optimizer though?

[Next page](https://forum.numer.ai/t/optimizing-for-fnc-and-tb-scores/5132.md?page=2)