How often do you re-train your model?

[spn30n]

My model performed well on the first few weeks, and then the performance degraded. I wonder how often should we re-train our model to incorporate the new data.

[anthill]

How sensitive you are to the most recent data may depend on your specific model. Some models do best with up-to-date data, others are looking more for stable signals that don’t change much over time. Models that use more stable signals would ideally have less volatility and so be less susceptible to drawdowns.

But ultimately you can figure out what works best for your model by doing step-forward training. Backtest on the validation data by training on the “freshest” data for a particular validation era, vs. data that is a few weeks old and see which does better.

[bguberfain]

I retrain my model weekly

[svendaj]

I retrain weekly, namely because it’s for free. I am fully automated on Kaggle platform, so it costs me nothing and I can focus on experimentation. If there were some costs related, I would certainly think twice about retraining frequency.

The frequency of retraining is almost like trying to time the market - a futile effort. For example, my best public model JOS_KAGGLE_SUNSHINE Profile - Numerai has been trained 10 months ago (no retraining since) and has reasonable 1Y return of 61.7% and its CORR20 performance is at #152 of the models leaderboard.

[jakasspeech2]

@svendaj

I retrain weekly, namely because it’s for free. I am fully automated on Kaggle platform, so it costs me nothing and I can focus on experimentation. If there were some costs related, I would certainly think twice about retraining frequency.

The frequency of retraining is almost like trying to time the market - a futile effort. For example, my best public model has been trained 10 months ago (no retraining since) and has reasonable 1Y return of 61.7% and its CORR20 performance is at #152 of the models leaderboard.

You could ensemble the static model with the weekly one to balance long term stability with short term adaptation

[gammarat]

When the data or formats are changed enough so that I have to. Numerai seems to change those just about every time I get some model running well enough to put money down on it. On the bright side, every time I have to adapt to new formats/data, I substantially change my approaches, and I do find that entertaining. I’m in the middle of building new ones for each of the competitions,so it’ll be awhile before they’r up and running.

[svendaj]

Some experimental data on the topic: When to Retrain a Model — A Casuistic Analysis - Tournament - Numerai Forum

[digitalforensicfocus]

Thats an awsome rate of return. Where can I subscribe.

[svendaj]

Easy. Here: Numerai