Where do you train your models (if not Google Colab)

[nganser]

As I’m starting to work with the larger datasets, my Google Colab functions are taking hours to run and thus disconnecting me in between when I leave my laptop.

Is there another tool out there that can run a jupyter notebook in the background? What are you using?

Thanks!

[svendaj]

I am using and recommend Kaggle. Not only that you can run for free jupyter notebooks with 4 CPUs and 30GB RAM (max 12 hours run each, 5 notebooks in parallel), but thanks to Kaggle API you can fully automatize your pipeline. On top, there is great community and possibility to learn more from competitions.

I have created a few public notebooks and datasets to simplify first steps for Numerai Tournament participants:

• weekly updated dataset with latest data, so that you do not need to download them each time:
  • V4.3 Midnight - latest data, notebook producing the dataset: numerai data v4.3 Midnight (kaggle.com)
  • V4.2 Rain, notebook producing the dataset: numerai data v4.2 Rain (kaggle.com)
  • Older V4 and V4.1 and notebook producing the dataset: numerai data (kaggle.com)
• Example models - typically forks of Numerai example models with improvements for better results:
  • Hello Numerai automated (kaggle.com), with upload via NumerAPI, model is also staked so you can check its performance on leaderboard ( JOS_KAGGLE_HELLO - Numerai and improved version JOS_KAGGLE_SHATTER - Numerai)
  • numerai Feature Neutralization (kaggle.com) - example model with feature neutralization, JOS_KAGGLE_MEDIUM_FN - Numerai
  • numerai Target Ensemble (kaggle.com) - example model of target ensembling, JOS_KAGGLE_MEDIUM_TE - Numerai
  • Numerai Example Model Sunshine (kaggle.com) - older more complex example model, JOS_KAGGLE_SUNSHINE - Numerai
• Kaggle automation tips:
  • Triggering notebook execution by webhook with Flask and Kaggle API | Kaggle
  • Scheduling notebook execution with cron and Kaggle API | Kaggle
  • Running pipeline of notebooks with Kaggle API

[nganser]

Thank you! I’ll review this!!!

[dfrank]

Thank you! Also a new user exploring ways to train models (turns out my ~10 year old gaming rig can’t handle much past the “small” feature set)