Automated numerai submission with Notebook

hello all, I’d like to share a new discovered platform https://deepnote.com for those who are using jupyter notebook for weekly submission.
The biggest advantage is that one can migrate easily the notebook to the cloud, together with all neccessary model files, scripts, environment variables, etc. All config takes about <5 min.
Once setup, one can schedual the notebook execution with the schedualer.
In terms of pricing, under basic hardware environment (2.5G ram/5G space), it’s all free (it’s adequate for my prediction process). Of course, one can choose to upgrade to performing environment for demanded calculation (like model training) with full transparent hourly fees.
Hopefully this can help people to alleviate the submission with physical machine.

5 Likes

Thanks for sharing. How is your experience after using it for a while ?

It works smoothly since day one. It’s simple and easy to configurate if you are working with notebook.
Once the job is scheduled, the life cannot be easier.

1 Like

@zwk hi, your post coming at right timing. i start to use deepnote to submit my prediction daily for both main and signal. thanks

Indeed! Glad it works well for you! The only missing point for schedular is to have webhook trigger, for which I have submitted a request to their Dev, hopefully can have it soon!

1 Like

Thanks for sharing. It’s very easy to set up deepnote

1 Like

I’m using google colab pro+ which also has a scheduler. It works very well. The only downside is that you need to download all the data and re-train the model every time (you can’t save/download a pickled model, and you need to redownload universe and targets from Numerai server every time you run it).

Files can be downloaded from and saved to your google drive but unfortunately google doesn’t allow you to do that without authenticating and giving permission in person when you start the notebook. Therefore the google drive seems to be inaccessible while using the scheduler.

Also, the google scheduler is limited to 13 days ahead (which is good enough for my purposes).

You may actually be able to auto-mount google drive per Colab notebook. It seems to work for me.

First, you need to create a blank notebook first (don’t copy or save as) that isn’t shared.

Then, mount Google drive using the file browser. After that, it should auto-mount. Not sure how reliable this method is but it seems to work right now.

Not sure if this works with the scheduler though.

1 Like

Thank you so much! I’ve just given it a try and it works

I’ve been mounting programmatically in the start of my programs which necessitated an online approval: