What is your submission automation cost?

Tournament
5

Don’t you update trained model with updated dataset?

6

I have a lot of older models that I’m afraid to retrain.

1 Like
7

@jacob_stahl @ryo_matsuzaka

I have published an article describing “How I automated my Numerai weekly submission pipeline for free, using Azure functions and python”.

I think you will find it useful!

8 Likes
8

Thank you very much for the information! I absolutely check it.

9

I use Deepnote. I’m able to submit daily predictions using their notebook scheduler feature.

3 Likes
10

Thats a thing?
d63c1ef766f4d12f14127148d3c0e45a

11

But now the submission changed to email triggered, how the schedule will work ?

12

My submission automation cost it around 0.6 €/Month on Azure. It runs on a container instance that loads the models from a cloud storage, predicts and stores the results on the same cloud storage until the next round. It runs very fast since it only predicts using live data, so no training is performed.It takes the container about 90 seconds to run my 12 models.
My whole cloud infrastructure consist in the container instance, the cloud storage, one http trigger for the submissions, one http trigger for numerbay sales and two scheduled triggers, one for daily rounds and one for weekend rounds.
The only small detail is that my container image is hosted on a free private repository outside Azure, so that cost it is not accounted for. Actually, that image repository would be the most expensive resource, costing about 4.2€ per month. Although I would not be surprised if there are less expensive alternatives out there.

Edit: I forgot to mention that this represents the cost of daily and weekly predictions. Since this thread is relatively old, maybe I should remark that. I am bad at predictions, but been smoothly predicting daily since day 1 of daily predictions :sunglasses: :stuck_out_tongue_winking_eye:

image
image707×266 17.5 KB

3 Likes
13

image
image769×528 34.2 KB

I have some non-Numerai data in S3 as well, so compute for one model is less than this.

1 Like
14

I am using Kaggle for training and for submissions, because they provide for free notebooks with 4 CPU cores and 30GB RAM. Their notebook scheduling capability is not usable for numerai submissions, so I have created workaround solution with (always :wink:) free Oracle Cloud compute node, running Kaggle public API and cron trigerring my numerai notebooks at announced round openning times. Inside of my notebooks I wait for actual round openning.
To sum it up: I remain at big fat zero automation costs :nerd_face:

15

thanks for the sharing. any idea to go further, to get kaggle notebook be trigger by an external API call ? like web hooks, or something else ?

@svendaj hi, i just had a thought to include your round readiness checking script in my notebook. and still leverage deepnote to run the notebook at every evening 9pm and check the round open status every 10mins. I think i will resolve the daily submission now. Again. thanks for the shasing.

1 Like
16

just tried script: napi.check_round_open() in deepnote.

And realize that deepnote gives following error: AttributeError: ‘NumerAPI’ object has no attribute ‘check_round_open’

Same code run in colab is working fine.

Any one encounter this issue before in deepnote ?

17

Sorry for later reply, so far I was working on running my notebook swarm in Kaggle. Today I have published Kaggle notebook which can create and control complex pipeline of other notebooks.

This was needed for my Saturday night numerai fever, when I was manually running 18 notebooks. Starting with weekly data update, immediately followed by 9 notebooks and 3 sub-pipelines with models to be retrained on weekly data. When this was ready (usually Sunday morning) I launched final stacking notebook.

Now I am able to put everything to one master notebook and launch it with cron from cloud automatically :sunglasses:

Webhooks will be my next step, but not sure when I will have some time to have a look at it.

1 Like
18

thanks for the sharing - “one ring rule them all”

1 Like
19

Would you mind enlightening on how you got the notebook to make submissions on Deepnote? What I really mean is getting the auto-submission to work. Using the Numerai submission API gives really funny errors

20

I have had exactly the same problem only this is in VSC on my computer. This must be something to do with the api. If you manage to solve the problem, please let me know.

21

update the numerapi package

22

I have finally completed solution which will launch my Kaggle submission and training notebooks with Numerai compute webhooks. My previous solution with cron scheduled execution of notebooks and waiting loop for round opening was sufficient, but on Saturday when I am running also training tasks, I sometimes got beyond 12h run limit of Kaggle notebook (because of long waiting for round opening).

It is based on python Flask server - easy to implement in various environments. I am using free Oracle Cloud Ubuntu VM instance.

So now I am completely switching to “compute” style automation - triggered by webhook.

23

Hi @svendaj why do you use a kaggle notebook instead of putting it all into the oracle instance?

24

I am a cheap guy :wink: so when something is for free, it’s the winner. I could not have 4 cores and 30GB RAM in always free tier of OCI. Plus, I have started in Kaggle about year ago and now I have 20 models in roughly the same number of notebooks, so it would not be trivial to run it elsewhere.

2 Likes