Help us improve Numerai Compute!

Feedback
42

I second @taori please keep the webhooks or at least a way to make external custom calls outside of aws.

43

Rest assured, we are keeping webhooks. The new Compute setup is not mandatory and never will be. Our goal with Compute Lite is to get as many people as possible automated and our initial offering relies on AWS. For users that are already automated, they can keep their existing setup whether it uses Compute or not.

5 Likes
44

Ah, I was worrying for no reason then. Thank you for clarifying this.

45

Compute should support the deployment and ensembling of multiple models!
Also it would be beneficial, if it could support reading model data from S3.

Building and reuploading a docker container with several models is very slow even with a good internet connection.

2 Likes
46
  • Are your submissions automated now?
    Yes
  • Do you submit from a home machine, compute/cloud or website?
    Cloud
  • Do you want to train your model locally or in the cloud?
    Cloud
  • How often do you retrain your models?
    About three times yearly.
  • What cloud platform are you most comfortable with?
    Saturn
  • Do you use version control for your model code?
    Yes.
  • What are the biggest pain points with the current Compute setup?
    Iā€™m not very familiar with docker containers. Itā€™s a steep learning curve.
  • How do you typically deploy a model to production?
    Straight from github.
47

  • Are your submissions automated now?
    Yes, I have just automated them on my home machine for the daily tournaments. Using cron, bash scripts and back end Rust. All I ask is some guaranteed deadline within the submission window, by which the data will be available for download.

  • Do you submit from a home machine, compute/cloud or website?
    Home machine

  • Do you want to train your model locally or in the cloud?
    Locally

  • How often do you retrain your models?
    Not often, just when I get some ideas for (hoped for) improvements.

  • What cloud platform are you most comfortable with?
    github actions

  • Do you use version control for your model code?
    Yes

  • What are the biggest pain points with the current Compute setup?
    Too pythonesque. I have put quite a lot of effort into setting everything up my own way, in Rust, so that it runs orders of magnitude faster. Any changes assuming all kinds of complications on your side induce pain.

  • How do you typically deploy a model to production?
    I get an idea, I change my rust source accordingly, train a new model and run some benchmarks. If the results appear to show promise, I put it into production. Sometimes I iterate this loop a few times.

None of your proposals are appealing, I know nothing about all that bull and donā€™t need any of it. Mandatory ā€˜Computeā€™ and the fees associated with it would make me pull out of numerai entirely. If you follow my suggestion and have a guaranteed UTC time for data being available, I can synchronise my cron to it and do everything locally in just a few short minutes, quicker than your ā€˜Computeā€™ can ever aspire to.

PS. I do not think it is at all reasonable to have a hard submission deadline for us but not to have a hard deadline for you to provide the data.

1 Like
48
  • Are your submissions automated now?
    Yes!
  • Do you submit from a home machine, compute/cloud or website?
    Azure Cloud
  • Do you want to train your model locally or in the cloud?
    Cloud, because of memory requirements.
  • How often do you retrain your models?
    Never.
  • What cloud platform are you most comfortable with?
    Azure
  • Do you use version control for your model code?
    Yes, I have a private DevOps git.
  • What are the biggest pain points with the current Compute setup?
    I have some monthly free Azure credits, so I prefer Azure vs AWS.
  • How do you typically deploy a model to production?
    I use a container instance to submit all my models, so I must update a script to include the model, then rebuild and push the new docker image.

I need only one trigger to submit all my models, so I had to put a compute webhook in one of my models, what is a little bit odd. Seems like I have only 1 computed model.
Maybe could be a place to define a webhook for all models.

1 Like
49

If you want to improve Numerai Compute, you could update the docs and provide an updated tutorial. The docs page Numerai CLI and Compute - Numerai Tournament hasnā€™t been updated in 7 months.

2 Likes
50
  • Are your submissions automated now?
    No.
  • Do you submit from a home machine, compute/cloud or website?
    Home machine.
  • Do you want to train your model locally or in the cloud?
    Locally
  • How often do you retrain your models?
    Very rarely
  • What cloud platform are you most comfortable with?
    Iā€™ve never used one. Iā€™m such a neanderthal.
  • Do you use version control for your model code?
    No
  • What are the biggest pain points with the current Compute setup?
    Cloud
  • How do you typically deploy a model to production?
    Run my script to generate prediction for all my models and then upload them one by one. deployment is just adding a trained model folder name to my pipe.
1 Like