Over 2.5 years ago, my first project here at Numerai was to overhaul the (at the time) only automation tool available to our community. I released Numerai CLI 0.3.0 and with it came a flood of requests for bug fixes, support for other data providers, better customization, deeper integration with cloud services, etc.
Several versions of Numerai CLI (aka Compute Heavy) have become available since then including Compute Lite, which recently was replaced with Model Uploads - a free, zero-setup automation method that comes at a cost: customization and control. Many users request features or packages that take us a month or longer to implement.
Compute Heavy remains the automation solution where you are in full control, and is open-sourced so the community can keep the architecture modern by adding tooling to support modern modeling methods. Today, we are announcing the introduction of some major improvements:
- Add support for Google Cloud
- Add support for Microsoft Azure
- Switch AWS from ECS to Batch
With Numerai CLI v1.0, you have more control than ever, including choice over cloud provider and - in the case of AWS - resource consumption.
To upgrade your version today, run
pip install --upgrade numerai-cli
If you’d like to use GCP follow the steps here, then run
numerai setup --provider gcp
If you’d like to use Azure follow the steps here, then run
numerai setup --provider azure
If you’d like to use the new AWS Batch architecture, you’ll need to replace your terraform and your nodes using:
numerai setup --provider aws
numerai destroy-all
numerai node -m [model-name-1] config [options]
numerai node -m [model-name-1] deploy
numerai node -m [model-name-2] config [options]
numerai node -m [model-name-2] deploy
...
AWS Batch offers a host of benefits including custom instance sizing, queueing, auto-retries, and more. It’s what we use for Model Uploads and now you can set up similar architecture in your own environment.
