I have been running Rust since the beginning. I have shell script cron automation on my local machine but sometimes deadlines get missed, for example because of internet connection problems. I wonder if there is any way of making use of the compute lite beta? It can be done. I am able to run Rust securely for example on github actions. All the instructions on numerai, however, seem to assume that we run interpreted (slow) Python.
2 Likes
I don’t think you’ll get a good response here, the dominant majority uses Python with some using R too.
But if you are determined, GitHub - PyO3/pyo3: Rust bindings for the Python interpreter might be worth looking into. No guarantees that it will work out the way you expect though.
Also, note that compute-lite is currently not well supported. See Discord
1 Like
I am more likely to go the other way and to use Parquet reader etc. directly from Rust. Still no server automation though.
Kudos for trying to use Rust to implement Numerai pipeline! I develop in Rust on a daily basis yet I still use Python for Numerai mainly for historical and compatibility reasons. But for instance polars looks to be a great replacement for pandas (and it should handle reading parquet files natively), so maybe I will move to Rust with my pipelines as well one day…
About automation - I would just pack your whole pipeline as a simple docker image (so compiling your Rust binaries will be part of the Dockerfile) and then use Amazon ECS & Fargate to schedule and run the container appropriately.
Some example how you could containerize your Rust programs: Getting Started with Rust and Docker – Collabnix
Good luck!
1 Like
Thanks for the headsup about polars. They look promising, I will try them.
If I can avoid going through all those python pyrotechnics just to convert .parquet to .csv,
then my pipeline will be so much simpler and faster!