Automated tool to setup aws ec2 spots instances for ml training

Council of Elders
1

Hi all,
First of all I wish you all the best and happiness for this new year.
I plan to launch very soon a platform of automated management of aws ec2 spots instances which would allow the community to :

docs.google.com

Quick survey

First of all I wish you all the best and happiness for this new year. I plan to launch very soon a platform of automated management of aws ec2 spots instances which would allow you to : - save up to 90% of the costs - develop, test and iterate quickly in a secure way, new algorithms of artificial intelligence I would be delighted if you could take 2 minutes to answer my survey.

1 Like
2

Have you heard about nimbo (GitHub - nimbo-sh/nimbo: Run compute jobs on AWS as if you were running them locally.)? Sounds similar to what you are trying to do.

1 Like
3

Hi @uuazed, i check your link. Nimbo is an interesting tool but it doesn’t totally cover my need, which is to set up an automated intelligent cloud solution managing aws spot instances (#cloudhack) by auto-scaling. It will help the community setup and run ec2 instances while saving up to 90%.
Thanks for the suggestion

4

Nimbo works with spot instances, just saying. I guess, what I am missing in the proposal is the numerai specific part.

Anyway, I am curious to hear what the community things about your proposal and the outcome of your survey.

5

How does Nimbo handle aws spot instance interruptions ?
Is there an automatic instance allocation function after an interrupt ?, or do I need to allocate manually by re-executing shell command ?

6

Is the price of aws spot instances refreshed every time you execute the command ?
Can we configure a scheduler to refresh the prices of spot instances dynamically with Nimbo ?
How long have you been using Nimbo, I just want to know if the tool is suitable for what I want to do.

7

I don’t know. Probably no automatic re-allocation. Results / files are stored in an s3 bucket, so retrieving some previous state might be possible

I assume the prices are refreshed with each call. There is probably no build in scheduler, as this is just a command line tool.

8

Thank you for your help. It’s appreciable. I am still investigating the different possibilities. Nimbo seems like something simple and easy to set up. I’ll let you know what happens next.