Here is a bash script I have written which automates submissions to daily tournaments from home machines, without using Compute or web hooks or any other complications. If you want to learn to write similar scripts yourself, you may like to buy my e-book Bash for Fun
Edit:
-
Now an updated version that executes Tuesdays till Saturdays inclusive (can be changed).
-
Simplified version that uses sleep instead of rewriting crontab.
It is assumed that your script does all else that is necessary: that is check for new validations and live data, download them when necessary, do the processing and send the predictions to the server.
#!/bin/bash
# Sets up cron checks for new numerai rounds TUE-FRI, from 13:01 UTC.
# Local timezone is taken care of, as long as it is not sub-hourly.
# Activate only once, verify with crontab -l, deactivate with crontab -r
# Make sure that your computer and internet connection are on at the target time
# days of the week to be checking for new tournaments - no spaces!
FIRSTDAY=Tue
LASTDAY=Sat
# hours in UTC to be checking from
HRS=13
# at MINS minutes intervals after the initial attempt at $HRS:01
MINS=5
# name of bash script here that downloads live data, generates predictions and uploads them
PREDICT=numbatch
# full path/name of fully installed numerapi CLI (python) script
NAPI=/home/l/.local/bin/numerapi
# name of log file here
LOG=cronlog
############ no need to change anything below ###################
# physical path/name of this script
SCRIPT=$(realpath $0)
# PREDICT script must be in the same directory as this script: $PD
PD=$(dirname $SCRIPT)
# crontab expects by default local time, so convert from UTC
LHRS=`date -d "$HRS UTC" +%H`
DAYS=`date -d "$HRS next $FIRSTDAY UTC" +%w`-`date -d "$HRS next $LASTDAY UTC" +%w`
# no argument given means first time manual startup to initiate crontab
# thereafter this script is invoked by cron with dummy argument 'r' to do actual processing
[ $1 ] || {
printf "1 $LHRS * * $DAYS $SCRIPT r >> $PD/$LOG\n" | crontab - && {
printf "$0: `date` crontab initiated:\n`crontab -l`\n" | tee -a $PD/$LOG
exit 0
} || {
printf "$0: `date` crontab initiation failed!\n" | tee -a $PD/$LOG
exit 2
}
}
# Invoked by cron at the specified time 13:01 UTC
while [ "$NAPI check-new-round --hours 1" != '1' ]; do
printf "$0: `date -u`: tournament has not yet started\n"
sleep "$MINS"m #sleep till tournament opens
done
# new tournament has started within the last hour, so download, predict, upload
printf "$0: `date -u`: new tournament has started!\n"
printf "$0: running $PD/$PREDICT\n"
$PD/$PREDICT >> $PD/$LOG
printf "$0: All done for today!\n"
exit 0
7 Likes
Here is a new version that should also cope with lots of errors that numerapi sometimes throws up, outside users’ control:
#!/bin/bash
# Sets up cron checks for new numerai rounds TUE-FRI, from 13:01 UTC.
# Local timezone is taken care of.
# Activate only once, verify with crontab -l, deactivate with crontab -r
# Make sure that your computer and internet connection are on at the target time
########### Change these values to suit (no spaces around =) #######################
FIRSTDAY=Tue
LASTDAY=Sat
HRS=13 # hours in UTC to be checking from
MINS=1 # after minutes of initial delay
DOZE=5 # sleep intervals in minutes after unsuccessful attempts
PREDICT=numbatch # bash script in this directory that does all the dowloading, processing and uploading
NAPI=/home/l/.local/bin/numerapi # absolute path/name of fully installed numerapi CLI (python) script
LOG=cronlog # main log file
ERRLOG=errlog # logfile for numerapi errors
############ No need to change anything below here ###################
SCRIPT=`realpath $0`
PD=$(dirname $SCRIPT) # absolute path directory to this SCRIPT and PREDICT
# crontab expects by default local time, so convert to it from UTC
LMINS=`date -d "$HRS:$MINS UTC" +%M`
LHRS=`date -d "$HRS:$MINS UTC" +%H`
LDAYS=`date -d "$HRS:$MINS next $FIRSTDAY UTC" +%w`-`date -d "$HRS:$MINS next $LASTDAY UTC" +%w`
# manual invocation with no argument just initiates the crontab and quits
[ $1 ] || {
printf "$LMINS $LHRS * * $LDAYS $SCRIPT r\n" | crontab - && {
printf "$0: `date` crontab initiated:\n`crontab -l`\n" | tee -a $PD/$LOG
exit 0
} || {
printf "$0: `date` crontab initiation failed!\n" | tee -a $PD/$LOG
exit 2
}
}
# Invoked by cron at the specified time (13:01) UTC
while :; do
NRES=`$NAPI check-new-round --hours 1` 2>$PD/$ERRLOG
# numerapi status is valid and the new tournament has started
[ $? ] && [ "$NRES" == '1' ] && {
printf "$0: `date -u`: new tournament has started!\n" | tee -a $PD/$LOG
printf "$0: running $PD/$PREDICT\n" | tee -a $PD/$LOG
$PD/$PREDICT | tee -a $PD/$LOG # download, predict, upload
printf "$0: All done for today!\n" | tee -a $PD/$LOG
exit 0
}
printf "$0: `date -u`: tournament has not yet started, sleeping for $DOZE minutes\n" | tee -a $PD/$LOG
sleep "$DOZE"m
done
2 Likes
I have now also multithreaded my Rust back end and it runs like the wind
Thank you for posting this. Trying to get this to work. I have a few questions.
PREDICT=numbatch # bash script in this directory that does all the dowloading, processing and uploading
Is this my own .py script (model)? Should I have a standalone python or can i reference utils?
NAPI=/home/l/.local/bin/numerapi # absolute path/name of fully installed numerapi CLI (python) script
Is this the python package or my own .py script?
Thank you sir.
Thanks, I have added comments to make this clearer.
Your own script ($PREDICT) has to be bash, i.e. if you want to run your python script, it will contain:
python3 mypythonscript
#!/bin/bash
# Sets up cron checks for new numerai rounds TUE-FRI, from 13:01 UTC.
# Local timezone is taken care of.
# Activate only once, verify with crontab -l, deactivate with crontab -r
# Make sure that your computer and internet connection are on at the target time
########### Change these values to suit (no spaces around =) #######################
FIRSTDAY=Tue
LASTDAY=Sat
HRS=13 # hours in UTC to be checking from
MINS=1 # after minutes of initial delay
DOZE=5 # sleep intervals in minutes after unsuccessful attempts
PREDICT=numbatch # your own bash script in this directory that does all the dowloading, processing and uploading
NAPI=/home/l/.local/bin/numerapi # absolute path/name of fully installed numerapi CLI (python) script (from numerai.ai)
LOG=cronlog # main log file
ERRLOG=errlog # logfile for numerapi errors
############ No need to change anything below here ###################
SCRIPT=`realpath $0`
PD=$(dirname $SCRIPT) # absolute path directory to this SCRIPT and PREDICT
# crontab expects by default local time, so convert to it from UTC
LMINS=`date -d "$HRS:$MINS UTC" +%M`
LHRS=`date -d "$HRS:$MINS UTC" +%H`
LDAYS=`date -d "$HRS:$MINS next $FIRSTDAY UTC" +%w`-`date -d "$HRS:$MINS next $LASTDAY UTC" +%w`
# manual invocation with no argument just initiates the crontab and quits
[ $1 ] || {
printf "$LMINS $LHRS * * $LDAYS $SCRIPT r\n" | crontab - && {
printf "$0: `date` crontab initiated:\n`crontab -l`\n" | tee -a $PD/$LOG
exit 0
} || {
printf "$0: `date` crontab initiation failed!\n" | tee -a $PD/$LOG
exit 2
}
}
# Invoked by cron at the specified time (13:01) UTC
while :; do
NRES=`$NAPI check-new-round --hours 1` 2>$PD/$ERRLOG
# numerapi status is valid and the new tournament has started
[ $? ] && [ "$NRES" == '1' ] && {
printf "$0: `date -u`: new tournament has started!\n" | tee -a $PD/$LOG
printf "$0: running $PD/$PREDICT\n" | tee -a $PD/$LOG
$PD/$PREDICT | tee -a $PD/$LOG # download, predict, upload
printf "$0: All done for today!\n" | tee -a $PD/$LOG
exit 0
}
printf "$0: `date -u`: tournament has not yet started, sleeping for $DOZE minutes\n" | tee -a $PD/$LOG
sleep "$DOZE"m
done
Thank you very much. Fingers crossed. It appears to be properly set up.
You are welcome. It works well for me.
The problem I have are the unpredictable times of the tournament start.
I could leave cron running and it would do it at the right time but
it is around midnight here and I want to switch my big computer off and go to bed.
If it at least was at the originally promised 13:00 UTC, it would not be so bad.
The latest improved version of my Bash script, now supporting cron activation of two different targets scripts ‘daily’ and ‘weekly’. Where only the latter might concern itself with the weekly data downloads.
#!/bin/bash
# Sets up cron checks for numerai tournament rounds TUE-SAT, from 13:30 UTC. Local timezone is taken care of.
# Invocation with no argument initiates the crontab for both weekly and daily runs.
# Invocation with `w` argument initiates the crontab only for weekly runs (Saturdays).
# Invocation with `d` argument initiates the crontab only for daily runs (Tuesdays-Fridays).
# Tue-Fri runs bash script named `daily`, on Saturdays runs script named `weekly`.
# both `daily` and `weekly` scripts must be present in the same directory as this script (cronrun).
# Activate only once, verify with crontab -l, deactivate with crontab -r.
# Make sure that your computer and internet connection are on at the specified target time (13:30 UTC).
########### Change these values to suit (no spaces around =) #######################
FIRSTDAY=Tue
LASTDAY=Fri
HRS=13 # hours in UTC to be checking from
MINS=30 # after minutes of initial delay
DOZE=5 # sleep intervals in minutes after unsuccessful attempts
NAPI=/home/l/.local/bin/numerapi # absolute path/name of fully installed numerapi CLI (python) script (from numerai.ai)
LOG=cronlog # main log file
ERRLOG=errlog # logfile for numerapi errors
############ No need to change anything below here ###################
SCRIPT=`realpath $0`
PD=$(dirname $SCRIPT) # absolute path directory to this SCRIPT and PREDICT
# crontab expects by default local time, so convert to it from UTC
LMINS=`date -d "$HRS:$MINS UTC" +%M`
LHRS=`date -d "$HRS:$MINS UTC" +%H`
LDAYS=`date -d "$HRS:$MINS next $FIRSTDAY UTC" +%w`-`date -d "$HRS:$MINS next $LASTDAY UTC" +%w`
SAT=`date -d "$HRS:$MINS next Sat UTC" +%w`
# Manual invocation with no argument initiates the crontab for both weekly and daily runs and quits.
[ $1 ] || {
printf "$LMINS $LHRS * * $LDAYS $SCRIPT daily\n$LMINS $LHRS * * $SAT $SCRIPT weekly\n" | crontab - && {
printf "$0: `date` crontab initiated:\n`crontab -l`\n" | tee -a $PD/$LOG
exit 0
} || {
printf "$0: `date` crontab initiation failed!\n" | tee -a $PD/$LOG
exit 2
}
}
# Manual invocation with `w` argument initiates the crontab only for weekly (Saturday) runs and quits.
[ "$1" == 'w' ] && {
printf "$LMINS $LHRS * * $SAT $SCRIPT weekly\n" | crontab - && {
printf "$0: `date` crontab initiated:\n`crontab -l`\n" | tee -a $PD/$LOG
exit 0
} || {
printf "$0: `date` crontab weekly initiation failed!\n" | tee -a $PD/$LOG
exit 2
}
}
# Manual invocation with `d` argument initiates the crontab only for daily (Tue-Fri) runs and quits.
[ "$1" == 'd' ] && {
printf "$LMINS $LHRS * * $LDAYS $SCRIPT daily\n" | crontab - && {
printf "$0: `date` crontab initiated:\n`crontab -l`\n" | tee -a $PD/$LOG
exit 0
} || {
printf "$0: `date` crontab daily initiation failed!\n" | tee -a $PD/$LOG
exit 2
}
}
# Invoked by cron automatically at the specified time (13:01 UTC)
# cronrun can also invoke this processing manually, if its usual time has been missed
# If it did run already, no harm done, it will just quit
[ "$1" != 'daily' ] && [ "$1" != 'weekly' ] && {
printf "$0: `date` crontab argument $1 unrecognised!\n" | tee -a $PD/$LOG
exit 0
}
[ -f "$PD/$1" ] || {
printf "$0: `date` crontab: script $PD/$1 not found!\n" | tee -a $PD/$LOG
exit 0
}
while :; do
NRES=`$NAPI check-new-round --hours 1` 2>$PD/$ERRLOG
# numerapi status is valid and the new tournament has started
[ $? ] && [ "$NRES" == '1' ] && {
printf "$0: `date -u`: new tournament has started!\n" | tee -a $PD/$LOG
printf "$0: running $PD/$1\n" | tee -a $PD/$LOG
$PD/$1 | tee -a $PD/$LOG # download, predict, upload
printf "$0: All done for today!\n" | tee -a $PD/$LOG
exit 0
}
printf "$0: `date -u`: tournament has not yet started, sleeping for $DOZE minutes\n" | tee -a $PD/$LOG
sleep "$DOZE"m
done
1 Like
