I was a bit dismayed on Sunday morning when I checked my R console to see that my script (which takes ~20 hours to run) crashed. I have tested this script dozens of times with no issues before. I had 24 hours before my submission was due, so I ran the script again hoping it was a fluke. I had to wake up at 5:30 CDT this morning to make sure it completed and thankfully it did! Submission was on time, whew.
One of my hypotheses is that the change in time screwed up my Windows instance which then killed R.gui. Logs showed it was due to a HungAppTimeout which occurred at precisely 4:00am (an hour after the time change). Here are some snapshots of my CPU usage:
I know this doesn’t have much to do with Numerai but I thought it was worth sharing. Perhaps I need to migrate to a Linux machine…
This whole incident did bring up a related question though. How badly are you penalized if your submission is late? Is it binary (either late or not) or does it get worse the later it is?