I have a compute node that calculates and submits all of my predictions, one per model, one after the other.
I recently added a few more models and with that addition I started seeing failures during the auto-triggered compute runs on Saturdays.
Specifically, the failures occur with the
upload_predictions() call from numerapi. The return message only contains the mysterious single word “data” (it normally contains the submission ID).
When I trigger the compute node myself a few hours later, it runs fine.
I suspect that heavy submission traffic might be causing this failure, and that the upload is timing out or something like that. Has anyone else experienced this? Any suggestions for how to avoid it?
Thanks in advance if so.