Introducing Queued Submissions & Submission Statuses

Queued Submissions

Models across all tournaments can now queue a submission for the next round. Whether or not the model has an on-time submission, it can now upload a late submission that will be automatically queued to be an on-time submission for the next round.

At the start of each new round (from round 452 onwards), queued submissions from the previous round are automatically converted to on-time submissions for the new round. You can then overwrite this on-time submission as normal if you choose.

Possible FAQs

  • What happens if I already have on-time submissions?

    • Your most recent late submission is queued for the next round and is ignored otherwise.
  • What happens if I have no submissions for a round, then submit late?

    • Your most recent late submission is treated normally (it will have all non-TC scores calculated) and is also considered queued for next round. Eventually we will stop scoring late submissions and only treat them as queued.
  • What happens if I make multiple late submissions?

    • Only your most recent late submission is considered to be queued, so all other late submissions will be ignored.

Submission Statuses

In support of Queued Submissions, we also released a new website feature that displays basic information about each of your models recent submissions. You can find it in the models page on both the overview and submissions tables as shown here:

Here is an example of how to get this data from our GraphQL API:

Example query:

query account {
  account {
    models {
      username
      tournament
      latestSubmissions {
        roundNumber
        roundCloseStaking
        roundClose
        roundOpen
        filename
        status
        timestamp
      }
    }
  }
}

Example Response:

{
  "data": {
    "account": {
      "models": [
        {
          "latestSubmissions": [
            {
              "filename": null,
              "roundClose": "2023-03-30",
              "roundCloseStaking": "2023-03-29",
              "roundNumber": 451,
              "roundOpen": "2023-03-29",
              "status": "none",
              "timestamp": null
            },
            {
              "filename": "numerai_example_predictions_data-TurVb05zpZGr.csv",
              "roundClose": "2023-03-29",
              "roundCloseStaking": "2023-03-28",
              "roundNumber": 450,
              "roundOpen": "2023-03-28",
              "status": "on-time",
              "timestamp": "2023-03-28T13:12:59Z"
            },
            {
              "filename": "numerai_example_predictions_data-xCS8SjRTdJTy.csv",
              "roundClose": "2023-03-28",
              "roundCloseStaking": "2023-03-27",
              "roundNumber": 449,
              "roundOpen": "2023-03-25",
              "status": "on-time",
              "timestamp": "2023-03-25T15:12:28Z"
            }
          ],
          "tournament": 8,
          "username": "integration_test"
        },
...
9 Likes

Hi @numerark, looks good, I assume that the time-window can still shift depending on whether or not your data providers are on-time and your pipelines have no issues? And anything outside of the 1-hour window will be considered a late submission?

Yes that’s correct, for the weekend anything after 14:00 UTC Monday is considered late and for weekdays anything outside of the 1-hour window (starting no earlier than 13 UTC) is considered late.

Is that possible to carry over predictions made during weekend (the Monday submission) for 4 more daily rounds? So that data scientists who prefer to make submission at weekends only can have their predictions automatically copied into the daily rounds in weekdays?

3 Likes

We have discussed offering this support for signals, but at this point we are confident that submissions older than 1 round will be significantly less useful to the Meta Model.

Is there a way to remove a queued submission so that I can submit a proper prediction the next day?

No need to remove. Just submit the proper one on time in the proper window.

Oh nice. Thanks a lot.

Yeah, whatever you upload last (that is accepted for the round you want it for) is the one that counts, and since “late” submissions are actually late from the previous day, if you do an on-time submission after that in the morning when the new round opens you’re all good. And I know at least some people are submitting late submissions everyday just to make sure they submit something if their pipeline breaks during the critical window. (And then there are people like me that submit late most of the time with the intention of it being used, proper or not.)