Hey everybody! Back in January I asked on Discord what happens when my Kaggle pipeline runs at the wrong time and the submission gets queued for the next round. The answer was “it carries forward”, and I ended that chat with “I may have been losing a small edge by submitting the previous day’s predictions..”. A friend told me he had looked at it and the difference was barely noticeable. I believed him, but I also kept re-running the pipeline by hand randomly some morning when the “submissions are due now” email arrived, like a human cron job, for months 
. So here are actual numbers, because so far this topic has only been opinions from what I have been able to investigate.
TL;DR: same models, same week, scored on same-day data vs on carried-forward day-old data. Classic payout score: fresh minus stale about -7% (95% CI -20% to +7%). Crypto corr: about +2% (95% CI -11% to +15%). Nothing to see, and the Classic lean is even in the wrong direction. The catch: with one account you can only detect a loss of roughly 20% of your score. A smaller loss is invisible, I put the power numbers below so nobody over-reads the “no difference”.
Micro introduction of what happens with late uploads
This part surprised me a bit and it is all in the API, no need to hover over the little green squares. The GraphQL query `submissions(modelId)` gives you per submission `filename`, `insertedAt`, `selected`, `sourcePlatform` and `dataDatestamp`, plus the round’s `number`, `openTime` and `dataDatestamp`.
-
Rounds open at 12:00 UTC Tuesday to Saturday. The deadline that matters (`closeStakingTime`) is about 13:02 to 13:09 UTC, so the window is ~1 hour. The Saturday round gets a longer window, lately until Sunday 16:00 UTC. The schedule has not changed in the whole period; what moved recently is only the first scoring date (a Tuesday round now gets its first score the next Monday), which happens after submission and does not affect any of this.
-
If you upload after that, the file is attached to the round with `selected = false` and **it is never scored**.
-
When the next round opens, Numerai copies it forward as that round’s submission (`sourcePlatform = carry_forward`). So it gets scored one day late, built from data one day older than the round’s data. On Tuesdays it is Friday’s file, so three days older.
Freshness of what got scored is simply the round’s `dataDatestamp` minus the submission’s `dataDatestamp`: 0 = fresh, 1 = one day stale, 3 = weekend stale. One script, all models, all rounds, done.
The “Experiment”
Kaggle picks the run time. Some days it lands inside the window, some days it does not, and since February I have also been triggering it manually when I was awake and near the laptop. None of that knows anything about the market that day. So within a model, fresh and stale days are assigned basically at random, which is exactly what you want for a fair comparison. I use the period since I started the manual triggering (February 2026 onward) as the main test, and the full history as a secondary check.
In the past I relied more in model uploads for classic, and migrated most of the models to my kaggle pipeline in November, which is easily shown in the plot. In crypto, since there is no model upload, it has always been kaggle pipeline and the increase in fresh scores correlate with the start of the manual triggers.
My setup
| Classic | Crypto | |
|---|---|---|
| models | 85 | 53 |
| resolved rounds in the period | 148 (1195 to 1342) | 157 (1195 to 1354) |
| model-rounds | 11,062 | 5,415 |
| fresh model-rounds | 5,621 | 2,986 |
| stale by 1 day | 4,064 | 1,912 |
| stale by 3 days (Tuesdays) | 1,377 | 517 |
| rounds all-fresh / all-stale / mixed | 70 / 30 / 48 | 85 / 67 / 5 |
Only resolved final scores. Classic rounds from 1343 onward (the new 3 x CORR60 + 15 x MMC60 payout) are not resolved yet, so they are out for now.
The headline metric is Numerai’s own payout score per round, using the per-round multipliers the API returns (`v2RoundModelPerformances.payoutMultipliers`): Classic 0.75 x CORR20v2 + 2.25 x MMC for rounds 1172 to 1342; Crypto 0.10 x CORR + 1.0 x MMC from round 1292 (0.05 / 0.5 before that). I also report corr and mmc separately. Crypto’s payout is dominated by MMC, and my MMC average there is around zero, so percentages on it are nonsense; for Crypto the honest headline is corr.
How I compared:
-
One observation = one model in one round.
-
Remove weekday effects per model first. Saturday rounds are almost always fresh (two-day window), so without this step you are comparing Saturdays with weekdays, not fresh with stale.
-
Within week: for each model and each week that has at least one fresh and one stale round, mean(fresh) minus mean(stale). Then average across models so that **each week contributes one number**. All my models go fresh or stale on the same days, so 85 models are not 85 samples, they are 85 copies of the same 30 weeks.
-
Adjacent weeks, same weekday: per model, pair each fresh round with the nearest stale round on the same weekday within 14 days. A second angle on the same question.
-
Newey-West standard errors (3 lags, rounds share overlapping targets), cross-checked with a 4-week block bootstrap. They agree.
-
Power: the smallest loss I could have detected at 80% power, and how many paired weeks a 5 / 10 / 20% loss would need.
Results (Feb to Sep 2026, weekday-adjusted, fresh minus stale)
| metric | design | weeks | diff | % of level | 95% CI | t | weeks fresh better |
|---|---|---|---|---|---|---|---|
| Classic payout score | within week | 30 | -0.0014 | -6.9% | -20% to +7% | -1.0 | 37% |
| Classic payout score | adjacent weeks | 55 | -0.0017 | -8.4% | -38% to +21% | -0.6 | 45% |
| Classic corr | within week | 30 | -0.0007 | -4.9% | -18% to +8% | -0.7 | 43% |
| Classic corr | adjacent weeks | 55 | +0.0008 | +5.4% | -20% to +31% | +0.4 | 49% |
| Classic mmc | within week | 30 | -0.0004 | -9.4% | -28% to +9% | -1.0 | 47% |
| Crypto corr | within week | 31 | +0.0017 | +1.9% | -11% to +15% | +0.3 | 58% |
| Crypto corr | adjacent weeks | 63 | +0.0012 | +1.2% | -15% to +17% | +0.2 | 46% |
| Crypto mmc | within week | 31 | +0.0033 | n/a | -0.0036 to +0.0081 | +1.1 | 68% |
Per model (payout score, within week): fresh was better on average for 19 of 84 Classic models and 36 of 53 Crypto models, and basically no single model is significant in either direction. Again, these are not independent votes, it is the same 30 weeks seen through 84 and 53 pairs of eyes.
Week by week, Classic leaned negative from February to May and then went flat; Crypto swung both ways. Neither path is outside what pure noise does over 30 weeks, I checked, I wanted the Classic dip to mean something hehe.
Full history as a sanity check (same method, every resolved round): Classic payout +3% (CI -17% to +24%, 86 weeks); Crypto corr -2% (CI -13% to +10%, 40 weeks). Same story.
The interesting bit: what this test can and cannot see
| metric | SE of weekly diff | smallest detectable loss (80% power) | weeks for a 20% loss | 10% | 5% |
|---|---|---|---|---|---|
| Classic payout score | 6.9% of level | 19% | 35 | 139 | 553 |
| Classic corr | 6.8% of level | 19% | 20 | 78 | 311 |
| Crypto corr | 6.6% of level | 18% | 20 | 80 | 320 |
So with ~30 paired weeks, the day-old data would have had to cost about a fifth of my score for this to show up. A 10% loss would take about two and a half more years of mixed weeks from one account, a 5% loss is a **retirement project** 
. The honest statement is the confidence interval: the data rule out a loss bigger than roughly 20% (Classic payout) or 11% (Crypto corr), and nothing tighter. If a few of us run the same table, the bound tightens fast, which is half the reason I am posting the method and not just the conclusion.
Caveats
-
One account, gradient-boosting models on the standard features. If your model leans on fast-moving features, your mileage may vary.
-
Numerai never scores the unselected fresh file, so the public scores cannot give the ideal comparison (same model, same round, both files). It is doable by downloading both files per model-round and scoring locally (the API lets you download your own submissions if the key has that permission). This could be an interesting update.
-
The February 2026 start of the manual period is inferred from the upload pattern (same-window uploads jump from ~0 to ~30% of days), I did not keep a diary.
-
Classic rounds under the new 60-day payout are not in here yet, I will update when they resolve.
My take so far
Manually re-running the pipeline inside the window buys me nothing I can measure. Day to day the features barely move, which is what everybody suspected, now there is at least one set of numbers behind it. If somebody has a model class where freshness should matter more, I would love to see the same table for it. I could automate the experiment with local on-time submissions but I dislike to have an always on laptop… 
This post was mostly written by Claude, I just dumped the data and the ideas, as usual hehe.
