# How does training data and validation data relate in "time"? **URL:** \ **Category:** Tournament\ **Created:** [May 1, 2021, 8:39pm UTC](https://forum.numer.ai/t/how-does-training-data-and-validation-data-relate-in-time/3161 "2021-05-01T20:39:30Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![slowmoe](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/slowmoe/32/2496_2.png) [@slowmoe](https://forum.numer.ai/u/slowmoe)\ **Post date:** [May 1, 2021, 8:39pm UTC](https://forum.numer.ai/t/how-does-training-data-and-validation-data-relate-in-time/3161/1 "2021-05-01T20:39:30Z")
Complete n00b here. I’ve been playing with the idea of engineering features that tell you something about the era of a point. After some toying, it seems to me that whatever function describes the different regimes of different eras is somewhat continuous on large enough scales. So me thinks nice, lets think more. When I look at the regimes in the validation set (naughty, I know), they seem to connect seamlessly to the last eras of the training set. Almost as if they are directly connected in time. Does anybody know if that might be true? To qualify what I mean with regimes of eras: when you compute the covariance matrix of all features per era, you get a bunch of rather beautiful 310x310 pixel art. If you use matrix norm as metric, those pictures vary pretty continuously over the training and validation set. Curious to hear what you think. ---
**Author:** ![wigglemuse](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/wigglemuse/32/3094_2.png) [@wigglemuse](https://forum.numer.ai/u/wigglemuse)\ **Post date:** [May 1, 2021, 9:03pm UTC](https://forum.numer.ai/t/how-does-training-data-and-validation-data-relate-in-time/3161/2 "2021-05-01T21:03:43Z")
Yep. Training data eras 1-120 represent 10 years (1 era = 1 month). So val eras 121-132 are simply the following year. Then a large gap to other validation eras 197-212 which are from fairly recent times. ---
**Author:** ![slowmoe](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/slowmoe/32/2496_2.png) [@slowmoe](https://forum.numer.ai/u/slowmoe)\ **Post date:** [May 2, 2021, 9:11am UTC](https://forum.numer.ai/t/how-does-training-data-and-validation-data-relate-in-time/3161/3 "2021-05-02T09:11:29Z")
I see, that makes a lot of sense. Thanks! Is this documented somewhere that I missed? ---
**Author:** ![ml\_is\_lyf](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@ml\_is\_lyf](https://forum.numer.ai/u/ml_is_lyf)\ **Post date:** [May 2, 2021, 9:50am UTC](https://forum.numer.ai/t/how-does-training-data-and-validation-data-relate-in-time/3161/4 "2021-05-02T09:50:28Z")
Unfortunately most stuff like this isn’t documented. You can pick this kinda stuff up though from Arbitrage’s office hours. I just watched every video on the Numerai YouTube channel, takes some time but I would recommend it, just watch them while your models are training 🙂 > **[Numerai](https://www.youtube.com/c/Numerai/videos)** > > An artificial intelligence hedge fund built by a community of anonymous data scientists ---
**Author:** ![jimmy\_woodford](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/jimmy_woodford/32/2831_2.png) [@jimmy\_woodford](https://forum.numer.ai/u/jimmy_woodford)\ **Post date:** [May 2, 2021, 11:24am UTC](https://forum.numer.ai/t/how-does-training-data-and-validation-data-relate-in-time/3161/5 "2021-05-02T11:24:03Z")
Here’s what I’ve found (take with a grain of salt): If we assume that era1 is January 2003, then the monthly data that we get is Jan 03 - Dec 12 in train and Jan 13 - Dec 13 & Jun 19 - Sep 20 in val. When I used this month variable in my models, it consistently improved performance by 5-10%. But what about weeks? If numbering is consistent, for weeks we should know exactly when they start and when they end, since live eras become test eras after completion. If this is true, era575 is week 2 2014 (starting January 9, ending February 5), and so on. Unfortunately, I think I never managed to bring these findings together, and my models including month as a variable performed poorly on live. I may have made a mistake. Numerai’s numbering might be inconsistent. Or since weeks often overlap months, the performance boost on monthly data may not carry over to weekly data. In any way, I think Numerai should provide more information to let us try to find some temporal relations. And give us weekly data to train and validate. ---
**Author:** ![gammarat](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gammarat/32/3281_2.png) [@gammarat](https://forum.numer.ai/u/gammarat)\ **Post date:** [May 2, 2021, 2:04pm UTC](https://forum.numer.ai/t/how-does-training-data-and-validation-data-relate-in-time/3161/6 "2021-05-02T14:04:26Z")
I quite agree, at least in some ways.FWIW, I really like your idea of turning the covariance matrices into pixel art. Maybe Numerai should produce NFTs from the covariance matrices for each live era - a big one for the over all round winner, smaller ones from the matrices of combinations of feature groups - and award them to the high scorers? I digress. Anyway, I’m personally intrigued by how each era relates to the others; so I’ve been exploring that over in the the [Analyzing Training Data](http://forum.numer.ai/t/analyzing-training-data/3143/3) thread. If you look at the evolution of the averaged STD for the feature groups Charisma and Strength from the first training set through to the live round there’s an interesting trend. ---
**Author:** ![slowmoe](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/slowmoe/32/2496_2.png) [@slowmoe](https://forum.numer.ai/u/slowmoe)\ **Post date:** [May 2, 2021, 9:04pm UTC](https://forum.numer.ai/t/how-does-training-data-and-validation-data-relate-in-time/3161/7 "2021-05-02T21:04:00Z")
yes, I should definitely listen to the podcast more. Again, I am new and there is so much left for me to explore. I think numbering months and weeks is somewhat problematic since the era length is not constant at all. There might be ways to mitigate that though. here is a pic of the “average covariance” of the training data I came up with btw. ![featurecorrs_baseline](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/1X/e19e3245e7c56721b6674821cbc515b4fbdfa7de.png) ---
**Author:** ![nrichers](https://avatars.discourse-cdn.com/v4/letter/n/8e7dd6/32.png) [@nrichers](https://forum.numer.ai/u/nrichers)\ **Post date:** [May 6, 2021, 7:01pm UTC](https://forum.numer.ai/t/how-does-training-data-and-validation-data-relate-in-time/3161/8 "2021-05-06T19:01:32Z")
> [@jimmy\_woodford](#): > > But what about weeks? If numbering is consistent, for weeks we should know exactly when they start and when they end, since live eras become test eras after completion 4-4-5 calendar may be useful to align week/month columns > **[4–4–5 calendar](https://en.wikipedia.org/wiki/4%E2%80%934%E2%80%935_calendar)** > > The 4–4–5 calendar is a method of managing accounting periods, and is a common calendar structure for some industries such as retail and manufacturing. It divides a year into four quarters of 13 weeks, each grouped into two 4-week "months" and one 5-week "month". The longer "month" may be set as the first (5–4–4), second (4–5–4), or third (4–4–5) unit. > Its major advantage over a regular calendar is that each period is the same length and ends on the same day of the week, which is useful for plan... ---
**Author:** ![jimmy\_woodford](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/jimmy_woodford/32/2831_2.png) [@jimmy\_woodford](https://forum.numer.ai/u/jimmy_woodford)\ **Post date:** [May 6, 2021, 7:44pm UTC](https://forum.numer.ai/t/how-does-training-data-and-validation-data-relate-in-time/3161/9 "2021-05-06T19:44:11Z")
Interresting, thanks 🙂 . I’d really like to get a way to add a time variable to the mix since it so clearly works in the training/val data.