# Diagnostics for #39 **URL:** \ **Category:** Data Science\ **Created:** [September 19, 2021, 8:40am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155 "2021-09-19T08:40:17Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![nyuton](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/nyuton/32/687_2.png) [@nyuton](https://forum.numer.ai/u/nyuton)\ **Post date:** [September 19, 2021, 8:40am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/1 "2021-09-19T08:40:17Z")
Diagnostics are built to help us, but they can be very misleading. My most successfull model, which stands at #39 at the time of writing (nyuton\_test8) has so bad diagnostics, that I almost threw it out at the beginning and I haven’t started staking on it until recently. Trust you CV! I’m writing this post partly for myself as a reminder, when I see similar results with the new dataset. To be frank, this model uses an ensemble of models from CV folds. Data also includes validation set. This is the diagnostic of the only part that doesn’t include validation data at all. Probably a good approximation of the other models as well. ![image](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/3/3ef94956fb0d1ba91c6dd97a39c2e61a04b9f2ff.png) An other model (nyuton\_test15) looks ever worse than that. And it got 14 medals in its first 9 completed rounds… ---
**Author:** ![restrading](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/restrading/32/2928_2.png) [@restrading](https://forum.numer.ai/u/restrading)\ **Post date:** [September 19, 2021, 9:42am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/2 "2021-09-19T09:42:41Z")
Defintely a great reminder, trust your CV indeed. Although I’d still keep an eye on risk because getting metals+high rank may also be a result of high variance models, which tend to do worse when regime changes ---
**Author:** ![sunkay](https://avatars.discourse-cdn.com/v4/letter/s/6f9a4e/32.png) [@sunkay](https://forum.numer.ai/u/sunkay)\ **Post date:** [September 19, 2021, 11:16am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/3 "2021-09-19T11:16:12Z")
Great reminder! But what is #39? ---
**Author:** ![qeintelligence](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/qeintelligence/32/2531_2.png) [@qeintelligence](https://forum.numer.ai/u/qeintelligence)\ **Post date:** [September 19, 2021, 11:53am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/4 "2021-09-19T11:53:07Z")
The other way around is also definitely the case, I had a NN with ‘reasonable’ diagnostics (everything green) which also performed well for 5-6 rounds, after that it got burned 4 rounds, only to recover now again. And no, I didnt use val for training 🙂 (but i suspect there is a leakage because of determining the amount of epochs with the use of val) ---
**Author:** ![nyuton](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/nyuton/32/687_2.png) [@nyuton](https://forum.numer.ai/u/nyuton)\ **Post date:** [September 19, 2021, 12:36pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/5 "2021-09-19T12:36:02Z")
Ranks 39 on the leaderboard. ---
**Author:** ![neosbrother](https://avatars.discourse-cdn.com/v4/letter/n/dec6dc/32.png) [@neosbrother](https://forum.numer.ai/u/neosbrother)\ **Post date:** [September 19, 2021, 1:45pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/6 "2021-09-19T13:45:28Z")
So your CV results looked very good but it did poorly on the full diagnostics? ---
**Author:** ![nyuton](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/nyuton/32/687_2.png) [@nyuton](https://forum.numer.ai/u/nyuton)\ **Post date:** [September 19, 2021, 5:27pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/7 "2021-09-19T17:27:00Z")
Yes, CV was great! Validation score was not so great! ---
**Author:** ![yxbot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/yxbot/32/2906_2.png) [@yxbot](https://forum.numer.ai/u/yxbot)\ **Post date:** [September 20, 2021, 9:46pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/8 "2021-09-20T21:46:59Z")
my most profitable model - running for 4 months with top 100 3M return - also have a rubbish diagnostics score - I was able to pick it out for staking because it seems to have by far the most stable daily score, additionally, after I settled on my validation scheme it was shown to be 4th among my models on non-corr related metrics i.e. ratios based metrics So yes, people should try to create validation methods that are at least not entirely dependent on the provided validation data. ---
**Author:** ![profricecake](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/profricecake/32/2528_2.png) [@profricecake](https://forum.numer.ai/u/profricecake)\ **Post date:** [September 22, 2021, 2:21am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/9 "2021-09-22T02:21:13Z")
Out of curiosity, what kind of mean correlation numbers do you see in your CV runs? Thanks ---
**Author:** ![nyuton](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/nyuton/32/687_2.png) [@nyuton](https://forum.numer.ai/u/nyuton)\ **Post date:** [September 22, 2021, 6:20am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/10 "2021-09-22T06:20:59Z")
0.043 when trained on train+validation ---
**Author:** ![profricecake](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/profricecake/32/2528_2.png) [@profricecake](https://forum.numer.ai/u/profricecake)\ **Post date:** [September 22, 2021, 2:32pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/11 "2021-09-22T14:32:00Z")
Thank you! Very helpful. I’ve been leaning on validation metrics for so long I didn’t know what to expect from a CV score! ---
**Author:** ![profricecake](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/profricecake/32/2528_2.png) [@profricecake](https://forum.numer.ai/u/profricecake)\ **Post date:** [September 22, 2021, 9:34pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/12 "2021-09-22T21:34:28Z")
One other question: When you find a good set of hyperparameters using CV on some number of folds of the combined train+validation data, do you then train your final model (the one you use for tournament prediction) by using those HPs and all the available data (instead of folds)? I’m assuming not because then you wouldn’t have a sane measure of when to stop the training. But if not, then do you just stick with training on, say, 4 of 5 folds and validating on the 5th? Thanks again. ---
**Author:** ![nyuton](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/nyuton/32/687_2.png) [@nyuton](https://forum.numer.ai/u/nyuton)\ **Post date:** [September 23, 2021, 6:59am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/13 "2021-09-23T06:59:14Z")
Oh man, this is a dead simple random forest! That’s the funny part of it. Apparently you can get this close to the top with an RF. NNs might give you a boost in MMC. But in raw accuracy (CORR) RF is hard to beat it. ---
**Author:** ![kenfus](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/kenfus/32/3397_2.png) [@kenfus](https://forum.numer.ai/u/kenfus)\ **Post date:** [September 23, 2021, 9:37am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/14 "2021-09-23T09:37:31Z")
Yea, I was also wondering why the diagnostics is pretty much useless when comparing models. It’s a good sanity check, but not much more. Could it be that some people have leakage to the validation-set, and thus they skew the ranks in the diagnostics to the top? At the beginning, I did use in CV the validation-set and even though the model never trained on the validation-set, this still lead to a massive overfit; All my metrics were light-green (99+), but the live tests were absolutely terrible ([Numerai](https://numer.ai/kenfus) before round 274). ---
**Author:** ![themicon](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/themicon/32/2487_2.png) [@themicon](https://forum.numer.ai/u/themicon)\ **Post date:** [September 23, 2021, 10:08am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/15 "2021-09-23T10:08:25Z")
Everybody had bad rounds before R274: [https://dashboard.numeraipayouts.com/](https://dashboard.numeraipayouts.com/) The median score was pretty much negative. You need to judge the relative performance of you model on live data, not the performance in isolation. ---
**Author:** ![kenfus](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/kenfus/32/3397_2.png) [@kenfus](https://forum.numer.ai/u/kenfus)\ **Post date:** [September 23, 2021, 10:29am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/16 "2021-09-23T10:29:14Z")
I agree, and this is what I’m doing currently. However, over 4 rounds my model was much, much worse than other models. Maybe I should have generated more data but being in 2 of 4 rounds below a 20 percentile is enough for me to say that it is “not optimal, probably”. ---
**Author:** ![nyuton](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/nyuton/32/687_2.png) [@nyuton](https://forum.numer.ai/u/nyuton)\ **Post date:** [September 23, 2021, 10:59am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/17 "2021-09-23T10:59:39Z")
That doesn’t say much at all, you need to test longer. The above mentioned model has also has more than 2 round in the lower 20 percentile… ---
**Author:** ![factorsparsity](https://avatars.discourse-cdn.com/v4/letter/f/85f322/32.png) [@factorsparsity](https://forum.numer.ai/u/factorsparsity)\ **Post date:** [September 25, 2021, 8:20pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/18 "2021-09-25T20:20:52Z")
This is for the old data format (310 features) or the new one? BTW: I can corroborate your initial statement on the diagnostics. They don’t always indicate whether a model is good or not. ---
**Author:** ![johnnywhippet](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/johnnywhippet/32/2803_2.png) [@johnnywhippet](https://forum.numer.ai/u/johnnywhippet)\ **Post date:** [September 27, 2021, 3:49pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/19 "2021-09-27T15:49:18Z")
The diagnostics for this model are appalling but… its a decent model. peaked at 4 for MMC and 750 or so for correlation. A consistent performer, more green than red. 19 medals in all. ![m3_diag](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/2/2aaa19b5c710a0bafa091c1eda9b92ff1304cbfd.png) ---
**Author:** ![johnnywhippet](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/johnnywhippet/32/2803_2.png) [@johnnywhippet](https://forum.numer.ai/u/johnnywhippet)\ **Post date:** [September 29, 2021, 6:39pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/20 "2021-09-29T18:39:01Z")
Those appalling diagnostics have garnered two more silver medals. I don’t get it. ![image](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/4/4456d6ff8033707b3a543f0151632ad26cef34b9.jpeg) [Next page](https://forum.numer.ai/t/diagnostics-for-39/4155.md?page=2)