# Diagnostics for #39

**URL:** <https://forum.numer.ai/t/diagnostics-for-39/4155>\
**Category:** Data Science\
**Created:** [September 19, 2021, 8:40am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155 "2021-09-19T08:40:17Z")\
**Posts on this page:** 5\
**Page:** 4

<div class="post-metadata">

**Author:** ![luee](https://avatars.discourse-cdn.com/v4/letter/l/c67d28/32.png) [@luee](https://forum.numer.ai/u/luee)\
**Post date:** [January 29, 2022, 7:44pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/62 "2022-01-29T19:44:59Z")

</div>

> [@kowalot](#):
>
> - validation data are only used for early stopping (different structure)

This part is in fact data leakage and could lead to such much overestimated performance. In live you will not be able to perform early stopping on the live data, but that is essentially what you are doing here

---

<div class="post-metadata">

**Author:** ![kowalot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/kowalot/32/3444_2.png) [@kowalot](https://forum.numer.ai/u/kowalot)\
**Post date:** [January 29, 2022, 7:56pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/63 "2022-01-29T19:56:59Z")

</div>

Anyway as @wigglemuse , as active member of the forum/chat (which I read probably the whole 😉 ) are skeptical I will look into it. I could make a mistake, which is quite possible, but at this moment I don’t see where. I used/tried a lot of tricks mentioned here by the community but also couple own techniques as well. For sure I want to be far from “cheating”.

---

<div class="post-metadata">

**Author:** ![kowalot](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/kowalot/32/3444_2.png) [@kowalot](https://forum.numer.ai/u/kowalot)\
**Post date:** [January 29, 2022, 9:13pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/64 "2022-01-29T21:13:32Z")

</div>

let me add results with doing early stoping on “unused”/untouched training data. I cut some portion of continous eras.

```auto
                       mean std sharpe max_drawdown apy    
max_drawdown 0.046483 0.023462 1.981191 -0.007393 815.262085  

```

As expected this part has even better results in term of corr. Of course model lost some of the signals from cut part.

Validation without any connection to the training process.

 ![Max_drawdown_eraly_stoping_on_train](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/f/f0e7cbf628c641b068532b936f721ccc9cfa1a04.png)

Feature exposure still quite good but lack of data suffered it a bit.

---

<div class="post-metadata">

**Author:** ![luee](https://avatars.discourse-cdn.com/v4/letter/l/c67d28/32.png) [@luee](https://forum.numer.ai/u/luee)\
**Post date:** [January 29, 2022, 9:47pm UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/65 "2022-01-29T21:47:44Z")

</div>

> [@kowalot](#):
>
> My previous models have several negative val eras I changed the architecture to get rid of them.

That would be the other form of data leakage that I see, it seems like you have tweaked your model specifically for your out-of-sample performance. While the model never explicitly sees that data, it is built with performance on that very specific sample in mind. I would go through the same exercise of building my model to fit the last 100 eras of training set the best, then use that model once on the validation set. This will probably be a closer estimate of your live performance and give you an idea of the impact of data-mining bias in your current process.

---

<div class="post-metadata">

**Author:** ![maxchu](https://avatars.discourse-cdn.com/v4/letter/m/dec6dc/32.png) [@maxchu](https://forum.numer.ai/u/maxchu)\
**Post date:** [January 31, 2022, 12:35am UTC](https://forum.numer.ai/t/diagnostics-for-39/4155/66 "2022-01-31T00:35:20Z")

</div>

I am also curious if you have done any feature naturalization as you have very skewed corr on different eras.

[Previous page](https://forum.numer.ai/t/diagnostics-for-39/4155.md?page=3)