# Validation metrics example script vs website diagnostics

**URL:** <https://forum.numer.ai/t/validation-metrics-example-script-vs-website-diagnostics/6051>\
**Category:** Data Science\
**Created:** [January 19, 2023, 12:31pm UTC](https://forum.numer.ai/t/validation-metrics-example-script-vs-website-diagnostics/6051 "2023-01-19T12:31:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tessier\_ashpool](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/tessier_ashpool/32/2633_2.png) [@tessier\_ashpool](https://forum.numer.ai/u/tessier_ashpool)\
**Post date:** [January 19, 2023, 12:31pm UTC](https://forum.numer.ai/t/validation-metrics-example-script-vs-website-diagnostics/6051/1 "2023-01-19T12:31:31Z")

</div>

I have been playing around with validation metrics recently, been using `validation_metrics` from the example script [https://github.com/numerai/example-scripts/blob/master/utils.py](https://github.com/numerai/example-scripts/blob/master/utils.py) and I’ve noticed that the values which I get are very different from model diagnostic available on the website. For example validation sharpe is `0.83` on website, but calculated via the `validation_metrics` it’s `0.58`.  
Does anyone know, how are the metrics on the website calculated? How can I replicate the performance and risks metrics calculated on the tournament website locally?

---

<div class="post-metadata">

**Author:** ![chanes](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/chanes/32/2895_2.png) [@chanes](https://forum.numer.ai/u/chanes)\
**Post date:** [January 24, 2023, 12:36am UTC](https://forum.numer.ai/t/validation-metrics-example-script-vs-website-diagnostics/6051/2 "2023-01-24T00:36:06Z")

</div>

Hey bigcube. Can you send me a specific diagnostics ID you are having issues with? I just tried to reproduce my validation sharpe using the code from example-scripts and it matches exactly what I see on the website.

---

<div class="post-metadata">

**Author:** ![tessier\_ashpool](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/tessier_ashpool/32/2633_2.png) [@tessier\_ashpool](https://forum.numer.ai/u/tessier_ashpool)\
**Post date:** [January 25, 2023, 6:25am UTC](https://forum.numer.ai/t/validation-metrics-example-script-vs-website-diagnostics/6051/3 "2023-01-25T06:25:48Z")

</div>

I have tested the validation metrics for the example script vs what numbers are shown on the website.  
So to reproduce, go to [https://github.com/numerai/example-scripts](https://github.com/numerai/example-scripts) and run the `example_model.py`. The results are as following (I have changed only that we should calculate all metrics).

![Screenshot 2023-01-25](https://canada1.discourse-cdn.com/flex009/uploads/numerai/optimized/2X/5/594763176fdb8230043cbc6596e93d0f3a9c1d3d_2_690x22.png)

The model for which the validation data has been submitted is `preds_model_target_neutral_riskiest_50` and as you can locally calculated sharpe is `0.976964`  
but when I uploaded the validation file on the website I’ve got something like this

 ![Screenshot 2023-01-25 690x455](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/0/052faa39199d6e76e3acb7f1685de53dc18ee23f.png)

The sharpe is ` 0.9278` which is clearly different from the locally calculated one, same with other metrics. This is quite confusing, as I’m not sure what the reason is.

---

<div class="post-metadata">

**Author:** ![shatteredx](https://avatars.discourse-cdn.com/v4/letter/s/ee59a6/32.png) [@shatteredx](https://forum.numer.ai/u/shatteredx)\
**Post date:** [January 25, 2023, 2:21pm UTC](https://forum.numer.ai/t/validation-metrics-example-script-vs-website-diagnostics/6051/4 "2023-01-25T14:21:43Z")

</div>

Website diagnostics only calculates metrics on validation eras 857 to 961.

If you’re calculating validation metrics locally, it’s going to be different if you do it on the entire validation dataset. It should nearly match if you only calculate them on eras 857 to 961.

---

<div class="post-metadata">

**Author:** ![tessier\_ashpool](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/tessier_ashpool/32/2633_2.png) [@tessier\_ashpool](https://forum.numer.ai/u/tessier_ashpool)\
**Post date:** [January 25, 2023, 2:48pm UTC](https://forum.numer.ai/t/validation-metrics-example-script-vs-website-diagnostics/6051/5 "2023-01-25T14:48:12Z")

</div>

Indeed, that was the missing link - using the range from the plot, thank you!

![Screenshot 2023-01-25](https://canada1.discourse-cdn.com/flex009/uploads/numerai/optimized/2X/1/10f0b0b2b68358ca55b81edb0d2a4a755b8df9d4_2_690x18.png)