Feature request - Easy comparison of model performance

[nyuton]

I often struggle to compare performance of my models. Comparing CORR and MMC model by model is very time consuming.

I found the rainbow chart on the models page useful, but many of my models don’t have 20 weeks of history and I would like to compare them on a CORR + 2xMMC basis as well.

I would suggest to improve the rainbow chart with the following features.

• add a “from round” input field. So that selected models are displayed and compared for the selected rounds only. (Less than 20 round) This allows easy comparison of new models on the cumulative chart, which have a few rounds of history only.

• allow CORR+MMC and CORR+2xMMC options as well. These are the metrics we are paid for, thus these give the most comprehensive information.

Thank you

Peter

[jacob_stahl]

It would also be nice to compare our models with the metamodel’s CORR, so we can use it as a benchmark.

[nyuton]

@degerhan 's script is very helpful for this purpose.
It would be great if we could get this on the homepage:

Sharpe and Sortino ratios on live performance of your models

As numerai models only earn or burn when the round is resolved, you may be ignoring the information content of the 20 daily scores in between. The daily gyrations of a model is similar to the daily account value of a buy-and-hold brokerage account. Yes, you don’t realize profit or loss until you sell, yet a slow and steady climb might be more deserving of your hard earned NMRs. More critically, historical daily scores are a high resolution indicator of your model’s volatility under different s…

[aventurine]

Is this what you might be looking for? https://dashboard.numeraipayouts.com/

[bvmcheckking]

This is great Aventurine, are you the developer behind this? Only thing missing for me is CORR + 2x MMC

[nyuton]

I’m aware of this dashboard. It’s pretty good, but it doesn’t have cumulative returns. It’s still hard to compare return over let say 10 round. One model is better is some rounds, another model in other rounds. The cumulative return counts most…

[aventurine]

Not me. I believe @ia_ai did it?

[ml_is_lyf]

Did you see my post a few weeks back? I added a notebook that pretty much does all this:

Notebook to visualize historic model performance and see effect of different MMC multipliers and payout factors

The compare model graphs on our user pages are really helpful for getting an idea of how our models are performing. But they only show cumulative scores, so we don’t see the compounding effect. Also, we don’t see the combined performance of our stake on CORR and MMC, as there is no option to plot CORR+MMC. To bridge this gap I made a notebook to do all of the above. You can tabualise how your model is performing like so: pc.tabualise_performance(model_name="ml_is_lyf", start_round=251, inclu…

If your point is though that this kind of functionality should be implemented into the website as its easier to access, I agree with that, as you’ll only come across my notebook if you know what you’re looking for.

[ia_ai]

Just saw this (about 7 months late )

https://dashboard.numeraipayouts.com/ is the result of two community projects:

• Data from my daily scores collection pipeline (with a simple CSV output). It is a part of my old dashboard project.

• The cool dashboard developed by @ceunen

So if you need to calculate additional stats like 2xMMC and cumulative returns, you can just download the CSV and play with it. The collection pipeline usually finishes around 30-40 mins after you see the daily-scores bot message in RocketChat.