I often struggle to compare performance of my models. Comparing CORR and MMC model by model is very time consuming.
I found the rainbow chart on the models page useful, but many of my models don’t have 20 weeks of history and I would like to compare them on a CORR + 2xMMC basis as well.
I would suggest to improve the rainbow chart with the following features.
add a “from round” input field. So that selected models are displayed and compared for the selected rounds only. (Less than 20 round) This allows easy comparison of new models on the cumulative chart, which have a few rounds of history only.
allow CORR+MMC and CORR+2xMMC options as well. These are the metrics we are paid for, thus these give the most comprehensive information.
I’m aware of this dashboard. It’s pretty good, but it doesn’t have cumulative returns. It’s still hard to compare return over let say 10 round. One model is better is some rounds, another model in other rounds. The cumulative return counts most…
Did you see my post a few weeks back? I added a notebook that pretty much does all this:
If your point is though that this kind of functionality should be implemented into the website as its easier to access, I agree with that, as you’ll only come across my notebook if you know what you’re looking for.
So if you need to calculate additional stats like 2xMMC and cumulative returns, you can just download the CSV and play with it. The collection pipeline usually finishes around 30-40 mins after you see the daily-scores bot message in RocketChat.