BTW are your ‘scores’ for your classic tourney data pure corr values or are they corr/mmc combos?
Since you clearly have the data loaded and available, I wonder if you might be willing to go further with this.
I suspect I’m not alone in hoping for positive scores each week, and that I value any positive final score greater than any negative final score. So in the wake of your latest post I’m curious:
If early scores are negative, what chance do I have that they’ll turn positive? And of course the opposite: if early scores are positive, what are the chances they’ll go negative? We could call this the “outcome reversal” probability. It is of course 0% on the final day of the round, but what is its value on day 15, 16, etc?
At what day during the round do most (say 95%) of the daily scores lie on the same side of zero as their final day counterparts? Aka, if my score is negative on day 15, what are the odds it will still be negative on the final day too? I guess this is just 1.0 minus the outcome reversal odds, so it’s really the same number.
Does the negative skew on early scores still exist if you separate the data into two chunks: final scores that were net positive (earns) and final scores that were net negative (burns)? Testing this would help support (or not) your intriguing suggestion that perhaps the negative skew is because our predictions get better as we get close to what we’re trying to predict.
This is pure corr. MMC tends to have less volatility, but is an interesting analysis on its own. Also, all good questions that hopefully I/we can answer soon. All of the code for this post is public in this notebook if you want to play around: Google Colab
Here are two really simple plots (using data from your jrai model) where I estimate the probability for each scoring day of the final score being a complete reversal of the daily reported score. In other words, if on day 1 the scores are positive, what are the odds that on day 20 the score will be negative? And so on throughout all 20 days.
Here is the plot for final scores that were positive:
And here is the plot for final scores that were negative:
They seem pretty similar on the whole. Here are my takeaway points:
There is roughly a 35% chance that your final round score will have the opposite sign of your day 1 score (aka a day 1 burn has a 35% chance of becoming a day 20 earn, and vice versa)
By approximately scoring day 11, you’re down to a 10% chance of a complete reversal.
I’d like to continue this analysis with magnitude of the correlation in mind. Aka, I suspect that higher absolute value corr values (either pos or neg) will result in less fate swapping by the end.
I would love to see some sort of marker of uncertainty added to the UI for the “out-rounds” that considers +/- 1 standard deviation scores given how many days are left in the round. Something like this:
Now that we have many more completed rounds with daily TC scores, this daily difference from resolution analysis seems like it can be extended to that metric. And, we also have percentile scores in the API, which is something I had wanted to look at for a while.
In sum, I’ve found that (on average across most models, but there are always exceptions) your TC percentile early on in a round is closer to your final day TC percentile as compared to that of your CORR percentile early on in a round relative to its final day percentile. In other words, if your first day TC score is in the 50th percentile, on average your final TC score will likely end up in the 30% - 70% range. However, if your first day CORR score is in the 50th percentile, on average your final CORR score will likely end up in the 15% - 85% range. I haven’t taken a closer/rigorous look at the tails, but I imagine there is different behavior there (i.e. if you start in the 100th percentile, will the daily paths be very different?)
Here are charts for all of integration_test’s rounds for TC percentile values and CORR percentile values:
