Is this new target going to be in the same column?
Also, my corr20v2 is like one quarter of corr20.
Is there a way of opting out of this?
Hi there,
I have downloaded the new dataset after I saw this topic and my model’s rmse is much worse using the same target_nomi_v4_20, same model. Is anyone seeing that as well? The previous dataset I was using was downloaded February 23.
Regards
1 Like
hi everyone, I’m having problems replicating the scores of corrv2 from the leaderboard using the code above, my correlations are not exactly the same. Have anyone succesfully implemented and tested the new metric?
Yes, me too. Most of my corr20’s which I was betting on have gone from 90% to 0%.
@master_key
We are getting very strange results with this new dataset, it look likes all the targets have changed after you added the cyrus target to it (mainly target_nomi_v4_20). Can you have a look, please?
Basically, I fit the same model on target_nomi_v4_20, using the same rows I had in the dataset I download in February 23 and get completely different results. Unfortunately, I overwrote the February dataset and can’t compare it against the new one.
Regards
I downloaded the train set just now to double-check (and the validation set yesterday to grab the latest eras w/ targets). Ignoring the new targets, I do not detect any changes in this data from data of weeks’ past, i.e. nothing that existed in the downloads 2 weeks ago has changed. Somebody on the discord was grumbling yesterday about some file corruption or something and couldn’t extract the data, and then Mike said he was fixing…you might re-download just to make sure if it really looks like the targets have changed, but much more likely you are doing something off (check the column headers) or your model results are much more stochastic than you realized using same data…
2 Likes
Although corr20 has a longer right tail, corr20V2’s left tail was pulled in quite a bit (likely a large contributing factor to the increased sharpes as well). The average corr20v2 score is lower than the average old corr20 score.
The correlation between corr20 and corr20v2 is quite high.
@master_key Is there still any work going into open sourcing some of the scoring pipelines?
(this post was edited to be the most up to date and removed the previous post to avoid confusion)
1 Like
You beat me to it - we are just triple checking things before we put it back on the website and add some more details to the post.
The scoring for the website actually had the same issue that was found in the correlation function in this original post, where the targets weren’t being centered properly.
So we’ve recalculated all of the historical scores, and you should see scores that are much more similar to the previous corr20 scores.
I see a 98% correlation between corr20 and corr20V2 rep for example.
It does look like the typical and best correlation reputations are expected to decrease, while the Sharpe of correlation tends to increase.
You might want to filter out reps from this analysis which have many missing rounds, as it makes it look like reputations are much closer to 0 than they are in practice.
5 Likes
After a lot of effort, I finally had a model that was performing really well on Corr20 and now you have messed it all up. Corr20v2 is much worse. I feel like I can not win here with these moving goalposts and constant changes for the worse. I am draining my stake.
You already have TC. I do not understand your motivation for redefining correlation to be “more like it”.
What was wrong with correlations > 0.015, that you have to actively prevent them?
Can the graphs in the thread be also updated?
@master_key corr20v2 is much lower than corr20 for every model which means you are automatically decreasing everyone’s payouts, why is that? Also, given you doing that, isn’t now time to allow users to set the corr weight to be higher the the current max of 1?
Regards
1 Like
@liborty Improving your corr20v2 may be as simple as retraining on the new target. As shown in the original post, retrained models should have similar corr and better Sharpe under the new scoring. Have you tried that?
Existing models were not trained to be good at the new Cyrus target so it is expected scores on the new target would be lower. After retraining scores should be much more similar, but with less draw-down risk!
They were correct initially actually so still valid
1 Like
I’m just thinking won’t this make things confusing for historic content? References on the forum/discord to CORR20 pre CORR20V2 are going to be talking about a different metric, and it won’t be obvious unless you know that the meaning of CORR20 changed. Also this is going to make our code confusing too, as we’re going to have to remember that our code referencing CORR20 pre this change were actually referencing CORR20V1. Wouldn’t it be clearer to keep referring to it as CORR20V2?
Also is this going to be the case for the API too, e.g. CORR20V2 will disappear from the API and will be queried via CORR20 instead? If so couldn’t that silently break people’s code?
2 Likes
Hi,
I thought that from May 13th the training dataset would use Cyrus as the column “target” but it seems, at least for the int8 version that I have just downloaded, that Nomi is still the main target (Corr 1 with “target” column).
Did I miss anything? Isn’t Cyrus being used from this round?
Thanks
1 Like
You’re right - this didn’t get changed on time but this round is scoring on Cyrus. Sorry for the confusion.
Are both TC and CORR20 payouts based on target_cyrus_v4_20 beginning with round 484?
TC doesn’t have a target.