Does NMR Staking encourage improved models?

numeraire

#1

In my opinion, the confidence metric feels like it means nothing. It is an arbitrary value, and whether my model barely does better than random, or is able to predict every result with perfection, I will still get my payout at whatever confidence I set.

I’d like to propose a change to the system, such that “confidence” refers to how much of an improvement your model will be over random. If your model does that well or better, you get a payout, but if it does worse than your bet (even if better than random) the stake is destroyed. This would allow Numerai to use the stake as an actual measure of the data scientists confidence in their model’s performance, rather than confidence that it will be some amount better than random.

Payouts would then work in reverse, with the stakes for the best performance being evaluated first, and then moving down the list. Does this sound like a good idea? Or have I missed some detail of the system that makes the current way effective?


#2

It doesn’t mean nothing. Yes, it isn’t really directly interpretable as a standalone value, but when taking into account the expected payout and the market value of NMR, you can convert it into an implied probability/confidence as is done here: https://numerai.000webhostapp.com/leaderboard/68.

The confidence value is deliberately placed at odds with the size of the payout so that you are pushed to provide a realistic confidence rating for your model.

And Numerai doesn’t necessarily care about how much better than chance a model performs. If a model can consistently perform better than chance, then it’s predictions can be used to make money regardless of how small the margin over chance is. The main competition already pays out for how good the log loss of your model is, so if you have a model that can beat chance by a wide margin, you’ll make good money there. The staking competition is about models being consistently better than chance. The more confident you are in your model and the more often it has outperformed chance, the higher you can set C to guarantee payout before the pool is exhausted.


#3

I don’t agree… People are using confidence just to climb up the possibility to be paid (due to the sorting on it). I still have some trouble in thinking the confidence value as a true confidence: I see it as a multiplier for my payout, but keeping it enough high to be sure to be in the first positions