I apologize for the probably naive question, but I’ve just finished reading the tournament rules, and I’m curious about the gap between submitter incentives and numer.ai’s incentives, and the possibilities for semi-brute-force participation.
To recap my understanding, the tournament requires that each participant submit a set of predictions for n rows of data (where n is approximately 45k at the moment and where predictions are a probability value in the range [0, 1]). If the predictions are “good” (i.e. they have a low log-loss at the end of the month), numer.ai pays small amounts of money to the participant. If the predictions are “bad”, there is no payout.
In theory, numer.ai at some point starts trading real money on the predictions (or the cumulative predictions) of users who have shown to be “good” at predicting.
Now, an obvious problem already is how numer.ai determines which participants they should trust for making real trades; an evaluation period (i.e. a long period of time of running the tournament but not trading real money) is (slightly) expensive (in direct proportion to the duration of the evaluation period) because numer.ai must pay winners but themselves do not trade based on those strategies.
But a slightly less obvious problem is that submitters have no disincentive to spam numer.ai and try to get a payout in the first month, even if they are unable to repeat their successful predictions later on (and even if numer.ai never trusts them or trades on their predictions).
Fortunately for numer.ai, true brute-forcing is infeasible; even if one simply assigned a probability of either 0 or 1 to each row, we have 2^n (i.e. 2^45000) possible inputs, so autogenerating a new account and submitting a new prediction for each possible outcome is truly infeasible. (Some of these inputs will not be valid because they will not have “concordance”, though how concordance is actually evaluated or defined, or how many of the possible inputs it will reject, is not very clear to me.)
But clever submitters can (and probably do) generate hundreds or thousands of predictions by varying assumptions or weights in their models and submitting them separately to numer.ai. This behavior is incentivized by the payout scheme.
Unfortunately, this behavior compounds the “which participant do I trust?” problem for numer.ai, because it creates a larger pool of participants. Numer.ai is probably trying to counter through the reputation bonus–this encourages even spammy participants to be consistent about submitting predictions from the same models from the same accounts, and so (similarly to how they must decide whom to trust when actually trading real money) they pay out extra to users who have shown long-term predictive consistency and accuracy.
So, my question: Is this a problem? One could argue that this is simply working as intended, and that numer.ai is happy to eat the (very small, in real terms) loss in paying spammy models small bonuses for one or two months of success in exchange for generating very real predictions from long-term consistent users. But the misalignment of incentives still seems somewhat worrisome; submitters have an incentive to submit as much as possible, which is zero-sum from numer.ai’s perspective–spammy predictions from young accounts generate payouts but no earnings for the fund. Or am I missing something?