Define “out of distribution” as a mismatch between the statistics used for training and production (i.e. non-stationary data). Define “OOD detection” as a task for measuring the similarity or dissimilarity between the training data and novel data.
Suppose I had a signal for how “out of distribution” the current era was. How can it be utilized to improve my current submission?
Allow users to score each of their predictions according to a confidence score (0, 1). When calculating COR values, weight each prediction according to it’s confidence.
- The current era has statistics which closely matches the training set. Competitor submits with overall high confidence to reap expected benefits of high correlation.
- The current era has statistics that doesn’t match anything they have seen before in the training set. Competitor submits with overall low confidence to indicate that their model’s performance is undetermined and therefore risky.
Sometimes the right answer is “I don’t know”, and it would be a valuable signal to have when creating the meta-model. Additionally, as a Competitor, I feel it would be valuable to have a mechanism to automatically adjust stake down according to volatile era statistics.