On September 20, 2024, we will be implementing a 15% churn threshold for all Signals submissions. Any submission that breaches this threshold will not be paid. This does not apply to Numerai or Crypto.
Churn is a statistic describing how the alpha scores of a signal changes over time. We recently open-sourced the code we use to calculate churn in Signals Diagnostics. You can find it here.
If a Signals submission has high churn, then Numerai can’t trade the signal easily. Many models built on Numerai data have low churn organically, but Signals Churn is very high. Most Signals models have > 20% week-over-week churn:
We know that this negatively impacts the churn of the Signals Meta Model because the average individual churn of Signals models is nearly 70% correlated with the Signals Meta Model Churn:
Signals Meta Model churn too high to be useful to Numerai:
To lower the churn of the Signals Meta Model, we must lower the churn of all Signals - so we are implementing a strict churn threshold that operates as follows:
- Any model that has not submitted in the previous week will have it’s stake set to 0
- Any model that does not submit weekly will naturally cause high churn in the Meta Model
- When you upload a new submission we:
- Calculate churn with respect to each of this model’s submissions from the previous week
- Check if this submission has >= 15% churn with respect to any of this model’s accepted submission in the previous week. If this submission breaches the churn threshold, it’s stake is set to 0.
FAQs
Is 15% too low?
No. Our v43.cyrus_plus_teager model has never breached 15% churn, so we know this is an achievable level of churn that will guarantee a sufficient reduction in overall Signals Meta Model churn.
How do I know what my churn is?
We have open-sourced the churn calculation we use in diagnostics so that you can calculate it yourself. Soon, we will use this code to display the churn on the Signals website. Any submissions that breaches the threshold will be highlighted as “high churn” in the website. Once the threshold is implemented, we will begin setting the stake to 0 for these submissions.
When will this churn threshold take effect?
On September 20, 2024 the threshold will be 15%.
What about Numerai models?
This does not affect Numerai models as they cannot control their churn level due to the obfuscation of the dataset. Instead, we have crafted a dataset that naturally results in lower-churn models. Signals, on the other hand, can easily reduce their churn because Signals models can easily calculate it. Signals models can be trained to minimize churn just the way we did with our v43.cyrus_plus_teager model.
What if everyone breaches the threshold?
The payout factor will heavily incentivize users to submit reliably low-churn models so they are not dropped out of the staking pool. Over time, the likelihood of everyone simultaneously breaching the threshold will drastically diminish.
