Echelon - a Numerai portfolio

[greyone]

Hello (Numerai) world!

We’re Echelon, a father-son duo who’s been participating as a team in Numerai. We started a year ago as a team of three who submitted predictions for the Signals tournament. With the passing of our algorithm developer in January, we pivoted from Signal model development to model investing. We are currently focused on Numerai Main Tournament model portfolio investment via NumerBay (Thanks, Res & CoE!).

We wanted to share a bit about our thinking, and invite you to follow along with our results (including the good, the bad and the ugly!).

Our thinking

We’ve gotten to this point through a lot of trial and error. On that path, we’ve started to define (and refine!) our beliefs. They are:

1. We believe in the Numerai data science community. It’s one of the most knowledgeable data science communities in the world.
2. We believe in Numerai. We’re comfortable investing our own money into a tournament where Numerai acts as the referee and the results are somewhat a black box.
3. We believe diversification will especially benefit staking on True Contribution (TC) due to how uniquely volatile and uncorrelated TC behaves.
4. We believe this portfolio can evolve to outperform the Numerai Main Tournament stake weighted average if we can employ insights from modelers and Numerai personnel. (This is one way that we will measure our ‘success’.)
5. We believe that this, in turn, will improve Numerai’s meta-model (only slightly; we’re certainly not Richie Rich) which is good for all of us.

We certainly don’t have all the answers, and don’t pretend to. If you have any feedback on the items above, we’d love to hear it.

How you can follow along

For transparency, we are creating a common Echelon name for all of our Numerai Main Tournament slots. They’ll each contain “ECHELON_” at the beginning. Rest assured that they’ll never be our own ML predictions, but predictions that we’ve purchased via NumerBay.

Periodically, we will report on some total risk characteristics – mainly the effect of diversification. Other than a bias towards some performance persistence, we do not currently have any alpha strategies. Over time, we believe alpha strategies will arise within the Numerai ecosystem as a result of modeler and Numerai personnel interactions and insights. We believe a fund within Numerai’s ecosystem will be a natural place for risk controlled tests and implementation of these strategies.

We also believe that cross modeler ensemble investing will offer benefits to the Numerai ecosystem. Our current Numerai model selection criteria include performance, Corr correlation, modeler methods, dialogue and community reputation.

We look forward to interacting with you and developing this approach. You can find us on RocketChat at @greyone (aka Gerry) or @aqsmith (aka Aaron). Excited to be a part of the Numerai community and can’t wait to see what’s next.

[restrading]

@greyone Thanks for sharing and for using NumerBay. I’m planning to add the following features soon to make things like this (diversification) easier:

1. (Essential) API for bulk download of performances of all models listed on NumerBay
2. (Optional) Leveraging the above to build a simple in-browser portfolio optimizer where users can import their own performance numbers and use the above data to select the best diversifying models to buy.

Do you have any addition / opinion on the above?

[by256]

Thanks for doing this openly. I’ve always wondered how a mini meta model of some of the best performing models would fare. Good luck! I’ll be following the ECHELON models and look forward to updates in the forum.

[rigrog]

@greyone, it sounds like you might be combining the the prediction files you purchase, to produce your own ECHELON_XXX prediction files. Rank averaging is one way that might be done.

Or, are you just staking on those predictions as-is? For those predictions offered on a “stake only” basis, where you don’t get any file, this would be the only way to go.

Or maybe… a little of this, a little of that? Just curious.

[greyone]

@rigrog , our current approach staking on predictions as-is. We do not foresee us being a driver of alpha additions. We do believe we can create additional ensemble options for modelers, either separately or in collaboration with other modelers, where their predictions remain secure from us, but we offer greater diversification and return exposure.

[aqsmith08]

Hey @restrading - Thanks for your response. Some quick feedback from me –

1. We have some scripts to hit the Numerai API to grab model information, and then do some data wrangling to estimate model value. This is definitely interesting for us, but we’re already doing some of this ourselves so I’m not sure who much we’d use it.
2. Hm, I may have to give this one some more thought. My initial reaction is that we do this via our scripts taking into account things like model performance, model correlation to other models, NumerBay purchase fees, and other things. My sense is that we likely wouldn’t be power users of that either, but I can totally see other NumerBay buyers would find that valuable.

How’s that for my two cents?!

[aqsmith08]

@by256 Thanks for your kind response. Excited to give this a try and we’ll definitely be updating this forum thread with results in the future.

[restrading]

@aqsmith08 thanks for the response. I understand you have your own ways to do these. I will deprioritize these for now.

However, I think it would be helpful if there’s an easy tool to generally incentivize diversification for buyers and not just performance-chasing.

[greyone]

@restrading The features you have outlined will be very helpful for the community. Competitive ecosystems inevitably foster some cat and mouse behavior…we’re not immune.

[zwk]

@greyone welcome to the jungle! I think FoF will be a really cool dimension to explore in order to activate our Numerbay market place, with more “non-data-scientist” / retail inflows.

Hopefully, with appropriate lending facilities, we can also imagine to create a “hedged” staking product in the future.

[greyone]

thank you BigCreeper!!

[qeintelligence]

Jeejj, lets hope many will follow! Ranked 3 now, LOL this is not Financial advice and no guarantees ofcourse

[greyone]

Echelon’s Numerai portfolio results

We are sharing the results of Echelon’s Numerbay model portfolio and some observations as they may be some benefit to the community.

Echelon ran a Numerbay-sourced Numerai model portfolio from rounds 310 to 394 (41 rounds). We had three objectives.

1. To establish a diversified approach to model portfolio selection.
2. To test for what we believed was performance persistence, both in Corr and TC.
3. To evolve towards a vehicle offering lower-risk performance-sharing for modelers with modeler-driven alpha judgment inputs.

Echelon’s model selection factors for the entire Numerbay Universe each week were: daily Corr and TC performance over 12 rounds, daily PCT’s of Corr and TC (for performance consistency), Sharpe ratios of Corr and TC, Model Corr and TC correlations, and qualitative modeler judgment.

We used Numerai’s Stake Weighted Index Performance (in orange) as a primary benchmark. Here is the universe/benchmark performance over the 41 rounds:

Here is Echelon’s performance vs the Stake-weighted Index:

Here is a comparison of Echelon’s results versus two other Numerai portfolio approaches (Crowdcent and ATOL) over the time period that all portfolio had results for (22 rounds: 329 to 394). We chose these two portfolios because their model ensembles are significant in stake size (25% of total Numerai stake) and we highly respect their approaches. In Round 394, Crowdcent’s 12 slot 154,191 stake made up 17.5% of total Numerai stake and ATOL’s 14 slot 65,531 stake made up 7.5% of total.

Here is a relative performance chart (portfolio vs stake-weighted benchmark) that more clearly illustrates the portfolio’s “non-beta” performance.

Comments:

During the first 2/3’s of Echelon’s activities, Echelon had success in selecting models that could at least match Numerai’s Stake Weighted index. Additionally, we were happy with the diversification the portfolio was achieving. After R349, Echelon had 4 consecutive negative periods of amounts that we considered out of bounds for the portfolio’s diversification and approach. While Echelon’s absolute and relative performance surged back after R369, this performance was largely achieved by high stake concentration on Paul (and later also on Redqueen, another ricricric model). Because heavy model-specific staking was not in our original intent, we have currently stopped staking on Numerbay models.

We still believe successful portfolio approaches will (and should!) develop that can benefit the modeler community.

We will remain active in this extraordinary community/endeavor in other ways. Appreciate you all for letting us non-modelers participate in this alien venture, and a huge kudos to ResTrading for his work on Numerbay.

Gerry (greyone) & Aaron (aqsmith)
2/9/2023

Appendix charts (for information and transparency):

[autratec]

Good job. Thanks for the update.