# Benchmark Models **URL:** \ **Category:** Announcements\ **Created:** [October 28, 2023, 9:04pm UTC](https://forum.numer.ai/t/benchmark-models/6754 "2023-10-28T21:04:49Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![master\_key](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/master_key/32/3343_2.png) [@master\_key](https://forum.numer.ai/u/master_key)\ **Post date:** [October 28, 2023, 9:04pm UTC](https://forum.numer.ai/t/benchmark-models/6754/1 "2023-10-28T21:04:49Z")
![](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/9/942d12f1abacbd4315a781ce81719810b1101071.png) Numerai develops new datasets and new targets to help our data science community build better models. Numerai builds models on each new target and data release. Today, we will begin giving out the predictions for all of these models, and details about how they are created. # Why? **New User Acceleration** Numerai has a steep learning curve. After you make it through the tutorial notebooks, you are left with several datasets, many targets, and many modeling options. There are an unlimited number of experiments you’ll want to run as you begin your journey to the top of the leaderboard. With benchmark models, you can immediately see how well different combinations of data and targets do. I think you’ll find that exploring these models and their predictions and subsequent performance will inspire even more ideas for new models you can build yourself. **Better Stake Allocation** If you’re a returning user and you’re a few updates behind, you can see at a glance if your model is still competitive, or if you’d be better off staking on one of the newer benchmark models until you have time to catch back up. **A Meta Model of Meta Models** Some users may not have the resources to train large competitive cutting-edge models themselves. However, by just downloading targets, the Meta Model predictions, and Benchmark Model predictions, it may still be possible to recognize that the Meta Model is underweight some types of models, or you might be able to find that certain targets ensemble especially well together, or you might have a strong belief that one target will outperform into the future. You can explore all of these possibilities yourself and even submit and stake on these ensembles with minimal resource requirements. # Where? Go to [numer.ai/~benchmark\_models](http://numer.ai/~benchmark_models) to see a list of models and their recent performance. Go to the [docs](http://docs.numer.ai/numerai-tournament/benchmark_models) to see more details about how they are made. ![](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/9/99908941c44aee643ec7feb01fd527fe5a44a38d.png) The validation and live predictions are available through the [api](https://github.com/uuazed/numerapi). > pip install numerapi ```auto from numerapi import NumerAPI napi = NumerAPI() napi.download_dataset("v4.2/validation_benchmark_models.parquet", "validation_benchmark_models.parquet") napi.download_dataset("v4.2/live_benchmark_models.parquet", "live_benchmark_models.parquet") ``` There is now a dotted line on your account page’s score charts to directly compare yourself with the benchmark models account. ![](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/b/bf95f3e6255d2d0b46b7caea9fcd2272d778b803.png) Happy Modeling ---
**Author:** ![numerologist](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/numerologist/32/2824_2.png) [@numerologist](https://forum.numer.ai/u/numerologist)\ **Post date:** [October 29, 2023, 3:44am UTC](https://forum.numer.ai/t/benchmark-models/6754/2 "2023-10-29T03:44:40Z")
Thanks for the hard work, @master_key and Numerai team. Would it be possible to add a toggle switch (next to “Cumulative” in “…”) to compare user/model performance to MetaModel instead of example models? As a not-very-new user, I’d be interested to see how I perform versus the competition and whether I underperform/contribute to the fund. ---
**Author:** ![degerhan](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/degerhan/32/3449_2.png) [@degerhan](https://forum.numer.ai/u/degerhan)\ **Post date:** [October 29, 2023, 7:53am UTC](https://forum.numer.ai/t/benchmark-models/6754/3 "2023-10-29T07:53:49Z")
Thank you @master_key very helpful. Is v42\_example\_preds a rename of the former 20k tree lg\_lgbm\_v42\_cyrus20? The docs say v42\_example\_preds is a standard model (I assume that means 2k trees), looking to understand for benchmark continuity. ---
**Author:** ![master\_key](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/master_key/32/3343_2.png) [@master\_key](https://forum.numer.ai/u/master_key)\ **Post date:** [October 29, 2023, 4:33pm UTC](https://forum.numer.ai/t/benchmark-models/6754/4 "2023-10-29T16:33:41Z")
Yeah that’s correct it’s a rename. And all of the benchmark models (still) have 20,000 trees. ---
**Author:** ![taori](https://avatars.discourse-cdn.com/v4/letter/t/ebca7d/32.png) [@taori](https://forum.numer.ai/u/taori)\ **Post date:** [October 30, 2023, 9:17am UTC](https://forum.numer.ai/t/benchmark-models/6754/5 "2023-10-30T09:17:49Z")
This is a bold move, I like it. ---
**Author:** ![zoliveres](https://avatars.discourse-cdn.com/v4/letter/z/ad7895/32.png) [@zoliveres](https://forum.numer.ai/u/zoliveres)\ **Post date:** [November 2, 2023, 8:09am UTC](https://forum.numer.ai/t/benchmark-models/6754/6 "2023-11-02T08:09:39Z")
@master_key Can you specify what the `rank_keep_ties_keep_na` function does in the `rank_gauss_pow1` function? I’ll be better put it in the Documentation. I found a similar funtion in the numerai-tools repo, is this what you are using? > ---
**Author:** ![danzell](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/danzell/32/3473_2.png) [@danzell](https://forum.numer.ai/u/danzell)\ **Post date:** [November 12, 2023, 8:47am UTC](https://forum.numer.ai/t/benchmark-models/6754/7 "2023-11-12T08:47:56Z")
Would be nice to have a functioning example ✌ ---
**Author:** ![danzell](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/danzell/32/3473_2.png) [@danzell](https://forum.numer.ai/u/danzell)\ **Post date:** [November 17, 2023, 9:00am UTC](https://forum.numer.ai/t/benchmark-models/6754/8 "2023-11-17T09:00:59Z")
I’m still confused. What do you exactly mean by: > **Ensembles** > > All of the ensembles use the following steps: > > 1. gaussianize each of the predictions on a per-era basis > 2. standardize to standard deviation 1 > 3. dot-product the predictions with a weights vector representing the desired weight on each model > 4. gaussianize the resulting predictions vector, and neutralize if there are any features to neutralize to It would be super helpful if you could provide an example 🤓 and share underlying code pls. ---
**Author:** ![tessier\_ashpool](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/tessier_ashpool/32/2633_2.png) [@tessier\_ashpool](https://forum.numer.ai/u/tessier_ashpool)\ **Post date:** [November 17, 2023, 1:39pm UTC](https://forum.numer.ai/t/benchmark-models/6754/9 "2023-11-17T13:39:53Z")
But they did provide the code. ```auto def gauss_pred(self, X: pd.DataFrame, ensemble_cols, weight_vector): for col in X[ensemble_cols]: if "era" in X.columns: X[col] = X.groupby("era", group_keys=False)[col].transform( lambda s1: rank_gauss_pow1(s1) ) else: # check X contains only a single era assert 1800 < X.shape[0] < 6000 X[col] = rank_gauss_pow1(X[col]) return X[ensemble_cols].dot(weight_vector) ``` as for the `rank_keep_ties_keep_na` method, I imagine it is something like this. for keeping ties use method average and instead of len(s.dropna()) do just len(s) or s.count() so in essence looks something like this ```auto def rank_gauss_pow1(s: pd.Series) -> pd.Series: # do rank-normalize # s_rank = rank_keep_ties_keep_na(s) # s_rank = (s.rank(method="average") - 0.5) / len(s.dropna()) s_rank = (s.rank(method="average") - 0.5) / s.count() # gaussianize s_rank_norm = pd.Series(scipy.stats.norm.ppf(s_rank), index=s_rank.index) # Standardize to 1 std result_series = s_rank_norm / s_rank_norm.std() return result_series ``` ---
**Author:** ![nasdaqjockey](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/nasdaqjockey/32/2933_2.png) [@nasdaqjockey](https://forum.numer.ai/u/nasdaqjockey)\ **Post date:** [March 10, 2024, 1:23pm UTC](https://forum.numer.ai/t/benchmark-models/6754/10 "2024-03-10T13:23:06Z")
It would be good if this was updated for MMC now.