# Super Massive LGBM Grid Search **URL:** \ **Category:** Tournament\ **Created:** [June 15, 2023, 1:48pm UTC](https://forum.numer.ai/t/super-massive-lgbm-grid-search/6463 "2023-06-15T13:48:33Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![maul](https://avatars.discourse-cdn.com/v4/letter/m/9de0a6/32.png) [@maul](https://forum.numer.ai/u/maul)\ **Post date:** [June 15, 2023, 1:48pm UTC](https://forum.numer.ai/t/super-massive-lgbm-grid-search/6463/1 "2023-06-15T13:48:33Z")
# Introduction We at Numerai have spent the last couple of weeks conducting a new grid search on the Sunshine V4.1 data set. We have built hundreds of models with different hyperparameters on the V4.1 dataset focussed on target\_cyrus\_v4\_20 which we believe is the best single target at present for our hedge fund strategy. We are sharing the best of these grid searched results in terms of correlation and correlation Sharpe to enable users to benefit from the grid search and either use these results directly in their models or allow them to do more targeted searches of their own around the sweet spots. # Experimental Setup We did the grid search using the following parameters:- Features = all the features in the V4.1 dataset Target = target\_cyrus\_v4\_20 Algorithm = Scikit-learn API LGBMRegressor Hyperparameter ranges for the better results are shared below:- n\_estimators = 30k - 60k learning\_rate = 0.001 max\_depth = 5, 6, 7 num\_leaves = 2\*\*max\_depth - 1 colsample\_bytree = 0.1 # Results The first plot below contains the correlation and the second the correlation Sharpe computed using the out of sample predictions from era 578 to 1059 inclusive. The training period was from era1 to era 574 inclusive. ![](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/7/7701075d801ab6492986c72657e4a71e3afd5173.png) Table 1 below contains the 20 best correlation results. Table1 ![](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/f/fc6c8fb1de192da64ecee82aa10be10e7e51098f.png) Table 2 below contains the 20 best correlation Sharpe results. Table 2 ![](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/9/95a483fbdd394079f467f806999f9d78e87cb743.png) \*\*The plot below shows the cumulative correlation of 1. The sunshine recommended param model with learning\_rate = 0.001, n\_estimators = 20K and max\_depth = 6. 2. The best correlation model from the above table. 3. The 2 best correlation Sharpe models from the above table. ![](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/3/355455ad2e9c5e408dd4d6e9a9061d122e50e81f.png) # Alternate hyperparameters for less compute The above parameters require 6 hours to compute for a tree of max\_depth = 6 , a learning rate of 0.001 and n\_estimators = 100k on a 24 core processor. This may be a heavy compute burden for users. We show below parameters that work with lower compute using a learning rate of 0.01, n\_estimators = 20k and columnsample\_bytree = 0.1. This reduces the compute time to less than 2 hours. ![](https://canada1.discourse-cdn.com/flex009/uploads/numerai/original/2X/e/e44d09ab251e00fd880f56b6b3cebbe33391afeb.png) # Conclusion The above results are very competitive in correlation Sharpe space while slightly worse in terms of pure correlation but with significant compute saving. ---
**Author:** ![thomasxthomas](https://avatars.discourse-cdn.com/v4/letter/t/ebca7d/32.png) [@thomasxthomas](https://forum.numer.ai/u/thomasxthomas)\ **Post date:** [June 16, 2023, 1:48pm UTC](https://forum.numer.ai/t/super-massive-lgbm-grid-search/6463/2 "2023-06-16T13:48:46Z")
Is that possible to rerun the best hyper-parameters for 5 different random seeds and report the mean and standard deviation of Corr and Sharpe for each hyper-parameter? ---
**Author:** ![lcrmorin](https://avatars.discourse-cdn.com/v4/letter/l/f1d935/32.png) [@lcrmorin](https://forum.numer.ai/u/lcrmorin)\ **Post date:** [June 17, 2023, 12:11pm UTC](https://forum.numer.ai/t/super-massive-lgbm-grid-search/6463/3 "2023-06-17T12:11:43Z")
Came here to say that. Results are usually really sensitive to seed. Also I am pretty sure l1, l2 reg, boosting mode, allowing for regression on the last level can change the results. ---
**Author:** ![dzheng1887](https://avatars.discourse-cdn.com/v4/letter/d/f475e1/32.png) [@dzheng1887](https://forum.numer.ai/u/dzheng1887)\ **Post date:** [June 18, 2023, 7:51pm UTC](https://forum.numer.ai/t/super-massive-lgbm-grid-search/6463/4 "2023-06-18T19:51:18Z")
Has anyone tried using a log loss cost function instead of a regression? That’ll just be reducing the log loss on that where actuals are 0, 0.25, 0.5, 0.75, and 1 instead of just 0/1. Wondering if probability conversion helps with fitting due to the curvature in the cost function. It also has other interesting properties like 0.5 guess on any observation is the same cost regardless of the true label, kind of “standardizing” 0.5 as an acceptable placeholder guess. Also, the comparative improvement of going from the worst prediction to okay compared to going from okay to perfect is higher using the log loss cost function than squared error. This seems to incentivize a model to find more “average” parameters that benefits the overall group of assets more than an individual asset. To illustrate, look at the actual == 0 observation - 10x improvement to go from 0.999 to 0.5 for log loss, 4x improvement for sq err - 693x improvement to go from 0.5 to 0.001 for log loss, 250,000x improvement for sq err | | | predicted | | | | | | --- | --- | --- | --- | --- | --- | --- | | | log loss | 0.001 | 0.25 | 0.5 | 0.75 | 0.999 | | actual | 0 | 0.001 | 0.288 | 0.693 | 1.386 | 6.908 | | | 0.25 | 1.728 | 0.562 | 0.693 | 1.112 | 5.181 | | | 0.5 | 3.454 | 0.837 | 0.693 | 0.837 | 3.454 | | | 0.75 | 5.181 | 1.112 | 0.693 | 0.562 | 1.728 | | | 1 | 6.908 | 1.386 | 0.693 | 0.288 | 0.001 | | | | | | | | | | | | | | | | | | | | predicted | | | | | | | sq err | 0.001 | 0.25 | 0.5 | 0.75 | 0.999 | | actual | 0 | 0.000 | 0.063 | 0.250 | 0.563 | 0.998 | | | 0.25 | 0.062 | 0.000 | 0.063 | 0.250 | 0.561 | | | 0.5 | 0.249 | 0.063 | 0.000 | 0.063 | 0.249 | | | 0.75 | 0.561 | 0.250 | 0.063 | 0.000 | 0.062 | | | 1 | 0.998 | 0.563 | 0.250 | 0.063 | 0.000 | ---
**Author:** ![shatteredx](https://avatars.discourse-cdn.com/v4/letter/s/ee59a6/32.png) [@shatteredx](https://forum.numer.ai/u/shatteredx)\ **Post date:** [June 19, 2023, 3:32pm UTC](https://forum.numer.ai/t/super-massive-lgbm-grid-search/6463/5 "2023-06-19T15:32:04Z")
Cool experiment! The case for 20,000+ tree models seems dubious. Maybe a gain of +0.0010 Corr? Glad you guys showed the results for sub 20k tree models. More important question: how many trees for good TC? EDIT: OK fine, I will admit the 20k+ tree models are generally more stable at higher depths. ---
**Author:** ![svendaj](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/svendaj/32/3285_2.png) [@svendaj](https://forum.numer.ai/u/svendaj)\ **Post date:** [June 22, 2023, 7:28pm UTC](https://forum.numer.ai/t/super-massive-lgbm-grid-search/6463/6 "2023-06-22T19:28:19Z")
Thanks guys. Can you publish the tables for less compute version? ---
**Author:** ![svendaj](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/svendaj/32/3285_2.png) [@svendaj](https://forum.numer.ai/u/svendaj)\ **Post date:** [June 23, 2023, 5:18pm UTC](https://forum.numer.ai/t/super-massive-lgbm-grid-search/6463/7 "2023-06-23T17:18:40Z")
… also: are you considering grid search for XGBoost? Especially for low compute settings? ---
**Author:** ![thomasxthomas](https://avatars.discourse-cdn.com/v4/letter/t/ebca7d/32.png) [@thomasxthomas](https://forum.numer.ai/u/thomasxthomas)\ **Post date:** [June 23, 2023, 8:56pm UTC](https://forum.numer.ai/t/super-massive-lgbm-grid-search/6463/8 "2023-06-23T20:56:00Z")
Maybe this preprint can help? [[2303.07925] Robust incremental learning pipelines for temporal tabular datasets with distribution shifts](https://arxiv.org/abs/2303.07925) ---
**Author:** ![richai](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/richai/32/2493_2.png) [@richai](https://forum.numer.ai/u/richai)\ **Post date:** [June 26, 2023, 7:36am UTC](https://forum.numer.ai/t/super-massive-lgbm-grid-search/6463/9 "2023-06-26T07:36:47Z")
How did the approaches in your preprint work in May @thomasxthomas. Did you have models which reliably avoided the drawdown so many models experienced? Which approach worked best? ---
**Author:** ![gbrecht](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gbrecht/32/3509_2.png) [@gbrecht](https://forum.numer.ai/u/gbrecht)\ **Post date:** [June 26, 2023, 6:28pm UTC](https://forum.numer.ai/t/super-massive-lgbm-grid-search/6463/10 "2023-06-26T18:28:17Z")
Is there a reason: - a full grid search was done and not a more sophisticated method of hyperparameter search? - important lightgbm parameters like `min_data_in_leaf` or regularization was not included? ---
**Author:** ![mantz](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@mantz](https://forum.numer.ai/u/mantz)\ **Post date:** [July 6, 2023, 9:28am UTC](https://forum.numer.ai/t/super-massive-lgbm-grid-search/6463/11 "2023-07-06T09:28:52Z")
I don’t understand this experiments, since better results were published here: > [@Target Cyrus - New Primary Target](http://forum.numer.ai/t/target-cyrus-new-primary-target/6303): > > Overview 4 new target variations are being released on Numerai. There are 20D and 60D versions of each, for a total of 8 new targets. They will be released in the v4.1 dataset starting with the round opening on April 18. One of them, target Cyrus, will become the official target used for payouts in one month, beginning with the round opening on May 13. Along with this change, we are also implementing a change in the way correlation is calculated. This change weights your lowest and highest p… What is the reason why you perform a huge grid search and get worse results?