# Numerai Tournament Example code using Pytorch NN and Optuna **URL:** \ **Category:** Tournament\ **Created:** [December 17, 2021, 7:56am UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639 "2021-12-17T07:56:28Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![meaten12121](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@meaten12121](https://forum.numer.ai/u/meaten12121)\ **Post date:** [December 17, 2021, 7:56am UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/1 "2021-12-17T07:56:28Z")
Hi, I dived into this numerai community nine months ago and keep developing my tournament code. There are many example scripts from this community, but I wanted to share my code base to get feedback from this community. Hope some newcomers find this helpful. > **[GitHub - meaten/numerai\_NN\_example: Numerai tournament example scripts using NN and...](https://github.com/meaten/numerai_NN_example)** > > Numerai tournament example scripts using NN and optuna Features: - era-boosted train, time-series cross-validation - era-batches training - model hyperparameter tuning on pytorch NN and GBDT model - several tips on Numerai Forum are also included Thanks! ---
**Author:** ![objectscience](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/objectscience/32/2913_2.png) [@objectscience](https://forum.numer.ai/u/objectscience)\ **Post date:** [December 17, 2021, 3:50pm UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/2 "2021-12-17T15:50:30Z")
Very nice work!. I’m going to ping the CoE on this to see if it will qualify for a bounty. I know for me personally, the day I can actually afford a couple of good GPU’s, this is going to be really useful. ---
**Author:** ![aventurine](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/aventurine/32/2825_2.png) [@aventurine](https://forum.numer.ai/u/aventurine)\ **Post date:** [December 17, 2021, 6:42pm UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/3 "2021-12-17T18:42:46Z")
Very nice. Also love the model name. Might want to try and fix the model link though because its not taking into account the \_ and its going to one of my models [Numerai](https://numer.ai/emerald) and not [Numerai](https://numer.ai/emerald_) 🙂 ---
**Author:** ![meaten12121](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@meaten12121](https://forum.numer.ai/u/meaten12121)\ **Post date:** [December 18, 2021, 6:45am UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/4 "2021-12-18T06:45:15Z")
Thanks for your comment! Bounty? I didn’t know that system. I appreciate it if you ping the CoE! The example models I added are simple and don’t need good GPUs. (about 1GB GPU memory consumption) I hope you will get some after this significant GPU shortage. ---
**Author:** ![meaten12121](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@meaten12121](https://forum.numer.ai/u/meaten12121)\ **Post date:** [December 18, 2021, 6:49am UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/5 "2021-12-18T06:49:58Z")
Sorry for the mistake on url parsing 😨 I already fixed it. ---
**Author:** ![gbrecht](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gbrecht/32/3509_2.png) [@gbrecht](https://forum.numer.ai/u/gbrecht)\ **Post date:** [December 24, 2021, 8:01am UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/6 "2021-12-24T08:01:15Z")
I use vast.ai to use cloud GPUs. Even with normal GPU prices there is economically no reason to get one ourself. I wrote a forum post on my setup. Please give me a ping if you want some more details. ---
**Author:** ![stoicism](https://avatars.discourse-cdn.com/v4/letter/s/4bbf92/32.png) [@stoicism](https://forum.numer.ai/u/stoicism)\ **Post date:** [December 24, 2021, 9:52am UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/7 "2021-12-24T09:52:18Z")
Hi mate, many thanks for sharing your hard work with us. Can you please explain what era-boosted training and era-batches training is? It would be booster shot for the newcomers! ---
**Author:** ![gbrecht](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gbrecht/32/3509_2.png) [@gbrecht](https://forum.numer.ai/u/gbrecht)\ **Post date:** [December 24, 2021, 9:57am UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/8 "2021-12-24T09:57:13Z")
Era Boosting is - Training on all eras - In sample prediction of all eras - Selecting a subset of all eras to train on based on in sample performance (usually bottom half) - Resume training with only the selected eras Era Batches apply to neural networks and there the common sense is that it is good to train in batches equivalent to eras. I.e. forward for one era and then backward. and then the next era and so on. ---
**Author:** ![stoicism](https://avatars.discourse-cdn.com/v4/letter/s/4bbf92/32.png) [@stoicism](https://forum.numer.ai/u/stoicism)\ **Post date:** [December 24, 2021, 10:01am UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/9 "2021-12-24T10:01:21Z")
So instead of treating batch-size as a hyper parameter, the era sizes (number of data points in each era) are assigned as the batch size, is that correct? Thanks for the detailed response @gbrecht ! ---
**Author:** ![gbrecht](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/gbrecht/32/3509_2.png) [@gbrecht](https://forum.numer.ai/u/gbrecht)\ **Post date:** [December 24, 2021, 10:02am UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/10 "2021-12-24T10:02:42Z")
That is correct. I personally use pytorch where you define a DataSet class and it is dead easy to return an era as the next batch. With keras I never got it to work. And I am sure it is super easy in JAX but despite @jrb pep talk I have never gotten something to run… ---
**Author:** ![stoicism](https://avatars.discourse-cdn.com/v4/letter/s/4bbf92/32.png) [@stoicism](https://forum.numer.ai/u/stoicism)\ **Post date:** [December 24, 2021, 10:06am UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/11 "2021-12-24T10:06:15Z")
Thanks for the clarification. One less hyperparameter to worry about. Although, I am interested to see if treating batch-size as a hyperparam yield something equivalent to era size or not. ---
**Author:** ![jrb](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/jrb/32/3472_2.png) [@jrb](https://forum.numer.ai/u/jrb)\ **Post date:** [December 24, 2021, 5:15pm UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/12 "2021-12-24T17:15:17Z")
> [@gbrecht](#): > > With keras I never got it to work. If you’re using `tf.keras`, subclassing [`tf.keras.utils.Sequence`](https://www.tensorflow.org/api_docs/python/tf/keras/utils/Sequence) is the way to go. The next method I’ll describe is more universal and works with `tf.keras`, Tensorflow and JAX. > [@gbrecht](#): > > And I am sure it is super easy in JAX but despite @jrb pep talk I have never gotten something to run… If you’re using Tensorflow without the keras training loop, or even if you are using keras: Start by converting your data into a [RaggedTensor](https://www.tensorflow.org/guide/ragged_tensor) first. You can then turn the RaggedTensor into a [`tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) using the `from_tensor_slices` method. And then, Bob’s your uncle! If you’re using JAX, use the [`as_numpy_iterator`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#as_numpy_iterator) method on the `tf.data.Dataset` object. One thing to bear in mind is that the JAX jit caches its traces based on the shapes of the tensors passed in. So, if you’re using eras as batches with the jit, your first epoch is going to be super slow because it has to jit your training loop for every era (because they all have different number of rows). This is the case with Tensorflow AutoGraph as well, but the `experimental_relax_shapes` argument to the [`tf.function`](https://www.tensorflow.org/api_docs/python/tf/function) decorator helps alleviate this issue. Using JAX without `jit` and Tensorflow in eager mode are perfectly feasible options. Empirically, I’ve observed that both are a bit faster than PyTorch. Although, I haven’t tried using PyTorch’s jit. In any case, you can efficiently mix and match Tensorflow, JAX (and even PyTorch, if you really want to) using [dlpack](https://github.com/dmlc/dlpack) tensors. All 3 frameworks support it. Heck, even XGBoost supports it. 🙂 ---
**Author:** ![jrb](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/jrb/32/3472_2.png) [@jrb](https://forum.numer.ai/u/jrb)\ **Post date:** [December 24, 2021, 5:17pm UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/13 "2021-12-24T17:17:45Z")
> [@gbrecht](#): > > Era Batches apply to neural networks and there the common sense is that it is good to train in batches equivalent to eras. I.e. forward for one era and then backward. and then the next era and so on. Alternatively, use minibatches, but ensure that each minibatch does not contain data from more than one era. ---
**Author:** ![meaten12121](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@meaten12121](https://forum.numer.ai/u/meaten12121)\ **Post date:** [April 25, 2022, 8:12am UTC](https://forum.numer.ai/t/numerai-tournament-example-code-using-pytorch-nn-and-optuna/4639/14 "2022-04-25T08:12:11Z")
Hi, I’ve just updated the above GitHub repository to adopt the changes below: - 10x Faster MLP models compared to the initial commit (nn.ModuleList → nn.Sequential) - use numerai v4 dataset I know there will be benchmark models ( talked at numeracon April 2022) and my example should be obsolete soon but I decided to continue updating for a while for the sake of my learning and who forked my repository 🙂 Future plan: - Transformer model to extract new features different from MLP or GBDT - Xfeat([GitHub - pfnet-research/xfeat: Flexible Feature Engineering & Exploration Library using GPUs and Optuna.](https://github.com/pfnet-research/xfeat)) to automate feature engineering Any advice or proposals for future updates are welcome. Thanks!