# Numerai Self-Supervised Learning & Data Augmentation Projects **URL:** \ **Category:** Data Science\ **Created:** [March 2, 2022, 12:08am UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003 "2022-03-02T00:08:24Z")\ **Posts on this page:** 15\ **Page:** 6
**Author:** ![slowmoe](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/slowmoe/32/2496_2.png) [@slowmoe](https://forum.numer.ai/u/slowmoe)\ **Post date:** [April 24, 2022, 10:15pm UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/105 "2022-04-24T22:15:49Z")
Ok, some progress that warrants an update: Re the cov-embedding: - Higher dimensions do not help at all. - inverse transforms do not generally yield valid (i.e. positive semi-definite) covariance matrices. This basically means you want to be really careful about the maths of any of this. Re the “cov as a model or regularizer of a model” topic: Note that covariance itself does not tell you anything about the shape of the distribution. However, when sampling the cov matrix, you have to know the shape of the distribution you want your samples to be in. In numerai data, feature distribution is uniform, but targets seem normal. To answer @jefferythewind: you can use numpy.random.multivariate\_normal() to make normally distributed samples with your cov matrix. You may write your own function to get different distributions. Also to your point: there seems to be something very peculiar to cov matrices. I tried embedding only its singular values or SVD decomposition, and others. None form any structure either. With covs, however, I could not find any parameters that do not yield a smooth string, even in higher dimensions. On V4, I could not get anything that outperforms the vanilla data yet (something something signal decay on longer valid period?), but you still get significant decorrelation to the example predictions. ---
**Author:** ![qstar](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/qstar/32/1566_2.png) [@qstar](https://forum.numer.ai/u/qstar)\ **Post date:** [May 4, 2022, 4:01am UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/106 "2022-05-04T04:01:15Z")
Is this considered solved/closed? I started working on this recently and wanna know if it’s still worth working on or not ---
**Author:** ![mdo](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/mdo/32/3398_2.png) [@mdo](https://forum.numer.ai/u/mdo)\ **Post date:** [May 15, 2022, 6:26pm UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/107 "2022-05-15T18:26:09Z")
This is very much an open area of research! ---
**Author:** ![nyuton](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/nyuton/32/687_2.png) [@nyuton](https://forum.numer.ai/u/nyuton)\ **Post date:** [December 20, 2022, 7:50am UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/108 "2022-12-20T07:50:10Z")
Haha, I was surprised to find my forum post in the investor presentation yesterday @richai🙂 For the record: I never managed to bring the deepdream idea to a statisfactory level. But I found another NN architecture that produces good augmented rows and have produced good results in the past months. ---
**Author:** ![jacob\_stahl](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/jacob_stahl/32/881_2.png) [@jacob\_stahl](https://forum.numer.ai/u/jacob_stahl)\ **Post date:** [December 21, 2022, 3:50am UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/109 "2022-12-21T03:50:59Z")
would that be a diffusion model? ---
**Author:** ![liz](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/liz/32/1103_2.png) [@liz](https://forum.numer.ai/u/liz)\ **Post date:** [December 25, 2022, 10:15pm UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/110 "2022-12-25T22:15:22Z")
haven’t read the entire thread but wanted to throw out the idea that when I was last active in the main tournament I did a pretty brainless data augmentation approach. I don’t remember exactly but I think I took 1000 random pairs of features (modifying the values so no 0’s to throw the logs) and made new features by taking one log the base of the other, and doing some sensible transformations after and then fit that data (joined with the original data). my model Urza did pretty well, not elite I think, but my point is fairly random and brainless data augmentation seemed to squeeze some extra juice out of the data available at the time (pre big data expansion), no academic exploration exerted. ---
**Author:** ![qstar](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/qstar/32/1566_2.png) [@qstar](https://forum.numer.ai/u/qstar)\ **Post date:** [January 20, 2023, 3:13am UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/111 "2023-01-20T03:13:56Z")
How long should the track record be for a model trained only on synth data? ---
**Author:** ![svendaj](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/svendaj/32/3285_2.png) [@svendaj](https://forum.numer.ai/u/svendaj)\ **Post date:** [January 21, 2023, 5:30pm UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/112 "2023-01-21T17:30:49Z")
> [@nyuton](#): > > Unlike PCA, Umap works with the Numerai dataset. > [@richai](#): > > I don’t think PCA works especially well Is there any analysis or examples of why PCA is not good for Numerai? Especially when Factor Analysis seems to work well for @katsu1110 [[numerai] factor analysis | Kaggle](https://www.kaggle.com/code/code1110/numerai-factor-analysis/notebook) ---
**Author:** ![olivepossum](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/olivepossum/32/2259_2.png) [@olivepossum](https://forum.numer.ai/u/olivepossum)\ **Post date:** [January 22, 2023, 8:28pm UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/113 "2023-01-22T20:28:40Z")
If I recall, I think it’s related to PCA working on linear relationships and just linear stuff doesn’t perform very well on this dataset (but who knows if there is people out there having more success with it). ---
**Author:** ![wigglemuse](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/wigglemuse/32/3094_2.png) [@wigglemuse](https://forum.numer.ai/u/wigglemuse)\ **Post date:** [January 22, 2023, 8:56pm UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/114 "2023-01-22T20:56:50Z")
If you are just targeting TC, throw all such advice out the window. But with CORR, yeah have had a hard time with pretty much any wholesale data transformations. (Augmentation working better than replacement.) ---
**Author:** ![qstar](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/qstar/32/1566_2.png) [@qstar](https://forum.numer.ai/u/qstar)\ **Post date:** [February 14, 2023, 10:58pm UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/115 "2023-02-14T22:58:47Z")
If anyone is interested I developed a data generation process akin to the the tournaments data. **This model sees no real data**. Very little meta model corr and decent corr, and very promising seeming TC. See the [model starq\_synth2 model here](http://numer.ai/starq_synth2) also [starq\_synth here](http://numer.ai/starq_synth) (starq\_synth seems to be more volatile, this model is only trained on some super old training data from a while ago) ---
**Author:** ![kayeffnumeraitor](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/kayeffnumeraitor/32/991_2.png) [@kayeffnumeraitor](https://forum.numer.ai/u/kayeffnumeraitor)\ **Post date:** [February 15, 2023, 9:40am UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/116 "2023-02-15T09:40:00Z")
I don’t want to downplay your work as it could still be a very good model, but in my experience it takes way longer than one resolved era to fully assess the capabilities of a model. After a few months of resolved eras the picture will be more clear. ---
**Author:** ![qstar](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/qstar/32/1566_2.png) [@qstar](https://forum.numer.ai/u/qstar)\ **Post date:** [February 15, 2023, 2:56pm UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/117 "2023-02-15T14:56:28Z")
totally agree, although tbf I asked above how many rounds and no one said anything 😆 ---
**Author:** ![andralienware](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@andralienware](https://forum.numer.ai/u/andralienware)\ **Post date:** [March 20, 2023, 2:06pm UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/118 "2023-03-20T14:06:30Z")
Rather than embedding singular values of the cov matrix, it may worth trying to embed the square roots of the cov matrix since even if the square root is not symmetric, the sampled “square roots of cov matrices” times the transpose of such sampled matrices should still be symmetric. On the other hand if you were trying to use something like a DAE rather than just umap for embeddings, you could create some sort of “asymmetry error” to do unsupervised encoding or decoding training and then fine tune on some held-out real data. ---
**Author:** ![qstar](https://yyz1.discourse-cdn.com/flex009/user_avatar/forum.numer.ai/qstar/32/1566_2.png) [@qstar](https://forum.numer.ai/u/qstar)\ **Post date:** [March 22, 2023, 2:27pm UTC](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003/119 "2023-03-22T14:27:47Z")
[starq\_synth](https://numer.ai/starq_synth/submissions) is trained on synthetic data, scored against some super old v4 data and randomly gets 99th percentiles. p interesting [Previous page](https://forum.numer.ai/t/numerai-self-supervised-learning-data-augmentation-projects/5003.md?page=5)