Better LGBM Params, Signals V2 Data, and Reducing Signals Churn

[gammarat]

Just a wee thought here - several years ago now Numerai expanded Val, Train, and Live files from a few hundred features to over a thousand for the Tournament, I suspect by tacking on feature data from 4 or so previous days (or weeks?). I suspect that because when I’d look at the correlations of the target with the features, one would get similar patterns repeated every 210 bins or so. I wrote about that back in 2021:

Visualizing the New Data

So I’ve started looking at the new data, first by looking at the correlations between the features and the targets for the training set. There’s some interesting patterns showing up. By way of process, what I did was take each era, Spearman correlate it with all the available targets. A small percentage of the targets, ~0.25%, had to be cooked (replaced with 0.5) as they were showing up as NaNs. There’s better ways, but that’ll do for now. There’s 21 targets for each era, two of which (“target…

Anyway, I think I might try something like that with the Signals data, it might help stabilize the predictions. That should keep me out of trouble for awhile