I’ve been trying to do some work to cluster Eras. However, my algorithm relies on the actual return of the period within era as well which we do not get with live data. I tried without the return of the period, but the clusters are not so good. Has anyone else made progress on this?
This is just some rough algorithm. I am just experimenting with things until I find something more promising to use some Bayes
I am working on clustering eras, but I’m using Signals rather than the Tournament. (Signals is great for this sort of experimentation). But the underlying process is the same, and if I get it working there I’ll move the ideas into the Tournament models.
As for the clustering, and in a very simplified form, what I’m aiming at is clustering eras by which eras can be used to invert, or solve, others, and how to identify the likelihood of a new era belonging to this or that cluster.
That decision rests on finding some sort of fingerprint, or signature, from the features themselves that can be used to identify how a new era most likely relates to known clusters.
I am surprised with how many people participate in signals. I figured it would be way more difficult to put together all your own data. Then you don’t even know how much your data is already in the tournament dataset? It seems like a lot more work no?
Yes, what you describe is my approach as well for the upcoming unknown era. But I find some function of return data of the period is most important. Otherwise, I do not get the orange lines are down like the picture above. Perhaps I didn’t cluster well enough though.
I use Gaussian mixtures extensively in the Tournament; they’re great. In Signals, I’ve just recently started using them to separate regimes, based on signatures taken from weekly eras. Finding a practical set of numbers from features only to define a signature took some experimentation, but this week it finally seems to be working—for example, without referencing target returns, it separated out the regions immediately after market crashes (when the Fed tends to really loosen the purse strings).
Hi, I cluster eras by analyzing how well you can predict an era by overfit another era.
The idea is: If you have an overfitted model to an era A and it is good at predicting an era B, then A and B should be similar.
How I do it:
Create one model per era and train it only on its era.
With every model predict every era.
Measure the correlations between the predictions and targets.
Create a matrix where each row is made of model correlations.
The matrix of correlations between eras sorted by era number:
After a little bit of preprocessing and dimensionality reduction I cluster them.
I played with different number of clusters and came up with 5,3, and lastly 4 being the best.
The next picture has same values as the first one, but now its sorted by cluster and number of an era within a cluster. That’s why you can see square patterns (clusters), and the waves across the squares (eras tend to be similar to eras that are timewise close to them)
I am currently working on a model that would harvest this knowledge, so far it was better to use era boosting method. The era boosting method does a similar thing, but it better targets the weaknesses of a little-meta model. With the new data I have more success but the computations take a lot of time, I hope it will be worth it at the end :).
I trained 3 models on 3 clusters (now I use 4) separately to see how they would do over time, and it seems that they nicely complement each other; therefore, if I am able to guess when to use which…
Also, I thought of using the correlation matrix as an era selection technique. It could be theoretically possible to identify “weak eras”. By weak I mean that the model don’t learn anything from them and or they are easy to predict. But my learning PC Abacus is non-stop working on other trials.
That is very interesting work, thank you for sharing. The models are definitely a more complicated version of the correlations between features and target cluster I performed. I realized something though.
Because I used the return in the correlations in the clustering algorithm, if I then used the same returns during validation or tournament prediction (not possible because we don’t know), then I think I am essentially causing some data leakage into the model. That is because the cluster information tells me something about the return in that period already. I am not sure if you are using the same thing when training on each individual era.
Instead, you will want to use the prior return period perhaps, or just throw returns out all together would be the safest probably.
Yes, you cannot use the era models to classify the validation/live data. I am trying to create meta model that would learn on aggregated features of eras, with clusters being the target. For simple example:
X = df[features+['era']].groupby('era').var()
y = era_clusters
but of course var is not enough. The best part is, you can add training data for this meta model from each epoch, because aggregated features cannot be obfuscated. You just have to determine the cluster of an epoch. Or you can add data from the outside of the tournament like S&P500 index, but it will take some time to collect them.
How did you determine when to stop training? Did you hold out some portion of an era as validation? Did you use CV? Or did you deliberately let these models overfit given that you only wanted to use them to measure era similarity?
To select hyperparameters I used CV. Then I trained for each era a model with the same parameters without any validation. The idea is to overfit the era, but not overfit the data points. If it makes sence.
No, I used same hyper parameters for every era. It would be time consuming to fine tune for every era, and I don’t think it is needed.
This is just my intuition, but I think that you don’t need to fine tune the hp for every era, because the number of rows and columns are the same. Yes, there should be a difference in the number of useful features, but in the worst case scenario the era model learns fewer of the less useful features, because it should prioritize the more important features. And that is something that won’t affect the clustering much.
Thanks for the response. I share your instinct about not needing new HPs for every era. But for the sake of accuracy, the number of columns are the same across eras but the number of rows varies with each era.