Feature Groups and Interactions

Data Science
#1

There are 6 groups of features in the dataset as everyone knows and I’ve been always thinking there should be some reasons behind that.
Followings are the avenues I’ve tried to explore so far:

  1. Train models on a feature group (i.e. Dexterity only, Strength only etc), a combination of feature groups (i.e. Intelligence & Strength, Dexterity & Charisma & Constitution etc). There can be so many variations. Take subsets from each group and combine them, ensemble predictions etc etc.

  2. Generate some representative features from each feature group (e.g. PCAs, correlations, stds…) and use them for predictions.

  3. Limit interactions within a feature group and look at interactions with other feature groups only. (I’ve tried a XGBoost version below.)
    https://xgboost.readthedocs.io/en/latest/tutorials/feature_interaction_constraint.html

None of these attempts have led to any meaningful improvement in the metrics so far, unfortunately…

Has anyone tried something similar? Will be great if you could possibly share you 2 cents. Thanks!

#2

I tried the XGBoost feature interaction constraint and notice no discernable difference between using it and not using it. I have since abandoned the project. I do believe that the next frontier in model development will include some type of feature restriction, but so far, the frontier lies ahead…

1 Like
#3

My understanding is that the interaction constraints implemented by XGBoost is to allow only interactions within groups, not across groups. This is the opposite of what we want.

1 Like
#4

Thanks for your reply.

Yes, you are right about the way XGB implemented that. So I defined 310 lists and gave them as constraints. [feature_dexterity1, all features EXCEPT other dexterity features], [feature_dexterity2, all features EXCEPT other dexterity features], etc etc…

#5

So each constraint group has singled out one feature group into “quarantine”, but this still allows interactions among other feature groups.

Does having these many constraints affect training speed significantly?

#6

It wasn’t a problem but the result just wasn’t great unfortunately… I still don’t know how to make a good use of feature groups. The way I train my model at the moment is completely ignoring the groups so it’s no different even if they are given as feature1-feature310.

#7

Just realized this isn’t a proper way. Non-dex features can still interact within the group…

#8

Yes, that’s what I was referring to

#9

Agreed. It’s impractical to define all possible allowed interactions…

1 Like