Some troubleshooting for setting up Numerai Compute

[ml_is_lyf]

I managed to get it working, looks like I just needed to be a bit smarter on how to load the dataset using pandas. Previously I was casting to np.float16 AFTER loading in the dataset. But I saw JRB’s post where he sets it to np.float16 during reading via the dtype arg:

Saving memory with uint8 features

I’ve noticed that there have been quite a few complaints in the chat rooms about memory usage on compute since the test set got bigger. I’d originally contributed the float16 patch to example_model.py to help alleviate this, a couple of months ago. I just realized that there’s a simple way improve on this and to nearly halve the memory usage in your inference pipeline if you’re using a tree based classifier (Xgboost, Catboost, Lightgbm, etc). We just need to rescale the features and cast them t…

So I changed my code to set the dtype as np.float16 for the feature column and target like so:

tournament_data = pd.read_csv(tournament_data_file_path, dtype={COLUMN_NAME: np.float16 for COLUMN_NAME in FEATURE_COLUMNS+["target"]})

And now my prediction script runs successfully on the gen-md node.