It looks like serious progress is being made to fix MMC from a motivation perspective. I would like to reiterate my suggestion to fix MMC from a data scientist perspective by providing the metamodel because what myself and others have found in practice is that having the example predictions does not provide enough information to train a model that is oblique to the metamodel. I have suggested publishing weights that Numer.ai publishes every week that only indicate which rows of the live data are relevant.
I think the main objection to the weights is that it seems to allow one to get extra information about the meta model itself and therefore this method can be gamed. Now I am sure that there is a way around this detail. Didn’t Numer.ai guys engineer a crypto currency? So they must know something about cryptography. What I am saying is push that cryptographic technique further to give us a one-way cryptographic loss function for MMC. It takes predictions as input and using encrypted truth spits out the MMC Spearman correlation function. Now why can’t you do that?
The encrypted truth that I am talking about is only the best estimate of what the metamodel would have done and I think you have more than enough data to estimate it.