It seems strange that a model would be profitable up until you’re ready to pull your money out of the account. All of a sudden lots of burns totaling most of the previous rounds payout. Very weird. Can a model just turn horrible on the dime after 4 or 5 months of being profitable? O r is there a penalty when I drain the account? Guess I should’ve left my money in to avoid the penalty.
Pulling money out of your model has no effect on the payout and there is no penalty for that. Yes it can happen that a good performing model suddenly does not perform well anymore, especially if it is a long time (> 1 year) since you trained your model, and it happens all the time. You have some sort of influence on that during training as you can optimize for having stable performance across eras rather than having maximum performance over most eras, a trade off between volatility and mean performance.