We turned off returns because we thought daily rounds might be polluting the numbers (they were not) and the current returns numbers are not easily verifiable from the outside. We are doing some more validation before re-enabling (soon).
The current Numerai and Signals returns calculations live in separate pipelines with their own bespoke implementations. The code is complex and hard to debug because it essentially boils the ocean from the beginning of time re-simulating every tournament scoring change, recalculating every payout, multiplier change, etc to derive returns.
Returns will return soon, and I will publish an accompanying worksheet explaining how users can audit/verify 1d, 3mo & 12mo returns using Excel themselves.
@wigglemuse You can in compute-heavy. You cannot in compute-light. That is what I desire. I have been complaining about it in all the channels (RC, the Trello, now here) without any success so far.
Hmmm…makes no sense to me, but I guess I don’t understand it. You’ve got an environment right, can’t it just do any arbitrary internet (and therefore api) call you want? If I can download int8 outside of compute why can’t you do it inside? What is actually stopping you?
no, in compute-light you do not have an evironmanet, but you deploy a pickled model file, the version of data (v2,3,4) and the list of features the model expects to the numerai endpoint. The rest is handled by numerai.
The models predict method is called during live with a pd.DataFrame of the data. That’s it.
That being said, I could of course start my predict method by ignoring the passed data, creating an API link and downloading the data. Apart from technical issues (can the script write to where it is running??) that completely defeats the compute-light goal of maximum integration so what I am asking for is to have another deployment parameter for int8 or float data.
Yeah, ok, pretty limited. I’m not saying they shouldn’t have int8 (and I’d need a whole lot more than that to make compute workable), but seems like you could get around it in the meantime as well. Can you just multiply the data * 4 to convert to int or does that cause memory issues?
Recognized that daily submission timeslot was changed. Like yesterday, the notification email was received at 1:43AM (GMT8), comparing with original schedule - 9PM (GMT8).
May i know this is temporary arrangement for test or there will be new fixed timeslot for data submission ?
How come that numerapi check-new-round --hours 1 returns 0 , except at the critical times after 13:00 UTC, when we need it to test for 1 to know when to start processing? But, then it just generates a whole stack of errors, making it useless and the daily tournaments opening times undetectable. It has been like this for the last two tournaments. Combined with the fact that you keep changing the opening times, it makes the daily tournaments totally impractical to participate in.
If you want us to be detecting the opening times, you must provide a functioning CLI test, not something buggy like this. If you are doing this to force us into using ‘compute’, then I am out of here. I am not willing to struggle with the complexities of AWS and pay for the ‘privilege’ when I have a perfectly functioning automation of my own. Or at least I had one …
Yesterday the data vendor was late, afterwards Numerai encountered an issue, delaying the opening time significantly. However, this scenario was already foreseen which is a major reason why Numerai pushes for “predictions on demand” rather than scheduled submissions.
What you can do is having an email listener on a low power computer waiting for a round open mail and starting your main computer via wake on LAN and running your submission script. For the past month, this has been my setup and worked perfectly. If there is interest I can clean up the code and put it on github.
So it has been 3 months now since daily rounds are active and almost the same timespan since this update, what is the status now? I turned my daily pipeline off for now to save some power costs, as there is still no payout whatsoever.
Are there still stability issues? Have priorities shifted? Are the preliminary results from daily MM predictions worse than expected?
It would be great to have a second update post like this one, even if it is just saying “We are still working on ABC but have troubles with XYZ”.