R Download and Submit Super Massive Data

of_s · September 12, 2021, 2:05pm

For any R users not on the #cran channel in RocketChat, I have posted updated functionality to the Rnumerai routines for downloading and submitting both the legacy and new datasets.

The data downloading updates can be found here:

github.com/Omni-Analytics-Group/Rnumerai

Update Rnumerai::download_data(...) for new .parquet files

opened 12:48PM - 09 Sep 21 UTC

OVVO-Financial

I came up with the following for downloading the new dataset, as well as leaving… the option to load into the global environment. I deprecated the `tournament` argument, because it's only `nomi` now, and left the defualts to the same output as the original. EDIT: Updated with `_int8` .parquet files as well as the `live_data` file. EDIT #2: Added `live_only`option download option, and converted the `tibble` output to a `data.table`. EDIT #3: Added `old_data_new_val` download to the `file_list`. ``` #' Function to download the Numerai Tournament data #' #' @name download_data #' @param location The directory path in which to store the data #' @param legacy logical; new dataset. If \code{legacy = TRUE}, legacy 310 feature .zip dataset downloaded #' @param load logical; to unzip / read.csv or read_parquet on data. #' @param live_only logical; download only the live .parquet dataset. #' @return A list containing the training and tournament data objects #' @export #' @importFrom lubridate today #' @import arrow #' @import httr #' @importFrom utils unzip #' @importFrom utils read.csv #' @examples #' \dontrun{ #' ## Directory where data files and prediction files to be saved #' ## Put custom directory path or use the current working directory #' data_dir <- tempdir() #' #' ## Download legacy dataset for current competition #' data <- download_data(data_dir) #' data_train <- data$data_train #' data_tournament <- data$data_tournament #' #' ## Download super massive dataset for current competition #' download(dat_dir, legacy = FALSE) #' } download_data <- function(location = tempdir(), legacy = TRUE, load = TRUE, live_only = FALSE) { tournament_id <- 8 round <- as.numeric(current_round()[1]) if(live_only==TRUE & legacy==TRUE) stop("There is no live only dataset for the legacy data.") if(legacy){ ## Get download link download_link_query <- paste0('{dataset(tournament:',tournament_id,') }') query_pass <- run_query(query=download_link_query) download_link <- query_pass$data$dataset message("Downloading Data...\n") ## Download File filename <- file.path(location, paste0("numerai_dataset_", today(), ".zip")) result <- httr::GET(download_link, write_disk(filename, overwrite = TRUE)) unzip(filename,overwrite = TRUE,exdir = location, list = FALSE) message("Finished downloading data\nReading dataset now...") if(load){ ## Read and return the data set data_train <- read.csv(file.path(location, "numerai_training_data.csv")) data_tournament <- read.csv(file.path(location, "numerai_tournament_data.csv")) return(list(data_train=data_train,data_tournament=data_tournament)) } } else { file_list = c("numerai_live_data_int8.parquet", "numerai_training_data_int8.parquet", "numerai_tournament_data_int8.parquet", "numerai_validation_data_int8.parquet", "example_validation_predictions.parquet", "example_predictions.parquet", "old_data_new_val.parquet") if(live_only) file_list = file_list[1] for(item in file_list){ message(paste0("Downloading Data...",item)) download_link_query <- paste0('{dataset(filename: "',item,'" , round: ',round,')}') query_pass <- run_query(query=download_link_query) download_link <- query_pass$data$dataset ## Download File filename <- file.path(location, item) result <- httr::GET(download_link, write_disk(filename, overwrite = TRUE)) if(live_only) end_file <- head(file_list, 1) else end_file <- tail(file_list, 1) if(load && item == end_file){ data_live <<- data.table::setDT(arrow::read_parquet(file.path(location, file_list[1]))) if(!live_only){ data_train <<- data.table::setDT(arrow::read_parquet(file.path(location, file_list[2]))) data_tournament <<- data.table::setDT(arrow::read_parquet(file.path(location, file_list[3]))) data_validation <<- data.table::setDT(arrow::read_parquet(file.path(location, file_list[4]))) example_validation_predictions <<- data.table::setDT(arrow::read_parquet(file.path(location, file_list[5]))) example_predictions <<- data.table::setDT(arrow::read_parquet(file.path(location, file_list[6]))) old_data_new_val <<- data.table::setDT(arrow::read_parquet(file.path(location, file_list[7]))) } } } } } ```

The submission updates can be found here:

github.com/Omni-Analytics-Group/Rnumerai

Update Rnumerai::submit_predictions() to include `version` and `source` in `create_submission` API call

opened 11:35PM - 11 Sep 21 UTC

OVVO-Financial

OK, can confirm this worked locally to submit to `legacy` tournament data, as we…ll as the new SM `example_predictions`. Added a logical parameter `legacy` to indicate the version number. EDIT: `diagnostics` argument added to run diagnostics on your validation upload. EDIT #2: Includes `source: "rnumerai"` for `createSubmission` and `createSignalsSubmission` GraphQL mutation ``` #' Function to submit the Numerai Tournament predictions #' #' @name submit_predictions #' @param submission The data frame of predictions to submit. This should have two columns named "id" & "prediction" #' @param location The location in which to store the predictions #' @param tournament The name of the tournament, Default is Nomi and is not case-sensitive #' @param legacy logical; if \code{legacy = FALSE}, submitting on super massive dataset, else submitting on old legacy format. #' @param diagnostics logical; set \code{diagnostics = TRUE} to to run diagnostics on your upload. #' @param model_id Target model UUID (required for accounts with multiple models) #' @param prefix The prefix to use for the submission csv file #' @return The submission id for the submission made #' @export #' @importFrom lubridate today #' @import httr #' @importFrom utils write.csv #' @examples #' \dontrun{ #' submission_id <- submit_predictions(submission_data,tournament="Nomi") #' } submit_predictions <- function(submission, location = tempdir(), tournament = "Nomi", legacy = TRUE, diagnostics = FALSE, model_id = NULL, prefix = tournament) { if(is.null(model_id)) stop("Must provide a model_id") ## Read the Trigger ID env variable trigger_id = Sys.getenv("TRIGGER_ID") if (trigger_id == "") { trigger_id = NULL } ## Match tournament ID tournament_id <- match(tolower(tournament),tolower(c("NOMI","SIGNALS"))) + 7 if(is.na(tournament_id)) stop("Tournament Name doesn't match") if(tournament_id == 8 && !all(names(submission)==c("id","prediction"))) stop("Column names should be id & prediction") if(tournament_id == 8 && legacy) version <- 1 else version <- 2 ## Write out the file date <- gsub("-","_",Sys.Date()) submission_filename <- file.path(location, paste0(prefix, "_submission_", date, ".csv")) write.csv(submission, submission_filename, row.names = FALSE) ## Get a slot on AWS for our submission if(tournament_id == 8){ if(!diagnostics){ aws_slot_query <- paste0('query aws_slot_query { submissionUploadAuth (filename : "',paste0(prefix, "_submission_", date, ".csv"),'", tournament: ',tournament_id,', modelId:"',model_id,'" ){ filename, url } }') } else { aws_slot_query <- paste0('query aws_slot_query { diagnosticsUploadAuth (filename : "',paste0(prefix, "_submission_", date, ".csv"),'", tournament: ',tournament_id,', modelId:"',model_id,'" ){ filename, url } }') } } else { aws_slot_query <- paste0('query aws_slot_query { submissionUploadSignalsAuth (filename : "',paste0(prefix, "_submission_", date, ".csv"),'", modelId:"',model_id,'"){ filename, url } }') } query_pass <- run_query(query=aws_slot_query) if(tournament_id == 8){ if(!diagnostics) numerai_aws_url = query_pass$data$submissionUploadAuth$url else numerai_aws_url = query_pass$data$diagnosticsUploadAuth$url } else numerai_aws_url = query_pass$data$submissionUploadSignalsAuth$url ## Upload the predictions mysubmission <- httr::PUT( url = numerai_aws_url, body = upload_file(path = submission_filename) ) if (mysubmission$status_code != 200) { stop(paste0("Uploading submission failed with status code ", mysubmission$status_code)) } ## Register our submission and get evaluation for it if(tournament_id == 8){ if(!diagnostics){ register_submission_query <- paste0( 'mutation register_submission_query { createSubmission (filename : "',query_pass$data$submissionUploadAuth$filename,'", tournament: ',tournament_id,', version: ',version,', source: "rnumerai", triggerId: "',trigger_id,'", modelId:"',model_id,'"){id} }' ) } else { register_submission_query <- paste0( 'mutation register_submission_query { createDiagnostics (filename : "',query_pass$data$diagnosticsUploadAuth$filename,'", tournament: ',tournament_id,', modelId:"',model_id,'"){id} }' ) } } else { register_submission_query <- paste0( 'mutation register_submission_query { createSignalsSubmission (filename : "',query_pass$data$submissionUploadSignalsAuth$filename,'", triggerId: "',trigger_id,'", source: "rnumerai", modelId:"',model_id,'"){id} }' ) } query_pass <- run_query(query=register_submission_query) if(tournament_id == 8){ if(!diagnostics) query_id <- query_pass$data$createSubmission$id else query_id <- query_pass$data$createDiagnostics$id } else query_id <- query_pass$data$createSignalsSubmission$id ## If error if(!is.null(query_pass$errors[[1]]$message)) stop(query_pass$errors[[1]]$message) ## Return submission id if(!diagnostics) message(paste("Submitted Prediction with id", query_id)) else message(paste("Submitted Diagnostics with id", query_id)) return(query_id) } ```

Please suggest any improvements or report any issues, thanks!

EDIT: diagnostics upload now available
EDIT (9/29): old_data_new_val.parquet download added and source added to submissions

jrb · September 15, 2021, 7:47pm

@of_s Thanks for unblocking all our fellow R users in the community, on being able to use the new super massive dataset. The Council of Elders has awarded you a 10 NMR discretionary retroactive bounty for your work on this. Keep it coming!

of_s · September 15, 2021, 7:58pm

Thanks CoE, I hope more R users participate!

The popularity of this post made me realize that I should post the fast R yahoo finance data download routine as well as the transformation from long to wide for Signals, which I did here (previously tucked away in the #cran channel in RC):

javibear · September 20, 2021, 4:39am

Hi thanks for doing this. I’m still getting the error: “Error in handle_url(handle, url, …) : Must specify at least one of url or handle”. don’t know what’s going on…

of_s · September 20, 2021, 10:49am

Hi @javibear, there’s a couple of possibilities: libraries httr and lubridate need to be installed, and the correct model_id parameter needs to be used in submit_predictions().

You can view your model_id from the get_models() call or alternatively view each ID via https://numer.ai/models .

LMK if any of these solutions work! Also, you can follow the discussions on the #cran channel in RC here.

javibear · September 20, 2021, 4:03pm

@of_s no luck. re-installed httr,lubridate. Still getting the error. by the way, I’m getting this message on download_data(). This started to happen just all of a sudden about 4 weeks ago, even before the super massive data intro.

of_s · September 20, 2021, 4:08pm

@javibear And you are using this download_data() version (https://github.com/Omni-Analytics-Group/Rnumerai/issues/35) saved locally after loading Rnumerai?

javibear · September 20, 2021, 9:35pm

@of_s my reply in rc:
javibear
2:29 PM
yes, did all those. tried writing the function directly onto the main script but no luck as well. from the error, it seems to functionally working and then it hits a walll or something. Downloading Data...numerai_live_data_int8.parquet Error in handle_url(handle, url, ...) : Must specify at least one of url or handle

of_s · September 20, 2021, 10:44pm

Just in case someone else runs into similar issue, having the latest version of Rnumerai (>=2.1.3) installed resolved this as noted in RC:

Thanks!