Not sure if this is exactly what you’d like here but I was playing with QLoRA to fine-tune the models. Just sharing something I worked on this weekend.
The idea was to, instead of training a model from scratch, how about fine-tuning with LoRA on numerai data. My initial thought was to simply use ChatGPT’s web access to let it understand and then ask it my doubts. Just like stable diffusion LoRAs, why not try it for text using a local model. More specifically, letting the LLM with a broader understanding of the world, fine-tune on all the Rocketchat discussions, Numerai docs (including OHwA), Forum and then asking it to explain me TC. In short, demystifying TC from all the discussions.
I downloaded all the textual data locally and tuned a model on Numerai related content. not much processing done.
With QLoRA I was able to tune the EleutherAI/gpt-neox-20b with the help of notebook provided. Due to limited resources for Colab Pro, I was able to train it only for 500 steps (~0.1 epoch) without much data cleaning. Maybe someone can use this notebook/model as checkpoint and train it further for multiple epochs. This is still an early notebook without proper evaluation but a proper tuning will require much more efforts and compute(which I currently lack).
-
base model’s output
-
output after tuning, and loading the model from HF hub
Notebooks:
- QLoRA training+inference notebook: bnb-4bit-training_numerai_forum_and_rc.ipynb
- Dataset collection notebook (may not work out of the box): 01_parse_discussions.ipynb
- HF HUB LoRA: surajp/500steps-gptneox20b-qlora-numerai
Potential improvements:
- LoRA on financial data; reduces compute requirements by a huge margin. (easy embeddings; potential alternative to BloombergGPT?)
- Instruction-tuning
- Use a better baseline model
- Train for longer
I apologize for directly training on the text without permissions (just couldn’t wait). Happy to delete the model from hub and the saved training data. Please DM
edit: Thanks @slyfox for the permission to keep it hosted