I tend to agree with Sam Altman when he was recently quoted saying “in the next 5 years I think some of of investing vehicle is going to figure out how to use AI to get crazy outperformance”. I also intend to spend the next few years proving Sam right with the Numerai community.
I wanted to open up a discussion on using LLMs in finance. Practitioners and academics are writing papers about using ChatGPT to predict stocks. Unfortunately, many of these papers are low quality and produce sketchy results. Nevertheless, it’s very clear that LLMs are good at comprehension and summarization of financial information (here’s Claude summarizing a 10-k from Netflix) and sentiment analysis on news.
Justina Lee, a journalist at Bloomberg who recently wrote about Numerai, tweeted a thread of papers which use ChatGPT in various ways for stock prediction. The problem with these papers is a lack of concern with implementation of the strategies (eg they ignore transaction costs or market impact costs) and the lack of rigor around whether the strategies are picking up on alpha or simply beta and factor exposures in some form.
Numerai Signals solved both the problem of a signal’s implentability and alpha discernment when it launched in 2020 by developing and releasing special targets. These targets are transformed versions of the subsequent return of every stock and span a universe of ~5000 global equities. They neutralize out market risk, factor risk, country and sector risk and therefore only evaluate the true alpha contained in a signal (not unlike Barra’s residual returns). They also focus on 1m subsequent return and remove the first 2 days of returns to account for implementation difficulties executing on the signal within size within 2 days. These transformations make the signals harder to predict on in the first place but also much higher quality and more likely to generalize into the future if you can develop a predictive signal on them.
When I first announced Numerai Signals in 2020, the use our language models to create signals was not lost on me. After all the goal of Numerai is to assimilate advances in artificial intelligence faster than any other hedge fund by being the world’s open hedge fund.
“There is little doubt in Jason Rosenfeld’s mind that modern language models like OpenAI’s GPT-3 are the future”, I wrote in the launch post for Numerai Signals. Jason who was formerly at the hedge fund Millennium has been a Numerai and Numerai Signals user for years. Today Jason works full-time on Crowdcent, a startup which stakes Numerai models and uses LLMs to build signals.
I caught up with Jason to discuss the research papers I’d been reading on ChatGPT in finance, there problems and think through ways someone might build a legitimately good and impressive signal on the Numerai Signals universe using LLMs. I’ve also spoken with a company collecting and processing vast amounts of financial news data like Nosible.
I wanted to share with the Numerai community how I would think about building a signal for Numerai Signals using LLMs. I think it’s valuable to share ideas so that the Numerai community can reach ‘hello world’ of a signal based on LLMs as fast as possible, and Numerai can begin to integrate more of these signals into the live trading of our hedge fund. I think the more that AI and quantitative finance secrets become public knowledge the better the world is and the stronger Numerai will be over the long term.
Download the targets from Numerai Signals
Go here: Numerai
In the left nav, click Data to download the data.
The file to open is called historical_targets.csv. It looks like the below.
The challenge on Numerai Signals is to create your own features and train a model to predict the target. The target highlighted in green is the one I would use for this: “target_20d_factor_neutral”.
Data
If anyone knows where one can get a free or cheap financial news or headlines dataset, tell us in the comments! Getting this type of data can be hard, ensuring that the data is point in time is hard, determining which article corresponds to which stock ticker is hard. Nevertheless, there may be easy ways to get started for example by just focussing on large US stocks where news data is very easy to find online.
The expensive but seemingly high quality data I’ve seen is at Reuters or Bloomberg but these are at inaccessible prices.
Embeddings as features
You can use language models to turn text into a numerical feature vector which has good properties. It might be sensible to turn all news headlines which were published before the “friday_date” column, obtain the embeddings (with OpenAI, BERT, etc) of each headline to generate 100-500 features for each stock which we can use to predict the target.
The embedding turns text into numerical features. With the features we can set up a regression problem would look like:
headline_embedding_value_1, headline_embedding_value_2, … , target_20d_factor_neutral
We could then model this with a simple LightGBM model. Because the stock market is hard, if we can get just 0.01 correlation with this model’s predictions to the target variable in out of sample predictions, this will be good. 0.015 would be excellent.
It could be a good idea to simply take the most recent news article headline from the last 30 days and leave all the features blank if there was no news in the past 30 days. For example, TSLA will have news almost every day so we can just use the headline closest to (but before) the friday_date which is probably the day before. For a smaller stock without much news, we might have to take news from 10 days ago.
I think it’s good to focus on headlines because if you have a full-text article the embedding can get weird and headlines are good compressions of the content anyways.
Prompt answers as features
Rather than use the abstract embedding of features of in the news headline you can prompt the model directly to give a response about it and even ground it through the prompt. For example, in one of the researchers papers above the prompt they used was:
Forget all your previous instructions. Pretend you are a financial expert. You are a financial expert with stock recommendation experience. Answer “YES” if good news, “NO” if bad news, or “UNKNOWN” if uncertain in the first line. Then elaborate with one short and concise sentence on the next line. Is this headline good or bad for the stock price of company name in the next month?
It seems very possible that prompts could be much better than embeddings to really get precisely the feature you want by directly prompting the model. Maybe it’s important to ask various creative prompts which get to different aspects of the article to create different features. For example, “is this news positive for the long term of the company and the short term?”, “does this news improve the moat or competitive edge of the company”, “even if the news seems to be positive in sentiment, is there anything in this news which suggests the company may beat earnings expectations?”
Point in time language models?
Language models are often trained with data up to a certain time eg 2021. This means the model might leak information from the future in the prompt. When giving news about Apple from 2014 and asking about Apple’s prospects, the model might use information it knows about Apple after 2014 in coming up with an answer and this will compromise all the results. Could a prompt that says “use strictly the information in the news article” help… maybe? Are there open source LLMs which are designed to be point in time… I don’t know.
I’m excited to see what the Numerai community can come up with in terms of finding or building a great financial news dataset, using OpenAI’s or other LLMs to create features, and building high performing orthogonal signals on Numerai Signals.
Retroactive bounties for super impressive work that gets shared on this topic! Looking forward to talking about it in the next Numerai Fireside Chat (which we’ll do on Discord on June 8th ).
