HSBC's blockbuster report: Can AI track the Federal Reserve?

Recently, HSBC released a report titled "Can AI Track the Federal Reserve?" which delves into the potential and limitations of generative artificial intelligence in analyzing Federal Reserve officials' speeches and predicting monetary policy directions.

HSBC developed an AI tool based on GPT-4, specifically designed to automatically assess the relevance of Federal Reserve speeches, summarize content, determine policy inclination (hawkish/dovish/neutral), and explain the reasoning behind it.

By analyzing 54 Federal Reserve speeches, HSBC found that for analyzing speeches highly relevant to monetary policy, the AI tool's performance is comparable to that of a "hardworking junior analyst," and it aligns closely with human experts' conclusions in most cases.

However, when handling speeches that are unrelated or have low relevance to monetary policy, the AI may generate inaccurate or even fabricated content, diminishing user trust.

Therefore, HSBC's conclusion is that while AI tools have the potential to enhance productivity, they are currently insufficient to fully replace human analysts, especially in tasks requiring high precision.

AI Analysis of Federal Reserve Speeches

The AI tool developed by HSBC analyzes Federal Reserve officials' speeches through a "sentence-by-sentence filtering + overall analysis" approach, focusing on four aspects: relevance filtering, content summarization, policy inclination judgment (hawkish, dovish, or neutral), and explanation of decision-making basis.

The research team hopes to answer a core question with this tool: Can AI reliably and accurately analyze the Federal Reserve's policy signals, thereby providing practical references for investors?

The report points out that the AI tool performs impressively when analyzing highly relevant speeches, with output quality comparable to that of a "hardworking junior analyst." Especially in determining policy inclination, the AI's results are generally consistent with those of human experts.

For example, in the 54 speeches studied, the AI completely agreed with human experts when judging a speech as "neutral," and there were also few significant deviations when assessing "hawkish" or "dovish."

"We can see that there is currently no disagreement on dovish or neutral speeches. For the 10 speeches the model considers hawkish, our human experts agreed with this view in 8 cases, while other experts believed that only 2 speeches were neutral."

AI's Shortcomings: Randomness and "Creativity" Challenges

Despite the AI tool demonstrating certain capabilities, the report also highlights several shortcomings of AI.

Firstly, the "creativity" of generative AI can sometimes lead to issues, especially when dealing with speeches unrelated to monetary policy, where the AI may fabricate information to "satisfy" user needs. This phenomenon is known as "hallucination," which can severely undermine user trust in the tool

"Artificial intelligence models have been trained to exhibit a strong desire to help, and they tend to fabricate information to respond to you rather than remind you that your question is confusing or unanswerable."

Moreover, the inconsistency of AI tools in repetitive tasks is also a key challenge. The report found that even under the same conditions, AI's analysis of the same speech may yield different results.

HSBC's research team emphasized that while human analysts also exhibit some subjectivity, a single analyst's interpretation of the same content at different times usually maintains consistency, whereas AI cannot guarantee this.

Can AI Replace Analysts?

HSBC's research indicates that although AI tools perform to some extent in analytical tasks, they currently cannot fully replace the role of human analysts.

The report points out that AI excels in some more basic and repetitive tasks, such as filtering relevant information and generating summaries. However, in tasks involving complex reasoning and deep interpretation, the gap between AI and humans remains significant.

HSBC stated that the performance of AI tools is akin to that of junior analysts, but they cannot continuously improve through experience and feedback like humans. This is particularly important for businesses, as cultivating human analysts is not only for current output but also for future development potential.

The report also emphasizes that the "randomness" and "creativity" traits of AI, while bringing flexibility to technology, have also become obstacles to widespread application. In high-precision demand fields such as financial markets, users have extremely high requirements for the stability and credibility of results. If the output of AI tools requires repeated verification by human experts, the so-called efficiency gains will be significantly diminished.

"But when you ask artificial intelligence tools for their opinions, their opinions change constantly. Unless there is a fundamental change in human nature, it is unlikely that AI tools can serve as reliable advisors. This seems unlikely to happen in the short term."