AI Index Report Summary 2024, Part II

Part II, with a special focus on AI Benchmarks. Slides 46-132.

Hello Everyone,

I’m always reading studies, reports and AI papers (INDUS, a comprehensive suite of LLMs for Earth science, biology, physics, planetary sciences, and more) and I wanted to continue our summary of the AI Index Report 2024, to see part I go here.

The Stanford Institute for Human-Centered Artificial Intelligence (HAI)

The AI Index 2024 has more info than meets the eye to contribute to our macro understanding of the Generative AI movement in the 2020s.

Credits

This report was made possible by: Nestor Maslej, Loredana Fattorini, Raymond Perrault, Vanessa Parli, Anka Reuel, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, and Jack Clark, “The AI Index 2024 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, Stanford, CA, April 2024.

Slide 45 of 500.

This continues my summary of this massive report.

Frontier AI Research

The explosion of large language models really started around 2020, but has since really gone more global and exponential by 2024:

• By 2023, the United States led with 61 notable machine learning models, followed by China with 15, and France with 8.

• However in 2024, China is beginning to catch up I have noticed. Read more here.

Is China late to Generative AI?

• According to the Index Report, for the first time since 2019, the European Union and the United Kingdom together have surpassed China in the number of notable AI models produced (Figure 1.3.3). Since 2003, the United States has produced more models than other major geographic regions such as the United Kingdom, China, and Canada (Figure 1.3.4).

Beijing's attitude towards generative AI seems to be one of measured progress rather than cutting-edge innovation, says Wendy Chang. I might add that China’s strategy is always long-term and doesn’t seem to take the same hype curve of drama like Silicon Valley or hyperscalers trying to boost their stocks tend to do.

In 2024 I will note that China is seeking to catch the lead of OpenAI in a wider U.S. AI market shaped by tech titans Microsoft, Google and Amazon, and well-financed startups including Anthropic, Mistral, Cohere and Elon Musk’s x.AI that is rumored to have nearly concluded a round that would give them nearly $8 Billion in funding.

China’s New Breed of Generative AI Model Builders Includes

1. Baichuan

2. 01.AI

3. MiniMax

4. Moonshot AI

5. Zhipu

They are all going to be well-funded at least in the drought of foreign investors in China since the last couple of years. To imagine China not narrowing the gap of how far they are behind, is ludicrous.

Before 2024, you would have thought China was being left completely behind. I suspect Chian could dominate Open-source models in just a few years time. Where China lags in foundational large language models, which are dominated by OpenAI and Google’s Gemini, China is closing the gap by using open source LLMs like Meta’s Llama 1, 2 and 3 (not released yet).

With the explosion of interest in Open-Source LLMs and smaller more efficient models, (Open-Weight rather), China is surprisingly well positioned to innovate in Generative AI.

Parameter Trends

LLMs are getting bigger, more expensive to train and a lot more efficient fairly quickly along with better AI chips for compute and Nvidia’s stock price that has created a Generative AI bubble in the U.S. market especially.

Since OpenAI or Google Gemini aren’t transparent about the number of parameters in their models, some of this is speculation. The trend means you may need Billions in the future to keep up as a pure-play LLM foundational model builder.

Who to Watch?

• OpenAI

• Anthropic

• xAI

• Mistral

• Cohere

• And dozens of others building Open-source (Open-weight) models.