Open Models and Maturation: Assessing the Generative AI Market
May 17, 2024
auto_awesome
Exploring the maturation of the generative AI market, from the emergence of open source LLMs to AI-generated music. Discussion on the sawtooth pattern of growth in AI applications, the importance of fine-tuning models, and the evolution of founders and products in the AI industry. Delving into advancements in model creation and the shift towards open-source models, with a focus on benchmarking and evaluating performance in the LLM market. Exploring the wide-ranging impact of AI breakthroughs in music and education.
Generative AI market growth follows a sawtooth pattern, with new spikes after flat periods.
Convergence of open source and close source AI models driven by extensive training data sets.
Deep dives
Future of Multi-Modal Models
The future of AI models seems to be oriented towards integrating multiple input and output modalities into one model, such as combining images and text. There is a widespread belief that model architectures are converging, with examples like the prominent role of Transformers in image generation evolving rapidly over time. As architectures continue to converge, these models are expected to become more popular and mature.
Increased Adoption of Open Source Models
There has been a notable trend towards the convergence of open source models with close source models in terms of performance and quality. With advancements like LLMs narrowing the gap, some open source models are now competitive with close source counterparts. The extensive training data sets used today are a key factor contributing to the convergence of model quality, where even models trained on similar data sets are producing comparable responses.
Evolution of Training Models
The paradigm of model training has shifted from focusing solely on compute power and model parameters to emphasizing the importance of training data sets. With training data sets exceeding 10 trillion tokens, there is a limitation on available textual human knowledge to feed these models. Overtraining models and the increasing reliance on data sets have led to the emergence of competitive models, indicating a shift in the training dynamics influencing model quality.
Emergence of Application Layer on Models
Applications layered on top of AI models like Chat GPT have catalyzed market growth, demonstrating the commercial scalability of several model providers, particularly open AI. The integration of AI models into consumer and B2B applications has accelerated the adoption and success of AI technologies, notably enhancing user interaction and workflow efficiency. Industries are witnessing a shift towards innovative AI applications that build on top of existing model capabilities, driving further market expansion.
a16z partners Guido Appenzeller and Matt Bornstein join Derrick Harris to discuss the state of the generative AI market, about 18 months after it really kicked into high gear with the release of ChatGPT — everything from the emergence of powerful open source LLMs to the excitement around AI-generated music.
If there's one major lesson to learn, it's that although we've made some very impressive technological strides and companies are generating meaningful revenue, this is still a a very fluid space. As Matt puts it during the discussion:
"For nearly all AI applications and most model providers, growth is kind of a sawtooth pattern, meaning when there's a big new amazing thing announced, you see very fast growth. And when it's been a while since the last release, growth kind of can flatten off. And you can imagine retention can be all over the place, too . . .
"I think every time we're in a flat period, people start to think, 'Oh, it's mature now, the, the gold rush is over. What happens next?' But then a new spike almost always comes, or at least has over the last 18 months or so. So a lot of this depends on your time horizon, and I think we're still in this period of, like, if you think growth has slowed, wait a month and see it change."