Training Large Arabic Language Models

17min Snip

00:00

Play full episode

Summary

Transcript

Episode notes

This chapter discusses the training of large Arabic language models, including the Jase 13 billion and 30 billion parameter models. They highlight the quality of the 30 billion model, express excitement about future releases through Core 42, and mention the Falcon or TI Institute in the UAE. Furthermore, they touch upon challenges with low resource languages and techniques presented at NURIPS to address them.

We are running an end of year listener survey! Please let us know any feedback you have, what episodes resonated with you, and guest requests for 2024! Survey link here.

We can’t think of a more Latent-Space-y way to end 2023 than with a mega episode featuring many old and new friends recapping their biggest news, achievements, and themes and memes of the year!

We previously covered the Best Papers of NeurIPS 2023, but the other part of NeurIPS being an industry friendly conference is all the startups that show up to hire and promote their latest and greatest products and papers! As a startup-friendly podcast, we of course were ready with our mics to talk to everyone we could track down.

In lieu of an extended preamble, we encourage you to listen and click through all the interviews and show notes, all of which have been curated to match the references mentioned in the episode.

Timestamps & Show Notes

* [00:01:26] Jonathan Frankle - Chief Scientist, MosaicML/Databricks

* see also the Mosaic/MPT-7B episode

* $1.3B MosaicML x Databricks acquisition

* [00:22:11] Lin Qiao - CEO, Fireworks AI

* Fireworks Mixtral

* [00:38:24] Aman Sanger - CEO, Anysphere (Cursor)

* see also the Cursor episode

* $8m seed from OpenAI

* Tweet: Request-level memory-based KV caching

* Tweet: GPT-4 grading and Trueskill ratings for rerankers

* [00:51:14] Aravind Srinivas - CEO, Perplexity