Implementing new technologies like advanced AI models into production requires companies to evaluate various factors including technological maturity, hardware capabilities, and commercial model spectrum. The uncertainty lies in whether the technology will be feasible in the long run, depending on the commercial models available. The evolution of commercial models, such as the emergence of smaller yet powerful models like Lama 3 with 8 billion parameters performing almost equally well as larger models like Lama 2 with 70 billion parameters, indicates a positive trend. The expectation is that commercial models will continue to expand in both directions; larger models will grow in size while smaller models relevant for commercial use will become more efficient. This evolution enables companies to harness the capabilities of advanced models on the cloud, ensuring optimal performance and energy efficiency over time.
The size of ML models is growing into the many billions of parameters. This poses a challenge for running inference on non-dedicated hardware like phones and laptops.
Argmax is a startup focused on developing methods to run large models on commodity hardware. A key observation behind their strategy is that the largest models are getting larger, but the smallest models that are commercially relevant are getting smaller. The company was started in 2023 and has raised money from General Catalyst and other industry leaders.
Atila Orhon is the founder of Argmax and he previously worked at Apple and NVIDIA. He joins the show to talk about working in computer vision, building ML tooling at Apple, optimizing ML models, and more.
Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.
The post Scaling Large ML Models to Small Devices with Atila Orhon appeared first on Software Engineering Daily.