We had a blast at ELC Annual 2024, so we wanted to bring our podcast listeners some of the best highlights from popular sessions! This episode features one of the ELC Annual sessions with Anupam Singh (VP of AI & Growth Engineering @ Roblox) & Maria Kazandjieva (Co-Founder @ Graft), as they discuss building AI/ML models at a massive scale. Anupam shares how Roblox – an immersive 3D platform with more than 77 million daily active users – scaled from zero to nearly 200 different AI models. They discuss strategies for deciding when to use open source vs. creating proprietary models; how to operationalize your models for 24/7 use; the importance of data pipelines; current and future challenges to keep in mind when creating / scaling AI models; and answer some questions from the live Q&A.ABOUT ANUPAM SINGHAnupam leads Roblox's AI & Growth engineering teams, which provide the infrastructure for high throughput AI services for safety, recommendations, and assistants. Before Roblox, Anupam was chief customer officer at Cloudera, where he led product, engineering, and field teams for Data Warehousing products. Anupam has co-founded two companies in the Big Data space, acquired by Cloudera and Marketshare, respectively. Anupam built his database expertise on the SQL Query Optimizer teams at Oracle, Sybase (now SAP), and Informix (now IBM). He graduated from Pune University in India and holds patents in the areas of automatic SQL performance tuning, object databases, and resilient query execution."The journey always starts with, 'Let's pick a model and first decide whether we want to build our own model or we want to use one of the open source ones.' The next step is, 'Do you want to do it on public cloud?' Roblox has 24 data centers worldwide and two massive data centers in America. We have hundreds of thousands of CPUs that we could use and so for us, it's very important to decide, 'Do we really need a large model? Can you take the 700 billion model, make it into a 7 billion parameter model, and magically get it to run on the CPU?'”- Anupam Singh   ABOUT MARIA KAZANDJIEVAMaria is a co-founder and an engineering leader at Graft, an early-stage AI startup. Prior to that, Maria worked at Netflix, where her team earned two Emmy awards for technical achievement. She holds a PhD in Computer Science from Stanford University. Outside of work, you can find Maria kickboxing & trail running, baking & eating carbs, or relaxing with a non-fiction book and her two feline supurrvisors, Foosball and Gemma.SHOW NOTES:How Roblox is being powered by AI (00:30)The process of scaling AI models from zero to 200 @ Roblox (2:34)Examples of Roblox starting with open source vs. building its own model (5:06)What AI models are doing in terms of safety for children (7:12)Strategies for deciding to use open source vs. building a proprietary model (11:19)Why Roblox is choosing to open source some of their own models (13:06)How to operationalize / engineer AI models for 24/7 use at scale (14:20)The importance of data pipelines in the AI journey (16:18)Current / future challenges as Roblox continues to scale its models (19:52)Tips for identifying use cases where implementing & scaling AI can be helpful (22:21)Audience Q&A: How do you make decisions when you’re lacking specific measurements / quantities? (24:29)When you deploy a model, how do you ensure confidence in its performance? (27:36)Recommendations for allocating / estimating the budget for a model (29:03)Anupam’s insights on maintaining so many models effectively (31:03)How do you imagine the multimodality of your 3D models moving forward? (33:11)This episode wouldn’t have been possible without the help of our incredible production team:Patrick Gallagher - Producer & Co-HostJerry Li - Co-HostNoah Olberding - Associate Producer, Audio & Video Editor https://www.linkedin.com/in/noah-olberding/Dan Overheim - Audio Engineer, Dan’s also an avid 3D printer - https://www.bnd3d.com/Ellie Coggins Angus - Copywriter, Check out her other work at https://elliecoggins.com/about/

A thrilling preview of the upcoming ELC Annual 2024 conference sets the stage for meaningful discussions on AI and leadership. Listeners are invited to connect with distinguished speakers ready to share invaluable insights. The emphasis is on expanding networks and igniting new ideas, making this conference a must-attend event for engineering leaders. Enjoy a special 10% discount on tickets with an exclusive code, ensuring that everyone has the opportunity to enhance their leadership journey!

Dave Rhodes, CEO of Sauce Labs, shares insights on the pivotal role of digital quality in business success. He discusses how quality issues can significantly impact a company's bottom line and offers strategies for effective quality testing. Dave highlights real-world examples, including a recent airline outage, to illustrate the cost of software bugs. He explores the necessity of fostering a quality-driven culture within engineering teams and emphasizes the importance of understanding the connection between user experiences and consumer trust.

Marcel Weekes, VP of Engineering at Figma, shares his insights from Config 2024, where he reveals the excitement surrounding the launch of Figma's Dev Mode and its emphasis on collaboration. He discusses how community feedback shapes product evolution and highlights the new layer renaming tool's impact on user experience. Marcel also touches on strategies for effective prioritization and the critical role of product support teams in preserving Figma's community. Listeners can anticipate his session on collaboration at an upcoming conference!

Farhan Thawar, VP & Head of Engineering @ Shopify, shares insights on motivating teams post-reorg, creating buy-in, and measuring developer productivity through demos. He also discusses creating memorable experiences, optimizing team productivity, and understanding team topologies for org-wide success.

Engaging conversation with Jessica McKellar, Andrew Lau, Samir Naik, and a former VPE at Robinhood about aligning engineering with business goals, team topologies, and strategic thinking in engineering leadership. Insights on metrics tracking, org design during hyper-growth, and optimizing virtual teams for customer focus.

Richard Wong, an experienced engineering leader, discusses challenges like increasing velocity without sacrificing quality, measuring productivity, and working cross-functionally with other teams. Insights from various eng leaders are shared, emphasizing the importance of aligning team goals with higher-level objectives and focusing on measurable outcomes. The podcast also explores the significance of clear requirements, transitioning responsibilities, and fostering healthy team collaboration.

Explore top challenges faced by engineering leaders such as coaching team members, motivating others, creating organizational buy-in, and managing up. Hear insights from industry experts like Netflix, Atlassian, and more on best practices and strategies to overcome these obstacles.

Farhan Thawar, VP of Engineering at Shopify, discusses the top 10 challenges faced by engineering leaders in a rapid-fire session. Insights gathered from various ELC events, podcasts & conferences.

CTO @ Lattice, Stephen Poletto, discusses scaling hiring processes, creating operational excellence systems, and incentivizing impact within engineering orgs. Insights on aligning hires with company vision, implementing hiring rubrics, and adapting rituals for growth. Strategies for rewarding impact over complexity, improving task management, and exploring health tech trends. Valuable resources for startup equity and hiring practices shared.