Will users trust on-device assistants with private data?

On-Device AI Agents in Production: Privacy, Performance, and Scale // MLOps Podcast #340 with NimbleEdge's Varun Khare, Founder/CEO and Neeraj Poddar, Co-founder & CTO.

Join the Community:

https://go.mlops.community/YTJoinIn

Get the newsletter: https://go.mlops.community/YTNewsletter

// Abstract

AI agents are transitioning from experimental stages to performing real work in production; however, they have largely been limited to backend task automation. A critical frontier in this evolution is the on-device AI agent, enabling sophisticated, AI-native experiences directly on mobile and embedded devices. While cloud-based AI faces challenges like constant connectivity demands, increased latency, privacy risks, and high operational costs, on-device breaks through these trade-offs.

We'll delve into the practical side of building and deploying AI agents with “DeliteAI”, an open-source on-device AI agentic framework. We'll explore how lightweight Python runtimes facilitate the seamless orchestration of end-to-end workflows directly on devices, allowing AI/ML teams to define data preprocessing, feature computation, model execution, and post-processing logic independently of frontend code. This architecture empowers agents to adapt to varying tasks and user contexts through an ecosystem of tools natively supported on Android/iOS platforms, handling all the permissions, model lifecycles, and many more.

// Bio

Varun Khare

Varun is the Founder and CEO of NimbleEdge, an AI startup pioneering privacy-first, on-device intelligence. With an academic foundation in AI and neuroscience from UC Berkeley, MPI Frankfurt, and IIT Kanpur, Varun brings deep expertise at the intersection of technology and science. Before founding NimbleEdge, Varun led open-source projects at OpenMined, focusing on privacy-aware AI, and published research in computer vision.

Neeraj Poddar

Neeraj Poddar is the Co-founder and CTO at NimbleEdge. Prior to NimbleEdge, he was the Co-founder of Aspen Mesh, VP of Engineering at Solo.io, and led the Istio open source community. He has worked on various aspects of AI, networking, security, and distributed systems over the span of his career. Neeraj focuses on the application of open source technologies across different industries in terms of scalability and security. When not working on AI, you can find him playing racquetball and gaining back the calories spent playing by trying out new restaurants.

// Related Links

Website: https://www.nimbleedge.com/

https://www.nimbleedge.com/blog/why-ai-is-not-working-for-you

https://www.nimbleedge.com/blog/state-of-on-device-ai

https://www.youtube.com/watch?v=Qqj_Nl2MihE

https://www.linkedin.com/events/7343237917982527488/comments/

~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~

Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore

Join our Slack community [https://go.mlops.community/slack]

Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)]

Sign up for the next meetup: [https://go.mlops.community/register]

MLOps Swag/Merch: [https://shop.mlops.community/]

Connect with Demetrios on LinkedIn: /dpbrinkm

Connect with Varun on LinkedIn: /vkkhare/

Connect with Neeraj on LinkedIn: /nrjpoddar/

Timestamps:

[00:00] On-device AI skepticism

[02:47] Word suggestion for AI

[06:40] Optimizing unique challenges

[13:39] LLM on-device challenges

[20:34] Agent overlord tension

[23:56] AI app constraints

[29:23] Siri limitations and trust gap

[32:01] Voice-driven app privacy

[35:49] Platform lock-in vs aggregation

[42:26] On-device AI optimizations

[45:38] Wrap up