Arch Gateway: Add AI To Your Apps Without Custom Development

Feb 26, 2025

In this engaging discussion, Adil Hafiz, co-founder of Ardenimo and an expert with a rich engineering background at Microsoft and Amazon, sheds light on the Arch Gateway. This innovative tool simplifies AI integration for developers, allowing them to focus on core functions while bypassing complex AI specifics. He highlights the project's use of Rust and Envoy to enhance performance, discusses community feedback's crucial role, and outlines future aspirations for developing a leading planning model and improving AI agent interoperability.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Envoy at Lyft

At Lyft, Adil Hafiz used Envoy to manage service deployments and gain observability into system failures.
This allowed quicker identification of issues like browning out or 5xx errors without additional code.

ADVICE

Avoid DIY LLM Integrations

Don't DIY complex LLM integrations; use purpose-built tools.
Arch handles tasks like guardrails and routing, saving developer time and reducing integration issues.

ANECDOTE

Red Hat's Conversational ACM

Red Hat uses Arch Gateway to create a conversational interface for their ACM product APIs.
This allows operators to interact with ACM using natural language instead of navigating complex UIs.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Summary
In this episode of the AI Engineering Podcast Adil Hafiz talks about the Arch project, a gateway designed to simplify the integration of AI agents into business systems. He discusses how the gateway uses Rust and Envoy to provide a unified interface for handling prompts and integrating large language models (LLMs), allowing developers to focus on core business logic rather than AI complexities. The conversation also touches on the target audience, challenges, and future directions for the project, including plans to develop a leading planning LLM and enhance agent interoperability.

Announcements

Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems
Your host is Tobias Macey and today I'm interviewing Adil Hafeez about the Arch project, a gateway for your AI agents

Interview

Introduction
How did you get involved in machine learning?
Can you describe what Arch is and the story behind it?
How do you think about the target audience for Arch and the types of problems/projects that they are responsible for?
The general category of LLM gateways is largely oriented toward abstracting the specific model provider being called. What are the areas of overlap and differentiation in Arch?
Many of the features in Arch are also available in AI frameworks (e.g. LangChain, LlamaIndex, etc.), such as request routing, guardrails, and tool calling. How do you think about the architectural tradeoffs of having that functionality in a gateway service?
What is the workflow for someone building an application with Arch?
Can you describe the architecture and components of the Arch gateway?
With the pace of change in the AI/LLM ecosystem, how have you designed the Arch project to allow for rapid evolution and extensibility?
What are the most interesting, innovative, or unexpected ways that you have seen Arch used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Arch?
When is Arch the wrong choice?
What do you have planned for the future of Arch?

Contact Info

Parting Question

From your perspective, what are the biggest gaps in tooling, technology, or training for AI systems today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@aiengineeringpodcast.com with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers.

Links

The intro and outro music is from Hitman's Lovesong feat. Paola Graziano by The Freak Fandango Orchestra/CC BY-SA 3.0