Understanding The Operational And Organizational Challenges Of Agentic AI
Apr 21, 2025
auto_awesome
Julian LaNeve, CTO of Astronomer, shares his expertise on the transition from simple LLMs to complex agentic AI systems. He stresses the importance of starting with easy applications to build foundational knowledge. The discussion delves into orchestrating AI workflows using directed acyclic graphs and highlights the necessity of robust data management. Julian also addresses the challenges of reliability and observability in AI, urging teams to thoughtfully evaluate their operational readiness and investment decisions in this dynamic field.
Transitioning to agentic AI systems requires teams to first gain experience with simpler LLM applications to effectively manage risks and complexities.
Observability and monitoring are crucial for assessing performance and reliability in AI workflows, helping organizations build trust in agentic systems.
Cost management is essential during the transition from LLMs to agentic AI, necessitating clear attribution of expenses to specific use cases for effective ROI evaluation.
Deep dives
Struggles of Data Integration in AI Applications
Data integration in AI applications often presents significant challenges, leading many teams to adopt Retrieval-Augmented Generation (RAG) methods, which can be costly and complex. This complexity stems from the difficulties in automating data ingestion and storage effectively across various AI systems. To address these issues, Cogni introduces an open-source semantic memory engine that automates these processes and generates dynamic knowledge graphs from the ingested data. This solution allows AI agents to better understand the data's meaning and provides accurate responses while lowering associated operational costs, enabling greater scalability.
Distinction Between Simple LLMs and Agentic AI
The conversation highlights a crucial distinction between simple language model (LLM) applications and more complex, agentic AI systems. Simple LLM applications typically follow predefined code paths, while agentic AI systems allow for greater autonomy, as the LLM dictates the workflow independently. The ease of use generated by leading model providers contributes to the perceived simplicity of LLMs, but transitioning to agentic AI can introduce unforeseen operational challenges. Teams are often encouraged to gain experience with simpler LLM workflows before attempting the complicated setup of agentic AI applications to effectively manage risk and complexity.
Benefits of Starting with Simple Use Cases
Adopting a phased approach to AI implementation often leads to more successful outcomes, where organizations benefit from first solving smaller, manageable use cases before advancing to complex agentic systems. For example, simple workflows such as monitoring GitHub activities or automating alerts can quickly demonstrate value and drive efficiency. This method not only allows teams to accumulate experience and intuition about what works well but helps avoid the trap of overcomplicating systems before demonstrating clear benefits. As organizations explore potential use cases, focusing on pragmatic solutions and low-hanging fruit can create a successful groundwork for future AI advancements.
Navigating the Challenges of Observability and Monitoring
As organizations develop complex AI workflows, challenges associated with observability and monitoring become increasingly important to address. Teams need to incorporate robust monitoring systems to ensure they can quickly assess the performance and reliability of AI agents and LLM workflows. Leveraging tools to trace and understand the interactions of these systems helps identify and mitigate issues that may arise during operation, ensuring effective performance. This approach to observability aids in building trust and confidence in AI-driven applications, especially as organizations scale their use of agentic frameworks.
Cost Considerations in Developing AI Applications
Cost management plays a critical role in the successful deployment of AI applications, particularly as teams transition from simple LLMs to more complex agentic systems. Users need to be aware of the multiplicative effects of running numerous LLM calls and the associated context size requirements to avoid unexpected expenses. Developing methods for clear attribution of costs to specific use cases is vital for evaluating the return on investment (ROI), allowing teams to measure the effectiveness of their AI solutions. Encouraging experimentation while monitoring costs effectively enables organizations to continue innovating without jeopardizing budgets.
Summary In this episode of the AI Engineering podcast Julian LaNeve, CTO of Astronomer, talks about transitioning from simple LLM applications to more complex agentic AI systems. Julian shares insights into the challenges and considerations of this evolution, emphasizing the importance of starting with simpler applications to build operational knowledge and intuition. He discusses the parallels between microservices and agentic AI, highlighting the need for careful orchestration and observability to manage complexity and ensure reliability, and explores the technical requirements for deploying AI systems, including data infrastructure, orchestration tools like Apache Airflow, and understanding the probabilistic nature of AI models.
Announcements
Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems
Seamless data integration into AI applications often falls short, leading many to adopt RAG methods, which come with high costs, complexity, and limited scalability. Cognee offers a better solution with its open-source semantic memory engine that automates data ingestion and storage, creating dynamic knowledge graphs from your data. Cognee enables AI agents to understand the meaning of your data, resulting in accurate responses at a lower cost. Take full control of your data in LLM apps without unnecessary overhead. Visit aiengineeringpodcast.com/cognee to learn more and elevate your AI apps and agents.
Your host is Tobias Macey and today I'm interviewing Julian LaNeve about how to avoid putting the cart before the horse with AI applications. When do you move from "simple" LLM apps to agentic AI and what's the path to get there?
Interview
Introduction
How did you get involved in machine learning?
How do you technically distinguish "agentic AI" (e.g., involving planning, tool use, memory) from "simpler LLM workflows" (e.g., stateless transformations, RAG)? What are the key differences in operational complexity and potential failure modes?
What specific technical challenges (e.g., state management, observability, non-determinism, prompt fragility, cost explosion) are often underestimated when teams jump directly into building stateful, autonomous agents?
What are the pre-requisites from a data and infrastructure perspective before going to production with agentic applications?
How does that differ from the chat-based systems that companies might be experimenting with?
Technically, where do you most often see ambitious agent projects break down during development or early deployment?
Beyond generic data quality, what specific data engineering practices become critical when building reliable LLM applications? (e.g., Designing data pipelines for efficient RAG chunking/embedding, versioning prompts alongside data, caching strategies for LLM calls, managing vector database ETL).
From an implementation complexity standpoint, what characterizes tasks well-suited for initial LLM workflow adoption versus those genuinely requiring agentic capabilities?
Can you share examples (anonymized if necessary) highlighting how organizations successfully engineered these simpler LLM workflows? What specific technical designs, tooling choices, or MLOps practices were key to their reliability and scalability?
What are some hard-won technical or operational lessons from deploying and scaling LLM workflows in production environments? Any surprising performance bottlenecks, cost issues, or monitoring challenges engineers should anticipate?
What technical maturity signals (e.g., robust CI/CD for ML, established monitoring/alerting for pipelines, automated evaluation frameworks, cost tracking mechanisms) suggest an engineering team might be ready to tackle the challenges of building and operating agentic systems?
How does the technical stack and engineering process need to evolve when moving from orchestrated LLM workflows towards more complex agents involving memory, planning, and dynamic tool use? What new components and failure modes must be engineered for?
How do you foresee orchestration platforms evolving to better serve the needs of AI engineers building LLM apps?
What are the most interesting, innovative, or unexpected ways that you have seen organizations build toward advanced AI use cases?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on supporting AI services?
When is AI the wrong choice?
What is the single most critical piece of engineering advice you would give to fellow AI engineers who are tasked with integrating LLMs into production systems right now?
