Integrating LLMs with Microservices: Tools and Terminology

discussion about integrating LLMs into enterprise Java applications, challenges with non-deterministic LLM outputs in deterministic code environments, limitations of chat interfaces for power users in enterprise settings, preference for form-based applications with prompts running behind the scenes, using LLMs to understand unstructured data while providing structured interfaces, maintaining existing CRUD systems while using LLMs for unstructured data like emails and support tickets, practical examples of using LLMs to generate code from business requirements, creating assistants with system messages and short user prompts, potential for embeddings to replace text prompts in the future, developer journey in learning LLM integration including prompts, tools, RAG, and agentic workflows, benefits of specialized agents over one general agent, using LLMs for code generation with limitations for complex use cases, hybrid approaches combining LLMs with human oversight, using LLMs for email routing and support case classification, potential for extracting knowledge from enterprise data sources like Confluence and SharePoint, quality assurance with LLM judges, discussion of small language models versus large ones, model distillation and fine-tuning for specific enterprise use cases, cost considerations for model training versus using off-the-shelf models with better tool invocation, prediction that models will become more efficient and run on commodity hardware in the future, focus on post-training inference and reliable results

Burr Sutter on twitter: @burrsutter