Hagay Lupesko, Senior Director of Engineering at Databricks MosaicAI, discusses the innovative open LLM DBRX, bridging quality and cost efficiency. Topics include data control, collaboration in the AI community, model training, serving and optimizing, sustaining open source models, future plans for DBRX, hybrid RAG, tool utilization, knowledge graphs, and engagement opportunities with the open-source project.
DBRX aims to balance quality and cost for AI applications like rag-type applications.
DBRX offers open LLM models for fine-tuning and ownership, contributing to the AI community.
DBRX features expert architecture for faster training, maintaining model quality while optimizing computational costs.
Deep dives
Development of DBRX and Its Aim to Fill Gaps in LLMs Landscape
DBRX was created to empower enterprises and the ML community to build applications and advance the field. It aims to find a balance between quality and cost, catering to key applications like rag-type applications.
The Need for Bridging Gap Between Open and Closed LLM Models
DBRX was developed to bridge the gap between open and closed LLM models, offering a high-quality open model. Open models like DBRX allow for fine-tuning and ownership of models, control over data serving securely, and contribute to the AI community for experimentation and improvement.
Technical and Architectural Innovations in DBRX: Mixture of Experts
DBRX features a mixture of experts architecture for faster training and inference. It optimizes computational costs while maintaining model quality, offering a trade-off in forward pass computation. Experimentation with different expert options led to improved model quality while balancing computational costs.
Efficiency and Size Considerations in DBRX Deployment
DBRX offers quantization options to reduce RAM requirements for deployment, enhancing operational efficiency. Through optimizations like TRTLM support and operator fusion quantization, DBRX achieves over 150 tokens per second with reduced RAM requirements. There is a focus on cost-effective deployment and optimal model performance in DBRX.
Challenges and Future Directions for DBRX and LLMs
DBRX is faced with challenges in differentiating itself in a competitive landscape and maintaining ongoing commitment to model updates. Looking ahead, DBRX aims to explore new models, enhance tooling ecosystems, and adapt to evolving AI community needs. Continued innovation and customer-focused developments are key priorities for DBRX.
The Potential of Knowledge Graphs in Enhancing AI Models
Knowledge graphs appear promising for providing context and explainability to AI models, fostering better answer generation. While not extensively utilized currently, the concept of leveraging knowledge graphs in AI applications is intriguing and could enhance model performance and interpretability. Practical adoption and integration challenges still need to be addressed for wider implementation.
In this episode, Hagay Lupesko, Senior Director of Engineering at Databricks MosaicAI, delves into the creation and aspirations behind DBRX, an innovative open Large Language Model (LLM) designed to bridge the gap between quality and cost-effectiveness for AI applications.