#275 Did Gen AI Kill NLP? with Meri Nova, Technical Founder at Break into Data
Jan 16, 2025
auto_awesome
Meri Nova, Founder of Break Into Data and an experienced AI engineer, dives into the evolution of natural language processing (NLP) and the transformative impact of generative AI. She examines when to apply traditional NLP techniques versus modern models, highlighting the balance between creativity and cost. The discussion includes the role of vector stores in handling unstructured data and the changing landscape of AI engineering. Meri also shares insights on harnessing AI for creativity and enhancing decision-making, particularly for those with ADHD.
The rise of generative AI presents new challenges for businesses, requiring a strategic decision between using traditional NLP techniques and advanced models based on specific use cases.
Knowledge graphs enhance AI systems by providing structured relationships between entities, improving data retrieval efficiency and fostering relevant outputs in various applications.
Deep dives
The Evolution of Natural Language Processing and Generative AI
Natural language processing (NLP) is vital in enabling computers to understand and generate human language, thus unlocking numerous applications across industries. Common uses include chatbots for customer support and content moderation in online communities, which have been present for over a decade. With the rise of generative AI, significant advancements have emerged, allowing for the creation of large volumes of text, revolutionizing how we interact with technology. This evolution indicates a transformative period in AI where creative tasks can now be tackled, moving beyond basic classification and understanding.
Choosing Between Traditional and Generative Techniques
Despite the popularity of large language models (LLMs), traditional NLP techniques remain relevant and effective for many straightforward tasks. For example, spam detection in emails can be performed efficiently with simpler models, not requiring the resources of LLMs. It is crucial to assess the specific business problem to determine whether to apply generative techniques for creativity or stick to traditional methods for tasks with clear patterns. Many organizations may benefit from leveraging both approaches, using generative methods only for more complex scenarios that truly justify their use.
Evaluating Performance in Non-Deterministic Systems
As AI progresses, particularly with generative models, evaluating the performance of these systems has become increasingly complex due to their probabilistic nature. Traditional measures of success are often inadequate, necessitating new benchmarks that consider factors like faithfulness and hallucination rates. These metrics are crucial for ensuring reliable performance in production environments, particularly for systems exposed to security vulnerabilities. Companies are developing various frameworks to assist in this evaluation, allowing for a more comprehensive understanding of how these models operate.
The Role of Knowledge Graphs and Vector Stores
Knowledge graphs significantly enhance AI systems by providing structured relationships between entities, which can improve the efficiency of retrieval-augmented generation (RAG) approaches. They allow for quicker searches by understanding the connections between data instead of relying solely on text similarity metrics. This creates opportunities for more relevant outputs in applications ranging from chatbots to search engines. While knowledge graphs have seen limited implementation in production, they represent an exciting frontier for future AI development, promoting a deeper understanding of complex relationships within data.
As AI continues to advance, natural language processing (NLP) is at the forefront, transforming how businesses interact with data. From chatbots to document analysis, NLP offers numerous applications. But with the advent of generative AI, professionals face new challenges: When is it appropriate to use traditional NLP techniques versus more advanced models? How do you balance the costs and benefits of these technologies? Explore the strategic decisions and practical applications of NLP in the modern business world.
Meri Nova is the founder of Break Into Data, a data careers company. Her work focuses on helping people switch to a career in data, and using machine learning to improve community engagement. Previously, she was a data scientist and machine learning engineer at Hyloc. Meri is the instructor of DataCamp's 'Retrieval Augmented Generation with LangChain' course.
In the episode, Richie and Meri explore the evolution of natural language processing, the impact of generative AI on business applications, the balance between traditional NLP techniques and modern LLMs, the role of vector stores and knowledge graphs, and the exciting potential of AI in automating tasks and decision-making, and much more.