Deep Papers

RAFT: Adapting Language Model to Domain Specific RAG

Jun 28, 2024

Sai Kolasani, a researcher at UC Berkeley’s RISE Lab and Arize AI Intern, discusses RAFT, a method to adapt language models for domain-specific question-answering. RAFT improves models' reasoning by training them to ignore distractor documents, enhancing performance in specialized domains like PubMed and HotpotQA. The podcast explores RAFT's chain-of-thought-style response, data curation, and optimizing performance in domain-specific tasks.

44:01

Episode guests

Sai Kolasani

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

RAFT improves LLM's reasoning by training to ignore distractor documents in specialized domains.

Optimizing document selection in Raft enhances LLM's performance in domain-specific datasets.

Deep dives

Raft: Incorporating Fine Tuning and Retrieval Systems

Raft is a technique aimed at training language models (LLMs) to effectively utilize context during model training. This approach stands out for focusing on teaching the LLM how to leverage context rather than just adhering to specific responses. By implementing detailed reasoning in the training process, Raft helps LLMs effectively navigate a mix of relevant and irrelevant information, enhancing their ability to reason and extract relevant answers.

Intro

3min

Fine Tuning Language Models

5min

Exploring the Raft concept for document identification improvement

2min

Training Language Model with RAFT Methodology

4min

Optimizing RAG System Robustness

3min

The Importance of Chain of Thought Reasoning and Data Curation in Language Models

9min

Optimizing Performance with Raft Pipeline for Domain-Specific Tasks

18min

Where adapting LLMs to specialized domains is essential (e.g., recent news, enterprise private documents), we discuss a paper that asks how we adapt pre-trained LLMs for RAG in specialized domains. SallyAnn DeLucia is joined by Sai Kolasani, researcher at UC Berkeley’s RISE Lab (and Arize AI Intern), to talk about his work on RAFT: Adapting Language Model to Domain Specific RAG.

RAFT (Retrieval-Augmented FineTuning) is a training recipe that improves an LLM’s ability to answer questions in a “open-book” in-domain settings. Given a question, and a set of retrieved documents, the model is trained to ignore documents that don’t help in answering the question (aka distractor documents). This coupled with RAFT’s chain-of-thought-style response, helps improve the model’s ability to reason. In domain-specific RAG, RAFT consistently improves the model’s performance across PubMed, HotpotQA, and Gorilla datasets, presenting a post-training recipe to improve pre-trained LLMs to in-domain RAG.

Read it on the blog: https://arize.com/blog/raft-adapting-language-model-to-domain-specific-rag/

Learn more about AI observability and evaluation, join the Arize AI Slack community or get the latest on LinkedIn and X.

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.

Deep Papers

RAFT: Adapting Language Model to Domain Specific RAG

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

Deep dives

Raft: Incorporating Fine Tuning and Retrieval Systems

Role of Raft in Specific Domains

Optimal Number of Documents at Test Time

Application and Production of Raft

Remember Everything You Learn from Podcasts