Legal Innovation Spotlight 37 - The Stanford AI Legal Research Paper: Analysis and Implications with Richard Robbins and Brian Blaho
In this episode, Ted welcomes back Richard Robbins and is joined by Brian Blaho from Reed Smith to discuss the challenges of integrating AI into legal research. You'll gain insights from the recent Stanford study, which highlights the limitations of current AI tools, and learn why these models often struggle with legal reasoning. If you’re exploring AI for your firm, this conversation highlights how transparency, collaboration, and smart implementation drive innovation.
In this episode, Richard and Brian share insights on how to:
- Approach and evaluate AI systems for legal research
- Assess the limitations of retrieval-augmented generation (RAG) systems in legal AI
- Navigate the potential biases in academic studies about legal AI tools
- Effectively benchmark legal technology and AI performance
- Integrate generative AI tools into law firm workflows
Key takeaways:
- The Stanford study criticized current legal research AI tools from major vendors, highlighting their limitations in accuracy and understanding legal nuances.
- Legal research is more complex than initially anticipated for AI models, particularly regarding how models handle the nuances of legal precedent and authority.
- Despite the shortcomings of current AI tools, vendors are actively working to improve their systems, but transparency and collaboration on benchmarks are essential for progress.
- A broader, community-led approach, rather than vendor-driven initiatives, is critical for establishing accurate benchmarking standards for AI in the legal industry.
About the guests, Richard Robbins and Brian Blaho:
Richard Robbins is the Director of Applied AI at Reed Smith, leading AI and data science initiatives. A former law firm partner and general counsel, he specializes in legal tech and generative AI. Richard also supports UC Berkeley’s graduate class on generative AI and frequently writes and speaks on the subject.
Brian Blaho is the Global Director of Library Services at Reed Smith, managing library operations, budget, and innovations. With over 20 years in legal research, Brian focuses on legal tech and analytics. He leads global research and training and is active in professional associations.
“It’ll be up to the vendors who build on top of the [legal AI] models to add the other workflows, the agentic systems, and other elements to get us there.” – Richard Robbins
“Generative AI can potentially help us get much better results so that we have to either spend far less time reading through and curating which ones we're going to pass on or no time at all.” – Brian Blaho
Connect with Richard Robbins and Brian Blaho:
Website: https://www.reedsmith.com
Richard’s LinkedIn: https://www.linkedin.com/in/richard-robbins-a88014117/
Brian’s LinkedIn: https://www.linkedin.com/in/brian-b-37684524/
Connect with Ted Theodoropoulos:
Website: https://getinfodash.com/
LinkedIn: https://www.linkedin.com/in/tedtheo