02/28/25: California bar exam faces technical difficulties, experiments with Deep Research, and more
Mar 6, 2025
auto_awesome
Steve Embry, a tech-savvy legal blogger, joins Nikki Black, a strategic mind at Affinapay, and Stephanie Wilkins, director at Legal Tech Hub. They dive into the chaos surrounding the California bar exam's online format, revealing the technical hiccups that could lead to legal battles. The trio also discusses the VALS AI benchmarks, questioning the transparency of AI tools in legal practices. Plus, they examine the intriguing evolution of e-discovery technologies and the looming competition from accounting firms stepping into the legal realm.
The VALS AI benchmarking study revealed operational differences in AI's efficiency across various legal tasks, stressing the need for ongoing scrutiny.
Panelists expressed concerns about potential biases in AI performance evaluations due to the involvement of firms using the tools.
The discussion highlighted the implications of KPMG's approval to practice law in Arizona, marking a shift toward alternative business structures in legal services.
Deep dives
Legal Tech Panel Introductions
The podcast opens with introductions from the panelists, each sharing their roles and affiliations within the legal tech community. Bob Ambrogi leads the discussion, establishing the format as one focused on recent stories in legal technology and innovation. Panelists include Steve Embry, who writes about legal tech, Nikki Black, a legal insight strategist at Affinapay, Stephanie Wilkins from Legal Tech Hub, and Victor Lee of the ABA Journal. Their collective expertise sets the stage for rich conversations surrounding the trends and advancements in the legal tech sector.
VALS AI Benchmarking Study Overview
A significant focus of the episode is the VALS AI benchmarking study, which evaluates various legal AI platforms. Stephanie Wilkins discusses the involvement of Legal Tech Hub in facilitating the study, emphasizing its independent nature. The study benchmarks the performance of AI tools against human baselines across various legal skills, helping to illuminate areas where AI excels and where it may fall short. The panel expresses excitement about the study as a foundational effort in understanding AI's capabilities in the legal field.
Performance Metrics and Findings
The study highlights that while many AI tools operate with remarkable speed, churning out results in a fraction of the time it takes human lawyers, the performance varies across different legal tasks. It specifically points to challenges in areas like legal research, which proved to be more complex than anticipated. The panel notes that several major AI vendors opted out of certain evaluations, raising questions about transparency and results interpretation. Overall, the findings suggest that AI can outperform humans in specific tasks but underscores a need for further inquiry into its effectiveness in others.
Skepticism and Industry Reactions
As the discussion progresses, panelists address skepticism surrounding the study, particularly regarding relationships between the involved parties and potential biases. Concerns arise about whether the participation of firms that use these AI tools could influence outcomes. The dialogue stresses the importance of transparency in the research process and invites constructive criticism to improve future iterations of the benchmarking study. The panel establishes that the legal tech community is eager for continued studies that explore the intersection of AI and legal practice.
Developments in Legal Practices and E-Discovery
The latter part of the episode delves into developments in legal practice, notably the growing acceptance of alternative business structures and their implications for the industry. The panel discusses KPMG's recent approval to practice law in Arizona and its potential impact on the legal landscape. They highlight concerns regarding compliance with existing legal frameworks and the legal implications of non-lawyer ownership. Additionally, the rise of generative AI in e-discovery showcases how technology continues to reshape legal practices, offering both challenges and opportunities for legal professionals.
Each week, the leading journalists in legal tech choose their top stories of the week to discuss with our other panelists.
This week's topics:
Remember the Legal Research ‘AI Smackdown’? I Decided To Bring OpenAI’s Deep Research to the Fight
Exclusive: With Its Latest Release Out Today, vLex’s Vincent AI Adds Multi-Modal Capabilities, Litigation Workflows, and Coverage for Four New Countries
From automation to generative AI, how e-discovery tools are evolving
The accountants are coming! The accountants are coming! But is it ethical?
California fails new bar exam, offers retake
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode