Closing the Loop with Andrew Camel

David Wakeling incubated a software product to help banks restructure contracts after the financial crisis, realizing the scale of work made it infeasible to do manually with lawyers. 10 years later, Allen & Overy has dozens of different software products managed by an interdisciplinary team of lawyers, software engineers, and data scientists. They're also one of the most advanced adopters of LLMs, where they're finding significant business value.Key TakeawaysBackground on Allen & Overy (A&O) & David [00:21]How did you arrive at this model for the Markets Innovation Group? [01:51]What about your background led you to do this? [04:08]How did you think about the revenue model for these products? [05:43]How does this interact with the by-hour billing model? [09:03]Does this enable you to sell other labor hours, or is it really just the software license? [11:00]How many products do you now have live and in market? [11:58]Is there a pattern to which of these solutions has been most successful, vs the others? [13:25]What cautions would you give to other firms / companies trying to build a similar model? [14:58]What was your first exposure to the "AI" space? [16:32]How exactly do you find using the platform? What use cases are most helpful? [20:40]How would you quantify the labor hours you've saved with this, across each category? [22:31]How do you think about this issue of governance? How do you advise clients on this? [26:53]Which of these problems do you feel are less well solved in the market? [29:09]How do you think about the buy vs build decision? [32:18]We believe we save a couple hours per week; across 3.7k lawyers, that's a lot of hours of lawyers' time [33:56]Where do you think value occurs then? [35:38]How do you think about "future-proofing"? [41:35]How do you operationalize this? [43:35]Does the product have to change between models? [44:22]

Historically, much of the automation/augmentation in insurance has occured in consistent, structured types of risk (e.g. auto). At a reinsurer like Swiss Re, risk is complex, inconsistent, and global in scale; LLMs are a major unlock. And this is just one of many learnings from Luca.Key Takeaways[00:12] Luca's background PhD and two post-docs, with a focus on the core math of machine learning[01:43] Background on Swiss Re's business[05:14] What are the primary functions in a given business line?[07:51] Where's the bulk of the headcount in the business?[09:45] What does a claim between an insurer and reinsurer look like?[13:32] When did we start transitioning from paper pushing into the first wave of digitization?[14:54] Fair to say that Swiss Re is involved primary in less standard types of risk given focus on reinsurance and complex primary insurance? And therefore less automated today?[16:25] What's the role of your team? [19:15] How do you address this problem of underwriting/pricing tail risk?[20:22] Breaking down data extraction into 1) extracting known data fields into structured systems 2) extracting information on a bespoke/one-off basis[25:13] Have you observed post ChatGPT that your colleagues were starting to use these tools organically? Or have you tried to disallow this behavior?[26:23] How exactly have you seen this tech changing workflows?[27:27] How have you thought about implementing these models? What challenges have you run into?[29:21] Are these tools something you'd like to buy externally or build in-house?[33:49] Do you think that the document-based Q&A format will hold as the long-term form factor?[35:53] Zooming out from doc data extraction in underwriting/claims, are there insights you can add in other areas e.g. new product development?[40:38] Are there other major categories or excitement or time investment for your team that we should be talking about?

What happens to evidence in the world of deepfakes? Does the NY Times have a reasonable complaint about OpenAI training on their content? How do you prevent someone maliciously generating cases at scale? Maura practiced law for 17 years, was a pioneer in eDiscovery, and is now a professor of computer science, and is incredible well-read and thoughtful on all these big questions.Topics Covered:(10:38) Why do you need to disclose that you use eDiscovery tools, and not e.g. Google search? Where is that line drawn?(16:06) Is there a standard test by which people evaluate tools to determine if they should be approved or not? (17:59) Why was the case about ChatGPT-created citations actually worse than most perceive?(21:00) How did you then see judges respond to this event?(23:48) Where will the line be drawn between disallow entirely or just disclose?(33:02) In the genAI curve of acceptance, where are we?(35:32) Evidence and deepfakes(41:28) What will the legal profession look like in this new world?(44:09) Do publishers have any recourse for models being trained on their data?

Amol takes us through CrowdStrike's journey from $6 million in ARR in 2014 to $2.5 billion today. Over that time, we talk through how threats have changed, how buyers mindsets' have changed, and where the opportunities are for new startups today in the face of consolidation. And as it relates to AI, we speak about CISOs concerns related to ChatGPT, the AI driven attacks we're already seeing, and where LLMs are most likely to play a role in security products.Topics covered:(6:29) What value did the cloud-only model bring to customers?(10:06) What were the governing rules of cyber when you joined CrowdStrike, and how do those compare to today?(14:24) Why do small groups now have access to more advanced attacks?(15:28) How are LLMs / AI broadly starting to play a role here?(18:28) To what degree is CISO concern valid with tools ChatGPT?(23:04) What do you think about the products / broader market around securing AI products themselves? (24:56) Broadly in cyber, what governs peoples' buying decisions today?(31:56) What are the specific functions in security that you think LLMs well-solve? (35:07)  Are there opportunities for new vendors to still succeed? Where are those distinctly non-platform opportunities?(38:23)

There are two major themes governing the cybersecurity market: a labor shortage in security talent, and phishing being the single largest threat vector. In a few different form factors, LLMs can start to fully close automation loops and solve this labor problem. Meanwhile, LLMs will amplify the scale and strength of phishing attacks, introducing a whole new set of problems. The opportunity set in both categories is immense.Topics Covered(7:43) Where else have you seen ML be well-applied in cybersecurity?(9:59) LLMs as a challenge for CISOs(15:59) Biggest opportunities in cybersecurity today(28:29) Where else do you see opportunity for emerging vendors?(35:12) How do you think about investing in security for ML when it's such a dynamic landscape?

A pipeline going down even 1% of the time means tens of millions of dollars in losses and serious health and safety concerns. And just one refinery can be 1000s of acres in size, so the scale over which inspection must happen is enormous. The availability of data to build models to run these inspections is a key bottleneck. New approaches in the areas of self-supervised learning show promise in solving this problem.Topics Covered(2:05) Breaking down Shell's business & key concerns(6:49) The role of regulation(8:21) Historical framing / what progress has already been made(12:48) The forefront of research in CV for inspections(16:28) Catastrophic forgetting(23:02) Commercial patterns that may emerge to help companies without in-house data science teams, what do small companies do in this situation?(26:06) The promise of multi-modal models(31:12) Key limitations that prevent models from being deployed today

Manufacturers of industrial equipment want a direct relationship with their customers to improve post-purchase customer experience and capture services revenue. Offloading monitoring data, using anomaly detection, and providing in-context maintenance information through LLM-driven agents is now possible. OEMs can leverage this new technology the build the relationships they want.Topics Covered(1:11) Background on Jon & Atlas(8:48) Why are OEM's sensitive to providing after-market services? Why haven't they done this?(19:57) Find long-term defensibility / avoid just being a GPT wrapper(25:14) Is this type of technology more important from a preventative perspective, or post-issue response?(27:00) What is the right way to distribute something like this? (42:14) 10x increase in investment in US manufacturing y/y in some categories

A significant portion of inefficiency in government contracting can be attributed to the writing & processing of long-form documents. Internal buyers from the four branches must write these documents to describe their needs and vendors must write them to respond. One forward-thinking group has already found promise in resolving this inefficiency with LLMs.Topics Covered:(0:24) Bonnie's role at the DoD(2:15) If someone wants to buy e.g. a tank, how exactly does that happen?(6:45) Where do you see those opportunities to accelerate the process? Where do you see opportunity for new technology? (10:46) What are those technology projects you're working on to help accelerate this process?(16:56) How DoD is integrating LLMs in the contracting process(20:05) What would usually take weeks or months I can now do in 30 minutes(31:49) What advice do you give entrepreneurs about how to best engage with you?

Mike Knoop, Co-Founder & Head of AI at Zapier, discusses how LLMs are transforming Zapier's business, driving 100x productivity improvements and deflecting customer support requests. They explore the role of LLMs in tool discovery, optimizing apps, and handling customer interactions. They discuss the vision for Zapier in this new world, challenges in building LLM-powered features, and evaluating model outputs. The episode also explores using language models for debugging, generating data, and improving error messages.

Few companies are better positioned to benefit from LLMs than Tegus; the company has 60k transcripts of calls between investors and experts, each many pages long and rich with interesting information. Michelangelo outlines several interesting product ideas, many of which are replicable in other scenarios.Topics Covered(5:15) The lifecycle of a Tegus transcript and where AI plays a role today(8:33) What are you most focused on in making better use of this large library?(14:17) The summaries product we built users loved(23:23) Internal use cases(29:05) Long-term vision: creating connections between data sources(30:17) The holy grail(30:42) What makes it challenging is you're mixing different kinds of content, so how do you deal with relative levels of "correctness" or "authoritativeness"(36:01) Build vs buy?(44:19) What is the top-level unifying goal for you and your team?