Exploring Sequential Decision-Making in Machine Learning

Show Notes:

In this episode of Unleashed, Will Bachman interviews Harsh Sahai, CEO and co-founder of Bridgetown Research, a company that has built an AI tool and he talks about it in this episode. Harsh previously worked at McKinsey, where he focused on commercial due diligence. He also ran a machine learning lab at Amazon, where they researched sequential decision-making algorithms. 

AI Pricing Algorithms and Convex Optimization

Harsh talks about his work at Amazon where main use cases were pricing products, as people tend to remember old prices and make decisions based on what they remember. For example, planning the sequence in which to launch products or introducing new shows on Prime Video could be done in a multi-step planning process. Harsh talks about his background in convex optimization, which is a mathematical model that can be used to represent various outcomes. Convex optimization is often used to model price versus volume, and it helps in making more sequential decisions for more than just pricing. 

Bridgetown Research Explained

On founding Bridgetown Research, many of Harsh’s former colleagues joined him in the mission to build tools for the consulting industry and more. Bridgetown Research developed a platform that automates data collection and analysis, allowing them to curate these analyses and deliver value to clients. The firm developed software products that can conduct interviews at scale at a fraction of the cost, run 300 common analyses, evaluate approximately 10 decisions, and work alongside clients to build interactive documents. The firm primarily serves investors in the software industry, similar to McKinsey due diligence.

Automating Consulting Groundwork

They use AI agents to conduct interviews, breaking down high-level questions into sub-questions that can be answered by the AI agents. The agents then map the best sources of data for each analyze, such as Gartner or Forrester, and compile secondary research. The AI agents are integrated with a few expert networks, which they recruit on the company’s behalf. They have a fully adaptive conversation, similar to a consultant's conversation, and then parse out the analysis to answer the main questions. The cost of these interviews is lower than a normal human-to-human interview because they can do it on their own schedule. Harsh also discusses the benefits of owning a research platform for consultants. They have researched this topic extensively and have 1000 interview transcripts of both people who hired a consultant and like consultants. The platform offers voice-based conversations, text prompts, and interactive screens for additional context.

Using AI Agents in Surveys

The AI agent in the discussion is similar to a traditional survey, but it allows users to answer questions directly on their screen. It can also embed multiple choice or ranked sorting questions, and can follow a different chain of questioning depending on the user's response. The agent constructs a hypothesis based on secondary research and uses adaptive questions to collect enough data to either prove or disprove these hypotheses. If it disproves the hypotheses, it goes back and looks at all transcripts to come up with new hypotheses and start collecting more data. One of the reasons for the cost efficiency is that, unlike regular surveys, the AI agent doesn't ask the exact same questions, reducing the length by about 20 to 25% once statistical conviction is reached. This flexibility allows for discounts from the person taking the interview, as it's extremely convenient for them.

Examples of AI Agent’s Responsiveness

The agent's responsiveness works by comparing the user's responses to previous answers, such as asking about the main reasons they chose a particular software versus another. The agent then moves on to the next question based on the user's response. Harsh offers a few examples and verifies that the agents have received positive feedback from experts who are willing to interact with the voice agent, but they also interviewed people with slightly different profiles than consultants at McKinsey.

More Information about the AI Tool

The AI tool used in this discussion is a work in progress that aims to provide insights into competitor archetypes and their strategies. It is designed to be more efficient than traditional human interviews, as it can gather data from mid-tenure professionals and frontline users closer to the business operations. This approach allows for a more comprehensive understanding of the business, reducing the need for frequent human interviews. The tool is fully scalable, allowing for 100 interviews in three days, which is the time it takes to recruit individuals rather than the time it takes to interview them. This allows for the creation of compelling projects within a week. Before the interview phase, the AI tool asks a set of questions and breaks them down into sub hypotheses. The tool then constructs sub questions to explain various factors, such as margins, go-to-market channel, and strategy. The tool is capable of explaining up to 200 different factors, making it a versatile tool for analyzing competitor archetypes. It can also provide examples of how to segment competitors and investigate their cost. The tool's output includes eight hypotheses, which can be investigated through secondary research or questionnaires.

Examples of the Tool at Work

The AI tool is largely a work in progress, with multiple steps taken to chase each hypotheses down. The team is working on improving the UI and UX to make the process more tractable. Harsh explains that the sentiment analysis workflow involves a series of custom trained machine learning models that perform various tasks to produce a final output. He gives the example of an agent searching Reddit posts, determining if they are positive, negative, or neutral, and extracting themes from positive quotes. The main model categorizes comments as positive or negative, extracts themes, and summarizes codes by themes. Harsh explains that there can be around 300 analyzes executed by a permutation of 40 fundamental tasks. Another example is analyzing the average case buying process, deviations from the buying process, and key factors considered in decision-making stages. The standard KPC analysis on the platform includes two fine-tuned models: one extracting mentions of keeper Chase criteria across the transcript, and other clustering words to represent different meanings. The third component counts the number of mentions by category, which is the relative importance metric for each key purchase criteria.

Research Completed Before the Interview Stage

The secondary research that the platform performs before the interview stage, such as creating lists of competitors, acquisitions, customers, and suppliers. The platform triggers secondary research by identifying areas of interest and providing cues to help users interact more smartly. For example, when creating a new interview, the platform can identify main competitor types and determine reliable domains for secondary analysis. The tool can create personas of people to chat with, based on their background, geography, work experience, roles, and competitor employees. The platform then generates an interview guide for each segment, which includes text, background checkpoints, and a series of questions for the interviewee to answer. Users can edit these questions or add more options. The platform also provides a multiple choice option for users to choose from blank solution providers. The platform also offers an estimate of how long it would take for the person to fill out the survey, allowing users to save time and edit questions. The platform then prepares a granular set of hypotheses for each question, breaking them down and collecting data to either prove or disprove them. This process is similar to machine learning, where the information provided by the respondent validates or invalidates the hypotheses.

The Future of AI Tools and Human Consultants

Harsh shows a more manual flow where users can have full control over each step and explains how it works. The role of human consultants in the future is becoming increasingly important as AI tools become more prevalent. Three main factors drive clients' assessments of a consultant's contribution: experience, expertise, authority, credibility, and connections across the organization. These factors are fundamentally human and hard to replace. The tasks of early and mid-term consultants will shift from writing interview guides or conducting interviews to using AI tools or competitors. They will need to master these tools and learn how to review, approve, or edit the interview guide, synthesize the results, and make judgments about the quality of the results. 

Bridgetown Research and the AI Tool

The main business for the tool is providing customized versions of the tool to clients, catering to their specific analysis needs. However, there is a long wait list for users of the common platform, and one of the goals for 2025 is to onboard small to mid-sized consulting firms to use the product hosted by the firm without modification and see if they like it. Private equity investors are using Bridgetown Research's tool to conduct their own research, generating results directly without hiring consultants. The tool is cost-effective and provides a 60-70% answer without much effort. Investors typically hire consulting firms when they have a high degree of conviction to invest, but they are now using the platform for any deal they come across. The marginal cost is practically zero, making it a rational choice to use the platform for early stages of the deal pipeline. The platform is also available for use with investment professionals, consultants, and other professional services.

Timestamps:

04:17: Explanation of Convex Optimization 

06:53: Overview of Bridgetown Research's Platform 

09:06: Details of AI Interview Process

13:59: Examples of AI Interview Questions and Responses

19:51: Feedback and Adoption of AI Interviews 

23:22: Secondary Research and Hypothesis Generation

28:08: Examples of AI-Generated Analyzes 

40:26: Customization and Integration with Client Data 

Links:

Website: https://www.bridgetownresearch.org/

LinkedIn: https://www.linkedin.com/company/bridgetownresearch/?originalSubdomain=ca

Unleashed is produced by Umbrex, which has a mission of connecting independent management consultants with one another, creating opportunities for members to meet, build relationships, and share lessons learned. Learn more at www.umbrex.com.