Building a Small Yet Effective Team for Algorithm Development

Keith Coleman (VP of product) and Jay Baxter (founding ML engineer), the minds behind Community Notes, reveal how a small, scrappy team inside Twitter/X built the most trusted crowdsourced information system on the internet—one that’s changing the way we understand truth online. What you’ll learn:

1. How Community Notes actually works—a deep dive into the groundbreaking algorithm that rewards “bridging agreement” instead of majority rule

2. The seemingly crazy yet brilliant way this idea survived multiple CEO changes—from Jack to Parag to Elon

3. How this project started with a dumpster fire GIF (literally)—the untold backstory of its early launch

4. The secret to running ultra-fast, high-impact product teams—no OKRs, no Jira; just one Google Doc

5. What Meta’s adoption of Community Notes means for the future of online (mis)information—why this open source system is becoming the industry standard

—

Brought to you by:

• WorkOS—Modern identity platform for B2B SaaS, free up to 1 million MAUs

• Productboard—Make products that matter

• Wix Studio—The web creation platform built for agencies

—

Find the transcript at: https://www.lennysnewsletter.com/p/how-x-built-the-best-fact-checking-system-on-the-internet

—

Where to find Keith Coleman:

• X: https://x.com/kcoleman

• LinkedIn: https://www.linkedin.com/in/keith-coleman-19b12b46/

—

Where to find Jay Baxter:

• X: https://x.com/_jaybaxter_

• LinkedIn: https://www.linkedin.com/in/jaybaxter/

• Website: http://jaybaxter.net/

—

In this episode, we cover:

(00:00) Introduction to Community Notes

(06:56) How the “bridging-based” algorithm works

(13:33) The impact and scale of Community Notes

(17:24) Understanding the note publishing threshold

(21:32) Challenges and philosophies

(26:26) The effect of notes on re-sharing content

(29:41) Origin story

(35:46) Embracing small teams for big impact

(40:23) The thermal project approach

(47:47) Algorithm development and internal competitions

(50:34) An inside look at how the team operates

(58:56) Working with Elon

(01:05:30) Launching Birdwatch

(01:10:48) The core principles behind Community Notes

(01:26:15) Anonymity and pseudonymity in contributions

(01:32:17) Sustaining the project through leadership changes

(01:37:57) Future directions for Community Notes

(01:42:12) Final thoughts and optimism for the future

—

Referenced:

• Community Notes on X: https://x.com/CommunityNotes

• Sign up to be a Community Notes contributor: https://communitynotes.x.com/guide/en/contributing/signing-up

• The Making of Community Notes: https://asteriskmag.com/issues/08/the-making-of-community-notes

• “Readers added a Community Note to this Tweet”: https://x.com/HelpfulNotes/status/1718103364792205704

• Note-ranking algorithm: https://communitynotes.x.com/guide/en/under-the-hood/ranking-notes#matrix-factorization

• Study: Community Notes on X could be key to curbing misinformation: https://giesbusiness.illinois.edu/news/2024/11/18/study--community-notes-on-x-could-be-key-to-curbing-misinformation

• Study Finds X’s (Formerly Twitter’s) Community Notes Provide Accurate, Credible Answers to Vaccine Misinformation: https://qi.ucsd.edu/study-finds-xs-formerly-twitters-community-notes-provide-accurate-credible-answers-to-vaccine-misinformation/

• Did the Roll-Out of Community Notes Reduce Engagement with Misinformation on X/Twitter?: https://dl.acm.org/doi/10.1145/3686967

• Kayvon Beykpour on LinkedIn: https://www.linkedin.com/in/kayvz/

• Jack Dorsey on X: https://x.com/jack

• “Birdwatch gives me the creeps” tweet: https://x.com/elonmusk/status/1589454464611540992

• Blake Scholl on LinkedIn: https://www.linkedin.com/in/blakescholl/

• Creating Truthtelling Incentives with the Bayesian Truth Serum: https://www.eecs.harvard.edu/cs286r/courses/fall12/papers/DW08.pdf

• Asana: https://asana.com/

• Spaces: https://blog.x.com/en_us/topics/product/2021/spaces-is-here

• Amazon MTurk: https://www.mturk.com/

• Community notes on GitHub: https://github.com/twitter/communitynotes

• What do I think about Community Notes?: https://vitalik.eth.limo/general/2023/08/16/communitynotes.html

• X’s community-led approach: tackling inaccurate and misleading information: https://blog.x.com/en_us/topics/company/2023/xs-community-led-approach-tackling-inaccurate-and-misleading-information

• Linda Yaccarino on LinkedIn: https://www.linkedin.com/in/lindayaccarino/

• Messi-Ronaldo rivalry: https://en.wikipedia.org/wiki/Messi%E2%80%93Ronaldo_rivalry

• Supernotes paper: https://arxiv.org/pdf/2411.06116v1

—

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.

—

Lenny may be an investor in the companies discussed.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.lennysnewsletter.com/subscribe