What is Refine and why build an AI referee

In this episode, we sit down with Ben Golub, economist at Northwestern University, to talk about what happens when AI meets academic research, social learning, and network theory.

We start with Ben’s startup Refine, an AI-powered technical referee for academic papers. From there, the conversation ranges widely: how scholars should think about tooling, why “slop” is now cheap, how eigenvalues explain viral growth, and what large language models might do to collective belief formation. We get math, economics, startups, misinformation, and even cow tipping.

Links & References

* Refine — AI referee for academic papers

* Harmonic — Formal verification and proof tooling for mathematics

* Matthew O. Jackson — Stanford economist and leading scholar of networks and social learning

* Cow tipping (myth) — Why you can’t actually tip a cow (physics + folklore)

* The Hype Machine — Sinan Aral on how social platforms amplify misinformation

* Sequential learning / information cascades / DeGroot Model

* AI Village — Multi-agent AI simulations and emergent behavior experiments

* Virtual currencies & Quora credits — Internal markets for attention and incentives

Transcript:

Seth: Welcome to Justified Posteriors, the podcast that updates its beliefs about the economics of AI and technology.

Seth: I’m Seth Benzel, hoping my posteriors are half as good as the average of my erudite Friends is coming to you from Chapman University in sunny Southern California.

Andrey: And I’m Andrey Fradkin coming to you from San Francisco, California, and I’m very excited that our guest for today is Ben Goleb, who is a prominent economist at Northwestern University. Ben has won the Calvó-Armengol International Prize, which recognizes a top researcher in economics or social science, younger than 40 years old, for contributions to theory and comprehension of mechanisms of social interaction.

Andrey: So you want someone to analyze your social interactions, Ben is definitely the guy.

Seth: If it’s in the network,

Andrey: Yeah, he is, he was also a member of the Harvard Society of Fellows and had a brief stint working as an intern at Quora, and we’ve known each other for a long time. So welcome to the show, Ben.

Ben: Thank you, Andrey. Thank you, Seth. It’s wonderful to be on your podcast.

Refine: AI-Powered Paper Reviewing

Andrey: All right. Let’s get started. I want us to get started on what’s very likely been the most on your mind thing, Ben, which is your new endeavor, Refine.Ink. Why don’t you tell us a little bit about, give us the three minute spiel about what you’re doing.

Seth: and tell us why you didn’t name your tech startup after a Lord of the Rings character.

Ben: Man, that’s a curve ball right there. All right, I’ll tell you what, I’ll put that on background processing. So, what refine is, is it’s an AI referee technical referee. From a user perspective, what happens is you just give it a paper and you get the experience of a really obsessive research assistant reading for as long as it takes to get through the whole thing, probing it from every angle, asking every lawyerly question about whether things make sense.

Ben: And then that feedback, hopefully the really valuable parts that an author would wanna know are distilled and delivered. So as my co-founder Yann Calvó López puts it, obsession is really the obsessiveness is the nature of the company. We just bottled it up and we give it to people. So that’s the basic product—it’s an AI tool. It uses AI obviously to do all of this thinking. One thing I’ll say about it is that I have long felt it was a scandal that the level of tooling for scholars is a tiny fraction of what it is for software engineers.

Ben: And obviously software engineering is a much larger and more economically valuable

Seth: Boo.

Ben: least

Andrey: Oh, disagree.

Ben: In certain immediate quantifications. But I felt that ever since I’ve been using tech, I just felt imagine if we had really good tools and then there was this perfect storm where my co-founder and I felt we could make a tool that was state of the art for now. So that’s how I think of it.

Seth: I have to quibble with you a little bit about the user experience because the way I went, the step zero was first, jaw drops to the floor at the sticker price. How much do you,

Ben: not,

Seth: But then I will say I have used it myself and on a paper I recently submitted, it really did find a technical error and I would a kind of error that you wouldn’t find, just throwing this into ChatGPT as of a few months ago. Who knows with the latest Gemini. But it really impressed me with my limited time using it.

Andrey: So.

Ben: is probably, if you think about the sticker price, if you compare that to the amount of time you’d have, you’d have had to pay error.

Seth: Yeah. And water. If I didn’t have water, I’d die, so I should pay a million for water.

Andrey: A question I had: how do you know it’s good? Isn’t this whole evals thing very tricky?

Seth: Hmm.

Andrey: Is there Is there, a paper review or benchmark that you’ve come across, or did you develop your own?

Ben: Yeah. That’s a wonderful question. As Andrey knows, he’s a super insightful person about AI and this goes to the core of the issue because all the engineers we work with are immediately like, okay, I get what you’re doing.

Ben: Give me the evals, give me the standard of quality. So we know we’re objectively doing a good job. What we have are a set of papers where we know what ground truth is. We basically know everything that’s wrong with them and every model update we run, so that’s a small set of fairly manual evaluations that’s available. I think one of the things that users experience is they know their own papers well and can see over time that sometimes we find issues that they know about and then sometimes we find other issues and we can see whether they’re correct.

Ben: We’re not at the point where we can make confident precision recall type assessments. But another thing that we do, which I find cool, was whenever tools that our competitors come out, like Andrew Ng put out a cool paper reviewer thing targeted at CS conferences.

Ben: And what we do is we just run that thing, we run our thing, we put both of them into Gemini 2.0, and we say, could you please assess these side by side as reviews of the same paper? Which one caught mistakes? We try to make it a very neutral prompt, and that’s an eval that is easy to carry out.

Ben: But actually we’re in the market. We’d love to work with people who are excited about doing this for refine. We finally have the resources to take a serious run at it as founders. The simple truth is because my co-founder and I are researchers as well as founders, we constantly look at how it’s doing on documents we know.

Ben: And it’s a very seat of the pants thing for now, to tell the truth.

Andrey: Do you think that there’s an aspect of data-driven here and that one of your friends puts their paper into it and says, well, you didn’t catch this mistake, or you didn’t catch that mistake, and then you optimize towards that. Is that a big part of your development process?

Ben: Yeah, it was more. I think we’ve reached an equilibrium where of the feedback of that form we hear, there’s usually a cost to catching it. But early on that was basically, I would just tell everyone I could find, and there were a few. When I finally had the courage to tell my main academic group chat about it and I gave it, immediately people had very clear feedback and this was in the deep, I think the first reasoning model we used for the substantive feedback was DeepSeek R1 and people, we immediately felt, okay, this is 90% slop.

Ben: And that’s where we started by iterating. We got to where, and one great thing about having academic friends is they’re not gonna be shy to tell you that your thought of paper.

Refereeing Math and AI for Economic Theory

Andrey: One thing that we wanted to dig a little bit into is how you think about refereeing math and

Seth: Mm-hmm.

Andrey: More generally opening it up to how are economic theorists using AI for math?

Ben: So say a little more about your question. When you say math

Seth: Well, we see people, Axiom, I think is the name of the company, immediately converting these written proofs into Lean. Is that the end game for your tool?

Ben: I see, yes. So good. Our vision for the company is that, at least for quite a while, I think there’s gonna be this product layer between tools, the core AI models and the things that are necessary to bring your median, ambitious

Seth: Middle

Ben: not

Seth: theorists, that’s what we call ourselves.

Ben: Well, yeah. Or middle, but in a technical dimension, I think it’s almost certainly true that the median economist doesn’t use GitHub almost ever. If you told them, they set up something that, a tool that works through the terminal, think about Harmonic, right?

Ben: Their tools are all, they say the first step is, go grab this from a repository and run these command line things to, they try to make it pretty easy, but it’s still a terminal tool. So a big picture vision is that we think the most sophisticated tools will be, there will be a lot of them that are not yet productized and we can just make the bundle for scholars to actually use it in their work.

Ben: Now about the question of formalization per se, I have always been excited to use formalization in particular to make that product experience happen. For formalized math, my understanding is right now the coverage of the auto formalization systems is very jagged across, even across. If you compare number theory to algebraic geometry, the former is in good shape for people to start solving Erdős problems or combinatorial number theory, things like that, people can just start doing that. For algebraic geometry, there are a lot of basics that aren’t built out and so all of the lean proofs will contain a lot of stories that the user has to say, am I fine considering that settled or not?

Ben: And that’s not really an experience that makes sense for someone trying to check their econometric draft, right? So we’re watching and I think as soon as we feel it’s the moment when we can take the typical, say economic theory proof and give a rigorous certification, we’ll be right on.

Ben: I would like us to be in a position to be right on top of it.

Seth: I blame Grothendieck for algebraic geometry being hard to formalize, hard to make into Lean.

Andrey: Even short of things like Harmonic, right? It’s certainly you can get useful things of putting some math or asking for some math from Gemini for example. How are people in the field using those tools and have you noticed that has affected the type and quality of economic theory you’re seeing?

Ben: Oh yeah. That’s zooming out from refine. I’m obviously a heavy user of AI tools for my own research. I think broadly we’re seeing two phenomena play out in parallel. It’s a lot easier, this idea that went viral a few weeks ago of work slop being much easier to produce. I think there is an experience, which I’ve experienced myself, where you owe your co-author something and you have some ideas, you’ve done some real work, but it’s much easier to put a section in the paper that is AI written that looks a lot that our natural checks see as real work. And that introduces obviously new kinds of risk. It makes work faster in some ways and more fragile in others. And I think about that a lot. By the way, one of the main new values of refine is as people are perhaps less moment to moment engaged with the exact, or less line by line engaged with their work, which AI is doing. They need that global eye and that obsessive look, which used to be more in one’s own head. But that’s the negative phenomenon. But I think in terms of having a pretty expert consultant in things you don’t usually work on just for getting started and forgetting ideas.

Ben: I can already see major gains in my own research. One thing I would be curious to see is just looking at measures of production of scientific literature. We should see something on speed that’s visible in we should see signs of science speeding up in the areas which are particularly sped up.

Ben: And I, it would be fun to formulate a hypothesis like where should we be looking to see that

Seth: Right. We recently recorded an episode, the open AI paper on early uses of AI in social science. And it seems to us one of the most obvious immediate use cases is just, can I find if somebody already proved this and I could just plug it in? Right.

Andrey: to be clear, not social science, but mathematics.

Seth: mathematics. Excuse me.

Seth: Yeah. Yeah. Science, science is,

Ben: Physics. So yeah,

Andrey: Yes, exactly.

Seth: Andrey always calls me out that I say economics or social science when he really means, when I really mean actual science.

Andrey: Just to be clear, there were

Ben: important. Yeah,

Andrey: A bunch of math in that paper, which is very cool.

Ben: This is known. I think economic theory, it’s important to me about economic theory that there is really such a thing that’s called economic theory, very distinct from math. Usually, unless something is going wrong, you don’t need to do any interesting math.

Ben: In an economic theory paper, you just find the relevant. So I think a lot of economic theorists who are successful and good at it, a lot of the trade is finding the right thing, learning enough of it to make it valuable for your application and just using it correctly. And that’s where that search problem is really accelerated. So I’m with Seth that there’s gonna be a huge speed up just for maybe not as, it’s not super intelligence. It’s better search, but that’s huge.

Andrey: So one economic theorist that I’ve talked with about this is Joshua Gans. I don’t know if you’ve had a chance to talk to him, but he’s been writing a paper a week,

Seth: Right. The guy, he is grinding him out with the AI help

Andrey: Is there some sort of weird proof of work thing that’s starting to fail? Because look, writing down theories of almost anything, it was, it took a lot of work, but you could, there was a recipe, right?

Andrey: As an

Seth: you can mathematize Marx right. The fact that I can rewrite marks in math doesn’t necessarily make Marx good.

Andrey: Yeah.

Andrey: So how do you think about that and what do you think are gonna be directions in economic theory that are really changing the game as a result of this?

AI, Work Slop, and the Future of Economic Theory

Ben: Yeah. You raise an interesting point. You can think of one vision of what social science is, or what economic theory is, that’s suggested by what you just said, which is that we’re commentators on social reality and we’ve developed a particular style of doing that, which involves, in the case of modern economic theory, a lot of math and the proof of work.

Ben: There’s almost an equilibrium where you, in order to say something, you have to really carefully and write well in English, but also do this mathematics and now that, at least superficially can be totally hacked, is that gonna stop? Is that gonna make the commentary aspect of economic theory lower signal in some sense?

Ben: Is it going to, and that’s a great question. So let me table that for a second and say what? I have a thought on this topic that’s related to that. If you’re really good at that and you produce these really jewel like economic theories and then suddenly everybody can write slop and produce economic theories that at least take a while to distinguish from your beautiful ones, then maybe you feel sad, like your art has been degraded.

Ben: And I do think that’s the way poets, I think. I talked to some people who are very interested in the experience of artists with AI and I think that’s an artist’s experience with AI. Then there’s another kind of person I have in mind, which is an idealized cancer biologist.

Ben: And you tell them, oh, your jewel like blot analysis that you do or whatever. Now they’re gonna be automated. And I think this guy’s first reaction is mostly not, oh, how will people be able to admire my art? Will people still appreciate my art as much or what will I do with my time?

Ben: But they’re like, oh s**t, we might move faster toward curing cancer. So one thing I think is wrong broadly with economic theory is that there are a lot of us whose reactions fall more into the artist category. And I would like, I think economic theory is not done. In fact, it’s quite bad what we’ve achieved on the whole.

Ben: So we should be

Seth: excluded of course.

Ben: Yeah. So as a group, as a community, right? And so if we, I would hope that we have it in us to say, look, now we have these incredible tools to take a run at questions that are really where the solution would be genuinely valuable.

Ben: And we could really try to do them better. And we have this huge resource now. I would like it to be, I would be happier about us if we had more of that reaction. I’m hoping that there will be parts of the profession, parts of the enterprise that grow and accelerate, because they’re driven by that as opposed to hand wringing over the art problems.

Seth: Right. And it seems like you could always add some more, get gatekeepers on the backend. Right? If we just make it easier to enter with, here’s my mathie paper. And the concern is you get too much slop. Maybe there is some way to filter. You don’t have to filter on the math anymore. You filter on something else.

Ben: Totally. All of these offensive weapons are also closely related to defensive weapons. So there’s a whole, and refine is obviously a natural, we think about that, that we can, at least, at minimum, we can help reject slop that’s written by cheap models without much skill and maybe we can help

Seth: How do you defeat slop? How do you defeat slop with bitter slop?

Ben: Yeah,

Andrey: Have you talked with some editors? Is there interest here?

Ben: Yeah. So Refine is doing pilots with several of the very top journals in economics. And we’ve been really encouraged by, I think because a lot of the editors are super genuinely pro-social people who want to take the tech, who wanna bring technology to bear as fast as possible, to improve the profession.

Ben: And so we, and I think there’s a feeling that they have that’s correct. That this phenomenon is here, and so the best way for the journals, for example, to deal with it is to be as up on it as anybody. And so we, I think the main use that is the easiest sell is just final due diligence right before publication at the conditional accept stage.

Ben: Can we make sure that papers are, any remediable, any mistakes that the author would be embarrassed to have published, the author has a chance to learn about it. Correct. That’s, everybody agrees with that. I think there’s a lot more design required to do it thoughtfully when stuff is incoming.

Ben: I have heard experiences from editors using REFINE and other tools. When they get a submission that they’re very suspicious about, they can just quickly run it through refine, see that there seem to be, and they’re usually experts in, right? So they can see, oh, this is surfacing really serious errors.

Ben: Now I can, for example, desk reject it with a lot more confidence. So we’ve, that experience does happen. That’s purely people’s own use of the tools, but.

Andrey: Are you worried that your tool is fundamentally, it’s interesting. Like many economists, it’s a tool of rather than constructivism in that it’s very good at finding problems. But is it ever gonna be, well, this is not a perfect paper, but it’s a beautiful paper nonetheless.

Seth: GPT-4o if you wanna sycophant to Andrey.

Ben: Actually, one thing we think a small version of that, and I’m curious for your guys’ sometimes refined produces, you give it a 50 page manuscript and it produces six comments. In fact, one of our engineers recently switched. He said, we switched to a new, we did some model upgrades.

Ben: And then he looked at it and he said, this only produced six comments. And it was on a paper by one of our friends who had been through refine and all the mistakes were gone. And so he was like, oh, it went from, if I just run this on the dumber models, they give me 50. Now it’s six.

Ben: And that was actually good because the feature question we have is in that case, should we tell the author, Hey, this has fewer things we can see wrong than 95% of papers. Right? That’s turns this question mark experience into maybe something encouraging. So we haven’t rolled that.

Ben: I’m curious if you guys think such a badge would be pleasant for an author.

Seth: Question mark experience.

Andrey: I, I, think you should, well, you should obviously run the experiment,

Viral Processes and the Refine Referral Program

Seth: Uh, maybe an interesting place to start is this referral program that you came up with. So where did that come from? Why did you design it the way you did?

Andrey: You just, well explain it first. Yeah. I think that’ll be the first. Yeah.

Ben: what we have, we actually, we, through the end, through the end of Decem through the end of November, we ran our, our first iteration of our referral program, which we will keep, which will tune and keep running, in various guises. And the way the program works is you, if you refer a friend, if you want to refer friends, you get a referral link from the site. You can share that with anyone you want. And every time somebody, if somebody that you refer ends up actually, paying for a full refine review, at least one, they, they get a full bonus review and you, the referer get one. So we, our, our top reviewers, I don’t think you’ll mind me sharing ‘cause he, he told, he basically told everyone he knew, but Joshua Gans, he, he was, he’s like, I think he has like 35 credits now because he just kept referring and

Seth: God bless.

Ben:because my co-founder, my co-founder and I were talking and we’re like, this is than we expected, should we’d be worried about.

Ben: So we were like, no, this is only good. This is, there’s nothing to be stressed out. Um, he can have, he can have lifetime refined use, free for, for being such a good, but that’s what, so I think economically, I think there are two thing. One, one immediate thing to think about is that some people are gonna be really good ambassadors for your product, but you don’t know who they are.

Ben: There’s an information problem and a referral to the extent, and interestingly, they’re the ones who are gonna value the credits, if they’re really good users of it, and they’re also gonna be the ones that, probably can identify others who know. And so getting those people to raise their hand, is not a trivial problem if you just had to do it without, but it turns out this, it, offering the referral to them kind of puts the incentives in the right place. And then, the others, obviously the other lens that I think of it through is, the lens of network economics and the viral process. So I, I’m happy to talk, but I actually, the information one, when we were thinking like, who should we recruit as an ambassador? It wasn’t obvious. And this got them to come forward.

Seth: You’ve done some work, I think, both in, definitely theoretically, but maybe even empirically too, about optimal seating. So did that, any results from that play in?

Ben: That’s a good, I would say the, the most, honestly, the most important insight that kind of was really top of mind for me was what I, in an, in my undergrad networks class, which I teach from, Networks, Crowds, and Markets by Easley and Kleinberg, they go through the basics of the viral process

Seth: Will Jackson be insulted that you don’t use his book?

Ben: well, no, ‘cause it’s, it’s graduate book.

Ben: I

Seth: Okay.

Ben: every year. I do say, you can go buy, you can, if you really wanna know everything, you can buy Matt’s book. But so,

Andrey: yeah, just as context for the listeners, Matt Jackson was Ben’s thesis advisor. Yeah.

Ben: and yeah, collaborator and overall hero. So I, and it’s funny because I, yeah. Small aside, but when I teach that class, I’m like, ‘cause I realized from these undergrads perspective, Matt Jackson, like, if you read these books, he’s just like, they think he’s probably dead. Like, he is like, seems like a very major, a major part of the field.

Ben: And then I drop somewhere in the middle of the quarter, like, oh, Matt was my, Matt was my advisor. Um,

Seth: Not dead yet.

Matt Jackson as an Advisor

Andrey: talking about this, this is a little bit of a tangent, but I hope you don’t mind Well. What was he like as an advisor?

Ben: oh yeah, he is, he was ama I mean, overall amazing. Like, I, I, the main thing to say about it is I met him right as he was about to move from Caltech to Stanford. I came to him as a Caltech Summer research intern student. He didn’t really havetime, but somehow I, I tricked him into like, not, to, not to being officially on the, on the program.

Ben: Uh, my advisor in the program. And then we, we started working on our first papers on social learning and information aggregation right then, and. He, I think he’s ex, the most salient trait of him is that he is just incredibly supportive and encouraging about research, but actually not at all. There was very little teaching that he ever, he ever did, explicitly, here’s how you do research. Everything I learned from him was, was ‘cause he was open to co-authoring and I just saw him do research and I learned by, by apprenticeship. my dad had actually told me that that was the best way to learn and I, and but he had like Soviet physics, in the 1970s as his reference point.

Ben: So I was pretty sure it was not good advice, but it actually ended up being exactly what worked for me, with Matt. But Matt was not, Matt was not prescriptive he didn’t, I don’t think, I think his, his default mode of advising is like, because he’s so incredible at research. He, his first best advising style is to leave the student alone and let them, and let them do their thing.

Ben: And one, and I, it made way more sense to me when I talked. I, I think I talked to him about. His experience with his advisor, Darrell Duffie. And I learned that it was just, it was all this dynastic thing where Darrel was exactly the same way. He just, like, Matt brought him a thesis and Darryl was like, this is really interesting.

Ben: This is good. They had been writing other papers, but that was the extent of, and I, I don’t, Mike’s Matt was more, was definitely a great mentor, but I think it was really freeing to have someone basically just trust you to do re to do research and be there as a, be there to teach by example when you needed it.

Eigenvalues and Network Dynamics

Andrey: here’s a question. Who likes eigenvalues more? You or Matt Jackson?

Ben: Definitely me. ‘cause Matt’s not, Matt’s not a math nerd. Matt. Matt is a, Matt really is a true, true, true social scientist. He’ll use whatever tool. I think there’s, I’ve always felt a little sheepish that this aesthetic thing of like, what, this tool is really like special to me. He’s, he’s not like that and I think it makes him a better social scientist that he’s not.

Ben: Whereas I, ‘cause I think when you, whenever you care about something other than explaining the social world, that’s gonna be like, a trade

Seth: Well let, let’s slow down for a minute for, people in the audience who don’t live with the, in the, in the glorious glow of the eigen value. And, thinking about eigen vectors of Jacobian matrices, can you give us a little, give us a little taste to someone who’s already not in love with eigen values?

Seth: Why should they love eigenvalues?

Ben: Yeah, that’s a great question. Well, so, okay, 0.1 is, algebra describes the world. You guys know that video where the guy that the, the math profs or like, like sweaty t-shirt math guy is yelling like, functions. Describe the world. I think the real thing, linear algebra describes the world, and I think in the AI era, we, we don’t, as Tyler Cowen says, it’s Rise.

Ben: Tyler Cowen says it’s rising in status. So it’s quite high in

Seth: There we go.

Ben: the tough thing about matrices is that they’re so damn complicated. There’s like, matrices, you can the, the whole world into that. And the amazing thing about. Values is that they, they answer the question of if a matrix had to be a number, what number would it be? Like if you, if a matrix lost its privileges of being, of being an end by inbox and couldn’t store all that information, you have to masquerade as a, as at, at worst, a complex number. What complex number would it, what, what mask would it put on to be itself as a number? And eigenvalues are a wonderful way of, of fully answering that question is the best you can do. And that’s like, that’s a powerful idea. And, and I, and so back to viral processes, if you think about a viral process unfolding in a network, there’s a way to model it as a matrix or a network with all of the, the sort of, activation events being modeled as like basically a big matrix, multiplication, that prop that makes your state kind of, yeah, for the, I guess. Yeah, I don’t wanna, I don’t wanna, I understand that this is probably not the most intuitive way of describing it, but it is really true that if you have a large population and you wanna track the evolution of a state like a virus, you can think of that as kind of a matrix operation that acts on the system and updates it to the next step, which is like the thing spreading further.

Ben: But often what we wanna know about a virus is not everything about how it’s proceeding, but we wanna know something simpler. Like is it like when back in COVID, is it tending to spread right now or is it dying off? Right? And so it turns out that you can compute an eigen value of a suitably defined operator or, or something that will answer that question.

Ben: And so when you’re trying to run a viral contagion, as we are at refine to get more people aware of our product, we are trying to get the viral coefficient, above one. And

Seth: Right. Okay. So yeah, so tell me what, what’s the special thing that happens when an eigen value goes from below one to above one?

Ben: Yeah, well, let’s think about numbers, right? I said so, sowe have this, this process that we’ve now distilled down to one number, the viral coefficient. And we’re, we’re doing that process, namely the next step of the, of the epidemic over and over, right? The next moment when the epidemic has a chance to do its thing, and mathematically taking a time step is applying the, the operator of the epidemic’s behavior to the system.

Ben: So you have a system you hit it with, you say, okay, one more time, step. When we compute the, the eigenvalue kind of captures just the overall extent, captures how a number. And if that number is above one, it means every time it acts, that process tends to expand the set of infected people. And so if you’re doing it over and over, you think of a number greater than one, like two.

Ben: If you keep

Seth: One of my favorite numbers greater than one.

Ben: Excellent. My, my favorite. Um, if you have two and you keep hitting it, that is multiplying it with two, you keep getting bigger and bigger and that’s exponential growth. And it’s, it’s actually, it actually works with 1.01 as well. Right. And so if you, the la the largest iGen value of the propagation matrix captures exactly that.

Ben: Is there, when, when you keep hitting that system with itself again, does it behave like raising two or 1.01 to higher and higher powers? That’s when you have expansiveness, that’s when you have viral spread.

Seth: if my eigenvalue were 0.9, my viral spread would be I contaminate 0.9 people who contaminate 0.9 people, and that adds up to a finite amount instead of everybody gets it

Ben: Exactly. And so,

Seth: now, tell me what a complex eigenvalue is.

Ben: no, not today, but I will, what,

Seth: It’s not, it’s not, it’s not an, it’s not an interview on Justified Posteriors if the guest doesn’t refuse a question.

Ben: But, but, I will say is that I, what I, what I taught in my undergrad class, what, the way that I sort of like, like tried to get them, maybe even a little more excited is, you, when you think about that tipping point 0.9 to 1.1, it doesn’t look like a big deal. Um, locally, it doesn’t look like a big deal when you super zoom in on the, on the process.

Ben: But when you look at the process’s overall behavior, it, it makes a huge difference. And so what I to what I tell the business minded undergrads that I often teach is, if you’re running, and this was always just a fanciful little illustration to me, if you’re running a company and you’re running a viral promotion, you really could, you might be willing to invest a whole lot of money to move that number only a little bit because

Seth: Infinite return, dude.

Ben: yeah. If you, if you can push it, that’s where the returns to that are very big. And so we’re, and I amusingly, I think we’re right there. I we’re, I think our viral coefficient for this referral program is just about one. I can talk about some subtleties of estimating that, but that means, one of, one of the ways that we wanted to build it is we have that to have prices in there.

Ben: So the, the, the rewards you get are a price, right? And we can in principle give you, give your give, change the price, give people more free stuff or roll lower, make it an introductory offer with a, and those are the things we can tune to change the viral coefficient.

Andrey: And I guess the other thing in practice to remember is that the viral coefficient isn’t constant.

Seth: Ah, right. So does linear algebra describe the world when it’s like a first degree Taylor approximation? Actually.

Ben: Well, the beauty of, yeah, the reason it’s not co like yeah, it’s not constant over time. And one of the reasons it’s not is because as your contagion pro propagates through the network, it’s hitting different people. Right? Um, and that’s definitely something that of course as Andrey as, as you both know, and Andre, and I have talked about is that the selection of people as any kind of, of social phenomenon, like a an advertising campaign is progressing.

Andrey: I.

Ben: getting as the next rung is, is different. And eigenvalues actually do capture that from a nerdy perspective. Like if you just had to the, if you teach the simplest possible model where you just, like everybody has three friends and they infect these three friends with some probability, there’s no room for heterogeneity.

Ben: But if you take a whole network, then actually the heterogeneity is in there and the heterogeneity is, is exactly captured by it. And so in some sense, the largest eigenvalue will tell you the average of this across the whole network. So there are tools, of course when you’re doing it in real life as I’m now you’re just tuning the knobs andyou know, doing it in a somewhat less scientific way.

Andrey: But I’ll, I’ll just say that like after this podcast airs, will have been infected, so

Seth: Yeah. Oh man. Your I, dude, we’re getting your eigenvalues up there. We’re boosting your eigenvalues as we speak, dude. Okay. So we, we talked a little bit about, contamination of like viruses, but now let’s talk about an even more insidious form of, viral contamination, which is the idea or the meme, which contaminates us with, mental illnesses such as good taste in movies.

The DeGroot Model of Social Learning

Seth:Um, I guess if we were bringing these ideas of linear algebra to, social learning, we would think about this thing called the DeGroot model of Social Learning. Can you tell us a little bit about what that is? And then we’ll kind of build up to why wouldn’t that be a good way to learn, and how will AI help us think about that?

Ben: Yeah. So the DeGroot model is just, and I, I, I used to call it the averaging model of social learning, is actually what I worked on with Matt Jackson when I came to him as an undergrad. Um, at Caltech in 2006. I, like many other had rediscovered. Um, the dud model just says, you form your opinion tomorrow by taking a weighted average of what your friends think today. You can forget the weighted part if you, it’s not that important. So I just look around and my friends, I say, what are, what do they think about whether AI is good for humanity or whether, whether, you know. Um, you should throw away all your black, spatulas because they have toxins in them. And, and then for on issues like that, people form sort of an opinion by, by social communication.

Ben: And the DeGroot model is the simplest possible model. And we can come back to this. It’s, it’s one that economists actually don’t tend to love when they first encounter it because it is extremely simplistic and kind of, robotic or animalistic. You just, you just take the average. And if you have a bunch of people doing this, that can be summarized with beautiful linear algebra, which is actually exactly the same math, more or less as the math that you do for Markov chain theory. So, that’s for the nerds. But sociologically it’s interesting be because if it, because you can immediately start asking questions like, will a population of people updating this way reach a consensus and will that happen fast or slow? And will this consensus be right or wrong? And it sort, it gives this tool, which is like a pocket calculator that, that, um. Anyone with a reasonable applied math, education could, could have reinvented as in fact many people, including me, did. And, and then, but you can immediately take it to also, I think one of the reasons it’s been, so popular in economics is just it gives you a lot of ways to ask simple questions and get answers, which is something the, I can talk about it, but the standard economic models of learning don’t actually tend to give, many answers in networks

Seth: What would a large versus a small eigen value in a DeGroot learning network mean?

Ben: so in the, the first eigenvalue, which is the first one people talk about, the biggest one happens to always be one for a DeGroot model, which captures the idea that everybody is averaging. So in some sense aren’t getting, there’s no natural amplification or shrinking in opinions, because if you’re averaging, that’s sort of like the, there’s an eigenvalue, which just captures that fact

Seth: There’s no way for our opinions to fly off to infinity. I guess maybe if I was like negatively waiting you could that happen?

Ben: That could happen actually, but yeah. But if you, but with normal, with sort of the, the first, the natural assumptions on weights, things will tend to stay confined

Seth: know. Having negative weights on some people’s opinion seems pretty natural to me. If you’ve been on Twitter,

Ben: I have an under, I have a brilliant undergrad thesis student right now who’s studying

Seth: ah.

Ben: negative weights in the root model. But, yeah, so, but there’s a, another eigenvalue, the second largest. And what that captures is, is a society converging fast or slow. So the second largest eigenvalue of an updating matrix, if it’s really close to one, that basically means that. You can, you can start people off. And even if the society is connected and people will eventually be tending to the same opinion, if they talk for a million years, it really will take a million years. They, the, the being close to one captures their being. And it turns out, as Matt and I, Matt Jackson and I discovered to re relate to this phenomenon of homophily, that if your network is basically if and only, if, the only way that can happen is if there are divisions in your society where people put very little weight across Democrats and Republicans or whites and blacks.

Ben: Uh, andso if that happens, you can converge really slowly and if it, and if the second eigenvalue is, know, not too big, like 0.7 or 0.5, then disagreement is gonna decay like what you Seth was saying before, 0.5 to the end, right? So it gives this beautiful one number measure of the slowness.

Andrey: what if, what if, one of us was very stubborn and just didn’t really care what other people thought about them? Would their opinion end up dominating the entire belief process, or were they just washed away in the average?

Ben: Oh, if, yeah, so, so if there’s someone who’s super stubborn, they don’t listen to, the extremists, they really don’t listen to anyone. They put all their weight on themselves and

Seth: Those are, that’s our rival podcast. Dogmatic posterior.

Ben: Exactly. So, yeah, so that’s, that’s a way to be very, that’s a way to be very influential. In fact, at the extreme, wewouldn’t even call that society connected because this one guy’s not really connected to anyone.

Seth: It might be connected out. I don’t know. Maybe.

Ben: yeah. But even if he puts a tiny little weight on others, if he’s stubborn enough, he’ll still dominate

Seth: And would that be bad?

Ben: usually. But unless he’s very, well, unless he’s very well informed, unless he, and so yeah, we, we ordinarily consider that bad because. A benchmark we like to, in a realistic case, we like to think about is information is dispersed. Everybody. Nobody know. Nobody knows God’s truth. Exactly. But everybody has has reasonable Yeah.

Ben: Nobody has

Seth: The average of this room knows God,

Ben: Exactly. Exactly. We do. you, if you could take, if you could take the God’s eye view and look at everyone’s information together, it would be enough to tell you like a whole, whole lot. But nobody, but everybody’s individual estimates are pretty, are pretty noisy. And so now how do we, how, can decentralize social learning, which DeGroot is supposed to be a simple model of get you to that.

Ben: Well, it really depends on whether one guy monopolizes all the influence or a few guys or, or di, whether influence is dispersed.

Seth: As, as the population goes to infinity, do we have, influential nodes, right, is the way you put it.

Andrey: So,

Seth: gonna ask the LLM question? Andre? You go for it.

Andrey: one second,

Seth: One sec. We’ll get there.

Cow Tipping and False Beliefs

Andrey: Ben. I don’t, I don’t know if you remember, but we, we’ve actually done a podcast before.

Ben: I was thinking about.

Andrey: Now. In that podcast we discussed the interesting phenomenon of cow tipping and how people seemingly believe that this is a thing that one does, even though no one actually goes cow tipping. So my question to you is, the past since

Seth: Thanks for ruining the joke, Andre, for literally everybody.

Andrey: Uh, in the past, year since, since we’ve done the podcast, have you noticed any social learning on this topic? Is it now understood that cow tipping is not a thing or is it still a belief that’s propagating

Ben: That’s very interesting. I have stopped using it as a, I, I somehow found that I have not used it as an undergraduate teaching example since COVID, now that you bring it up. So one thing, something happened to me during COVID teaching. I was teaching my, this was the last year, 2020. I was teaching the last undergrad class I taught at Harvard in fall of 2020. And it was a wonderful group of students actually, but they were all dispersed. Some, most of them at their homes. A few of them lived in like group houses with other students. And I was doing the cow tipping lecture in the way it goes. Just for the, to a little more context. Yeah. So like, it’s a great,

Ben: how many people know what cow tipping is? One thing I’ve noticed by the way, is fewer hands go up because I think Varsity Blues and that generation of movies was an important, was the way that it got into the culture. And kids these days don’t have an, watch those movies. So I don’t know whether they’ve been exposed, but, but these kids sort of knew, they were like, I was like. I asked, the usual question is I asked some factual questions about it. Like, what do you think is the prevalence in the United States? How many incidents of cow tipping have there been in the last year? And people will say, very few people will say like a firm zero. Um, but in the Zoom class, one of the students, they had their, like, their apparent or a relative in the background, and they were like, no, cow tipping happens.

Ben: I’ve seen it. So then I had to, like, in the middle of my class, I have to interview this person to, assess like whether my whole understanding of things is wrong. It wasn’t a very exciting, I was like, well, did you see it? Like, what, what did they, what did

Seth: Is the cow tipper in the room with us right now?

Ben: exactly, they were like, they were like, well, they, they were drunk and they really like ran at the cow and they hit the cow.

Ben: And I’m like, then what happens to the cow? And they’re, I don’t know, I ran away. So that’s the usual, that’s like

Seth: Are you saying that, the eigen values of the cow’s response to tipping are less than one? Is that,

Ben: Exactly, yeah. Is I, values are very important in mechanics. So. But for the other piece of context, en engineers have written papers kind of proving that you can’t under reasonable assumptions, like, knock over a cow with your shoulder or

Seth: are you gonna tell us that Santa’s not real, dude? What is this podcast about? We’re just killing people’s joy. Or, anyway, I’ll let you finish your example.

Ben: In terms of false beliefs, I think things are bad. I think my, my naive sense, it’s very hard to know ‘cause we don’t, you have to really study it and scientifically, but we had like a, since my wife and I have have, had a baby, we’ve interacted with, like, we had a baby nurse live with us for three years and she, she was from a very different community.

Ben: You know, she’s like, and I heard things her friends were saying, and beliefs and my, my sense is that. Strange beliefs about matters of fact are very much out there. And, and I, and I feel like TikTok, I think like TikTok propagates them actually in a way that’s more powerful thanany vector I knew that I personally experienced.

Ben: Like when I was in high school, for

Seth: Is that interesting? I mean, is that surprising from a DeGroot perspective? ‘cause it seems like in from a DeGroot perspective, you get communities with weird beliefs ‘cause they’re disconnected. But now the statement is they’re connected and that’s giving them weird beliefs.

Ben: I think what the basic DeGroot model is missing is that people talk about things very, that that people’s propensity to, to. First of all, I don’t think like these beliefs, like claims of cow tipping or other urban legends or, or wild statements about what Hillary Clinton does recreationally are like, I don’t think they’re like deru where we average what people think.

Ben: You just propagate interesting information. And I think what the DeGroot model is really missing and a lot of models of social learning is that what people share depends a huge amount on whether they think it’s interesting and like surprising and much less on whether it’s true. And moreover, people don’t adjust for that when they hear, right?

Ben: Like Tyler Cowen might, but most people, they’re not, they’re not aware of that bias in the information they’re hearing. And so they’re not, adjusting their posteriors. They’re just kind of accepting, you know? And, and so I, and I think TikTok has made it much more power, much more, much more viral to say something really interesting and get it into a lot of minds.

Ben: And that’s more like a yes on or off viral state, not like, do you believe, not like. What, what do you think the interest rate’s gonna be next, next quarter, but more like, do you think that people really landed on the moon, like a yes or no? Or you do you believe in some crazy conspiracy that’s like, like more like a virus that takes hold of you and it’s not a matter of degree of belief.

Sequential Bayesian Learning and Herding

Seth: Well, so if people, if people aren’t good bayesians, another model that you’ve worked with is called, the, or sorry, I guess a Sequential Bayes. If people, if people aren’t learning this connected way, maybe they’re learning in this kind of sequential, sort of herding-y way, which is sometimes called a Sequential Bayes model.

Seth: Uh, Andre, are you gonna let me move on to this topic? Or you wanna jump in with something?

Andrey: make a, I wanted to make a very brief observation since we’re talking about this. I happen to notice a book in the, in the background of, of Seth, actually The Hype Machine, which is

Seth: My machine with Ana roll. Yes. What’s, yes, what he says. It’s, it’s not true. Things that spread. It’s, novel and emotionally intense things that spread. So shout out to, a friend of the show. Sinan Aral.

Seth: All right. So, yeah. All right. So pe, so pe No, that’s good. No, that’s good. So people don’t learn in this connect way.

Seth: Maybe. Maybe, maybe they just see what the last guy did and try to figure out the state of the world from that. Is that a better model of what you’re describing, or is it also wrong?

Ben: I think what I’m describing some, some, like, having in mind intending to propagate, a little pellet of false information, like people tip cows. I think that’s just like a virus and that’s a good model. It’s also not be irrational. I mean, I think there’s some rationality to it, but I think the best model of it is like, if it’s interesting enough, it goes viral and a lot of people believe it, but Seth absolutely, like the models, Bayesian sequential updating where you hear something. I think where that model really shines is in thinking about something like, which, you know. Should I get, should I get flood insurance for my house or which accountant in our, there’s like three accountants in our industry and which one should I use? I think there, people think very much like what that model posits, which is I could research this, I could get my own signal.

Ben: I don’t have any special confidence that I would be particularly good at that. And this other person, I know that what they, that they’re not probably acting on amazing information either, but it’s probably still got a little more information content in it than mine. And let me just, so let me just follow and so you end up with a lot of like in economic context that I think are important.

Ben: I think the, the choices people make about insurance. Like when I talk to people their, who thought their whole lives about do people buy enough fire insurance or flood insurance or whatever, they basically talk about it like a social convention. And so you, you buy some and you don’t buy other, and you don’t buy stuff that people around you don’t buy.

Ben: Not because you’ve taken any time to analyze your personal, portfolio problem, but just because you assume other people have it like. That the social signal contains more information than you’re likely to gather.

Andrey: There’s also an interesting aspect of it, like if you follow the herd, then even if it goes wrong, you’re like, well, who can blame me for, for doing that? Right? But if you go against the herd, like, oh, that idiot didn’t buy insurance. Like he deserves what he, what he got. Right?

Seth: You have to get an awful, strong signal.

Ben: in a business context, right. There was this saying nobody ever got fired for buying IBM because, and that was exactly hurting on IBM, that at the, are you gonna really get blamed for using the same vendor that everybody uses?

Seth: So, how does, so is, is that great? We all coordinate on doing the right thing, or can that fail somehow? Why, why wouldn’t that be a good approach to learning?

Ben: You absolutely get big. I mean, the main was, the main first result about the herding model is that you can get quite dramatic failures of information

Seth: Oh no.

Ben: Where? Um. If people did experiment, if people, if we could ask like the first a hundred people to make this decision to ignore the social signal or just deprive them of access to other people’s past choices, and we made them decide based on their private signal, then we’d get a hundred hunches aggregated, and that would, and then after that we’d have a hundred people’s information, averaging into some vibe about what the sensible thing to do is.

Ben: But, but the sequential model shows that if you, if, if the first people already are contaminated by having access to previous decision makers, it’s just rationally they won’t get this started. So you have a kind of tragedy of the commons where collectively, we could like. Maybe compensate the first movers or just pick some of us to be unlucky and have to make this decision solo. And we would, society would learn a lot that way from, but, but what we in fact do is just, herd and actually online platforms spend a lot of energy thinking about like how to get enough experimentation going on. You know, should Google re Google Maps recommend, shortcut that it doesn’t think is the best to learn about it, should Yelp send people, try to send people to a restaurant that it doesn’t think is the best to get more information about it.

LLMs and Information Aggregation

Seth: How does LLMs change all this? Alright, so I’m kind of split ‘cause I kind of feel like these two models have different implications for whether it’s gonna help or hurt with aggregation failure. So help me out with this. It seems like in this sort of sequential Bayesian framework, LLM sort of should hurt our information algorithm, aggregation, right?

Seth: Because, nobody is in the position of being ignorant. We can always just question the model. The model tells us what the last hundred people did. Uh, we’re gonna herd harder by virtue of all having, none of us being in that state of ignorance, that state of blissful archipelago ignorance. Do you think that that is a mechanism that’s potentially at play?

Andrey: Wait, Seth, can you just clarify something? Why

Seth: Please,

Andrey: LLM tell you what the last a hundred people said necessarily? I,

Seth: it’s gonna tell me what the last hundred books written about the subject are. Let’s say.

Andrey: I mean, we can take that as a premise. I’m not sure if I’d buy it, but,

Seth: I mean, well what are they? They’re based on, this is what I’m trying to say is LLMs are based on the things LLMs have read. Andyou might say maybe this is a version of model collapse, right. LLMs are based on the last hundred on some thing of some of the last things. The LLMs read

Andrey: The last

Seth: just the last hundred tokens.

Seth: And then, somebody reads that and then they write a book based on having read the LLM. And now we get herd to whatever our opinion was in 1850.

Ben: What do you think buying it?

Andrey: no, I mean, I just, I, I guess it depends on the decision, right? But to, to the extent that models are able to reason and to the extent that your,

Seth: What if it’s a pure fashion question? What if it’s, what if it’s just black shirts are in versus white shirts are in? Could it, could it lead a stronger herding there?

Andrey: Well, it would rationally know that you don’t wanna wear what everyone else is wearing. Right. I mean, I mean, there’s a, there’s an element of like, that it can really be, have a lot of context about you, which is different than else.

Seth: Yeah.

Andrey: that’s, that’s the aspect where I’m not exactly sure that that’s how we should model it, but I’m happy to consider that version of the model.

Andrey: Sure.

Ben: Um, yeah, I’ve never thought, I haven’t thought about it in a sequential learning setting exactly. But I think there’s a different, a different dimension which seems related and important, which is like a narrative that I’ve heard repeatedly and that I think has a lot of truth about what’s happened to western society and politics is that there used to be, a focal provider of, of focal baseline, of facts, basically

Seth: Catholic church.

Ben: well, I would say the six o’clock news,

Seth: Six. Okay. All right. I always wanna go. I always wanna go back to Habsburg times. Dude, you can see this is my Habsburg wall.

Ben: I don’t know. I, and I think this was probably a unique moment because I’m not sure, I think that, that the newspapers we should ask like, Gentzkow and Shapiro about, newspapers in 1900, which was I’m sure a very different, environment with all. But like, there’s this moment which is now kind of seen, which is, valorized a little bit, that there was the, a national truth and you could, you had to get pastsome regulatory, there was regulatory exclusivity for the major broadcasters and basically nothing too crazy.

Ben: You could get broadcast too widely that Right. And then we move to this TikTok world where, where it’s a free for all. And, and it does seem like, that has some, the breakdown of a shared reality seems like an, something that’s happening to some extent and now coming like. ChatGPT. It’s, I think it’s a real empirical question.

Ben: To what extent in normal people’s normal lives does that serve as like the six o’clock news? Again, the coordinating device. Um, if you’re debating something, my wife Annie, who’s, who’s a also a Northwestern professor, had a hilarious story at a dinner she was debating. She went to MIT and she’s a big MIT snob and always reminds me that Caltech, where I went to for undergrad is way worse and is like way less cool.

Ben: And so there was, but to my surprise, her dinner can be, I wasn’t at the seminar dinner, but a guest of ours thought that Caltech was great. So I was like, the kids, it

Andrey: To.

Ben: and she was, yeah. And she was like, and he was like, wait, are you telling me that if you ask, you ask 10 people, they’ll all who, who care about this?

Ben: They’ll say that MIT is better. She was like, yeah. So of course they took out ChatGPT and that settled, and she,

Seth: Pirate, get John Horton on the phone. Tony Stark went to, Tony Stark, went to MIT Dude, that’s what people know about.

Ben: So I thought that was, and I think that’s gonna happen a lot around a lot of dinner tables and kind of, it has an effect. I, I think of it as a shared, I think of it as a powerful shared signal. Um, andI think that really reshapes things, in, in a lot of different ways. Um, that’s the main way I’ve been thinking about it.

Andrey: You know, it’s, it’s funny ‘cause what I, my very opinionated bias take is that the average quality of the undergrads atCaltech is obviously higher than at MIT in my experience, and I think a lot of people who know would agree.

Ben: Yeah, I think that’s, I think she’s been a little bit per, I think she’s been a little persuaded over time because my, my, my good friends, like the, the relationships I’ve kept from undergrad are, um. John Schulman, who was a, who was there, were two of the biggest ones. Or John Schulman, who was a, one of the, was maybe the, is often credited as being, a creator of chat, GPT andAdam D’Angelo, who’s, who is of course the co founder where I worked and and is a, is a very big figure in ai and I think that does you, there there’s a sort, so I think that’s made a, made an impression actually that there’s some kind of person that the place was good at incubating

Andrey: So

Seth: so

Andrey: is all listeners. This is actually all a ploy to get John Schulman on justified posters.

Seth: come on.

Ben: those two are Caltech alum in case it, it was not.

Seth: Uh, so, okay, so, so let me, so let me take that argument a step further. So, the way we should, one way to think about LLMs in the social information aggregation function is as being a central node that all of us are connected to. Um, we, you just reminded us that in these DeGroot models, having, influential node in the long run means that influential node gets to, set a little bit of the opinion and it might not just be the average of everyone’s opinions.

Seth: Is the concern there, or is the observation there that, whoever ends up controlling the most important three LLMs ends up having a real thumb on their scale in the opinions of society.

Ben: yeah, exactly. So, it’s funny when it, when Matt and Jackson and I were working on this in 2007, 2008, were very, the ba the basic first observation is exactly what, what you said, that if one person gets a lot of weight, they’re gonna, their errors are gonna matter. They’re gonna contaminate everything.

Ben: And so they’re gonna prevent, even if society as a whole has the information collectively to wash out all the error, the fact that this guy talked in a way, first or talked loudly, means that everybody’s going to be influenced by whatever. That note says, but there is an exception. Or when you try to prove those things mathematically, that’s not necessarily true because something that can happen is if that note is very good at themselves being an aggregator, and it actually does, it figures out the right information.

Ben: Um, and rebroadcast, that’s also one of the most efficient ways of figuring it out. So I think

Seth: A

Ben: the

Seth: post, a reliable pollster.

Ben: Exactly. And so the selfer, there’s something irritating about the Selfer, way in which some of these AI companies regard themselves, or it’s like that they, thinking really earnestly about stewardship of, of, the model’s preferences or whatever.

Ben: But I actually think this, that, it, if the model is say left bias, this liberal liberal bias, then that’s gonna, um. it into a lot of opinions andthat matters. And so they, they should think about it. And I, I do actually admire efforts that they make, to be basically good aggregators, good pollsters.

Ben: And interestingly, like before we could have pollsters on a few issues that you could distill numerically, but now this is a pollster that kind of up internet text about anything. It’s like a qualitative pollster, which is a really remarkable kind of device that we couldn’t have imagined when we were writing those papers.

Seth: Should we be RLH fing these models so that they have the median social opinion on all social issues?

Ben: What does that even mean? Right? How do you

Seth: I, you go to Pew and it says, the median person thinks abortion should be legal at 27 months. Whatever. What? Sorry? 27 months. 27.

Ben: But even that,

Seth: 27 weeks. Okay.

Ben: didn’t like. The interesting thing is that the LMS are doing their own embeddings of these issues into their, so people will just talk to them and say, and talk about abortion in a way. They’re doing an averaging but not one that’s, that’s, that’s numerical one that’s qualitative. And, and I, I kind of like it that way. I, I, I don’t think people have coherent views on almost any issue of public interest. And so if you try to make it numerical and try to average it that way, that would be like garbage and garbage

Seth: Right.

Ben: and.

Seth: Trying to recreate the mind of the median American voter will make you insane.

Andrey: I, I really wanna go back now to this personalization aspect of things, right? Um, it, especially with something like Chad, GPT, I don’t view it as a monolith. There is a model router involved. It has all your previous conversations. And if me and you asked it a question, and this is an interesting, it would be an interesting empirical exercise actually, is like. We might get a very different answer about like, is it, is it, normal to, I guess, I guess it depends on what we’re asking. It’s like one of the things like for myself, like, is it, should I wear a hoodie to a business meeting? Right. You know, and it might give me a different answer than you guys.

Seth: Did play League of Legends during the business meeting.

Andrey: yes, yes, Uh, but, but if I ask it, what does the average person in society think about this question? We might get the same answer, but I don’t know, these things are a little unpredictable in this way. Right.

Ben: Yeah, and there’s a bunch of

Andrey: I.

Ben: papers just suggested by what you just asked, right? If people, because of course the system prompt. If you’ve done a, if you’ve now had your custom prompt, all bets are off because you could, you could ask it. Please don’t tell me. Things that might upset me with this mental illness that I have.

Ben: And then they, we wouldn’t get probably accurate answers on, on if it’s really, then it has. So yeah, people do get, the personalization issue is super interesting. but for now, yeah, I just wanna make the point for the moment that as a focal before the market has matured to the point that there’s a niche little LLM for everybody, these items are actually new kind of animal in the, they’re not like Facebook, they’re not like they’re, they’re a new kind of sort of public object that everybody interacts with.

Ben: Um, and despite the heterogeneity that Andrey said, they, that’s, that might shift things in a way closer to a, a, a former time.

Seth: Or will people just all choose, I’m a lefty going in, so I’m gonna use lefty, LLM, and you’re already going in. You’ll use righty. LLM.

Ben: Right. But it is, isn’t it remarkable that gra, I mean, there’s like a popular Twitter joke, but after trying, after trying to train the wokes, the, sorry, the, the anti wokes, LLM imaginable, it has like, it has like wine mom views, like

Seth: You can only, you can only, you can only, right wing eyes, the LLM so much.

Ben: Yeah. Except on the rare, like, it’ll say, it’ll occasionally say Hitler is great, but other, other than that, it’ll like,

Seth: Only when it’s role playing.

Simulating Social Learning with LLMs

Andrey: has anyone tried to

Seth: Ooh.

Andrey: some of these social learning games with LLMs?

Ben: yeah, that’s, I, that’s a great, I I’ve been trying to learn, keep track of this. I, it’s been proposed to me by students. Um, and I know that there are people. That. So I was gonna say that when we, ‘cause before, before the podcast, we’d sort of discussed, some topics, and I’ve been thinking about this one that like, how will it affect social learning?

Ben: But it made me think, how will it affect studies of social learning? And now you can, you can, implement, you can simulate it, you can, try to forecast how groups of people would behave. And it’s interesting because people like John Horton have done studies of how good is it as a simulator of a, of an individual. the question of how good is it as a simulator of a community, would be super interesting. I think just intellectually, I’m sure people are doing it. I’d love to, if people listening are aware, I would love to like tweet it at me or something.

Seth: You heard it, folks, dm d dm, Ben, with all of your, simulation ideas

Andrey: yeah.

Ben: tweet.

Andrey: Well, I, I guess theclosest thing that I

Seth: posted on our Discord I’ll, we’re at the, we’re at the end.

Andrey: Yeah, is the, is the AI village, know, where the, there are like different ais, different models, and they’re like co cooperating, slash they’re given a task to do and they see if you can do the task. And some tasks are like, can you sell a t-shirt online?

Andrey: Or something like that. And it’s hilarious how they try to cooperate with each other and all their foibles andso on. Uh, which is kind of not narrowly the, the specific formulation of social learning, obviously, but related,

Ben: Yeah. Yeah.

Lessons from Quora and Startup Experience

Andrey:so one, you, you mentioned, your friend Adam D’Angelo. I’m curious what, what you learned, at Quora, that you’re bringing to your current startup experience, or alternatively what you learned at Quora that you brought to your research.

Ben: Yeah, that it was such a formative time that I really didn’t understand at the time, how important it would be in my life. That I think the biggest thing, I never thought I would, I never expected that I would do anything entrepreneurial just because, I think that for one, I didn’t expect that there would be a technology like AI that would be kind of like, have the exact shape that, that is, has been important for, for me to be able to actually try to do something, at the technological frontier.

Ben: But at that, but I was, what was remarkable to me is that I

Seth: Thought you said linear you, I thought you knew that Linear algebra destri describe the world and you’re the king of eigenvalues. Come on, dude.

Ben: No, but I guess I never had that deep faith or I thought it was a few steps away that I was upstream in

Seth: Mm-hmm.

Ben: the innovation

Seth: Fair enough.

Ben: of commercial applications. But I remember, like, it was huge for me that they, that they were, that Adam’s always been very interested in economics. He just reads, like he reads texts on industrial organization recreationally.

Ben: And, and I think he had, he always had this respect for economists. Um, that was very, and and so he would, we would just occasionally chat about things often through the lens of economics. And Quora had some specific, he had some economic ideas of for, well, one thing I did was moderation. ‘cause I was just a very active user.

Ben: So I was involved in kind of, some of the housekeeping of the moderation operation, which I actually wasn’t good at. So I, my, at the time, the interesting thing is I wasn’t like, I wasn’t a good community community manager and but, but when, then, when I was in the company. Adam got curious about this idea of credits and actually having an internal currency, and that so that people’s like, basically so that the scarce resource of some people’s attention, like, especially on early Quora, a lot of the answers were written by really visible people whose, who were, people were very excited to see them there, but their attention was scarce.

Ben: So how could you efficiently bid for people’s attention? You wanna create some kind of token, right? And so I was just like the consultant who, thought about the very basics of the design of that system, like the central banking. How much money do you issue it? How do you, but that was what I did. but what I learned was actually like just getting to watch a startup. And it was right at, when I joined there were about, I think 27 people. And so seeing a startup at that stage, I learned a huge amount about. About running a business andespecially in tech, I think the strongest, people often say that startups are like a magnification of the founder’s personality. Um, and I think that’s really true in this case. ‘cause,

Seth: Getting, getting how, how, frustrated it, refined was with some of my notation where it was like, you called this a node. I, it took me a while to figure out what you mean, but I would not call it a node. Uh, your personality really does come through.

Ben: it’s funny because, yeah, I’m very, I’m very pedantic. I, I’ve spent, I, I, I feel, yeah. So I’ve created, and Adam is very, very thoughtful and deliberate and kind of like likes to make decisions with principles and in a thoughtful way and make decisions, like I think a lot of good, good leadership skills, like focus on, focus on one focal goal at a time and and. Propagate that and communicate that. And then, think really thoughtfully about design The core was a very design first company andmaking design decisions, not as an afterthought, but as a core thing. I think there were a lot of those like principles, I think similar to growing up in families, like there’s just certain values that are embodied in where your environment.

Ben: And when I was there, like I realized after that I, I’m a pretty good sponge and I wasn’t directly involved in any like, decisions having to do with design, but you know, the guy I sat next to at Quora was, Joel Lewenstein, who’s now the, the head of design at Anthropic. And I can, and like, but I didn’t, I think what the amazing thing is, it was this like, combination of amazing people and all of them were really thoughtful and really good at what they did.

Ben: And they talked about startup uping in a very intellectual, thoughtful principles first way. And so that when I, I, when it came time to think about a business, I felt like. That was a natural way to be, and I realized I never would’ve had the, that kind of, those kinds of vibes, if not for those six or eight months that I spent there.

Andrey: Very cool. Um, do you have any thoughts about why more companies don’t use virtual currencies and have you thought about the use case of virtual currency for internal allocations of GPUs?

Ben: Great questions? Um, I think virtual

Seth: You imagine going to Walmart and they tried to pay you in Walmart coin instead of money, people would riot.

Ben: Yeah. Well, but you could, I mean, internal currencies. I think one of the problems that, I wasn’t around when Quora eventually decided to get rid of them, but I think one of the problems is that, um. Currencies are focal and they create people, they, they motivate people to do things in a way that they sort of take up too much oxygen in the ecosystem. And so when you’re designing a social product where you want many kinds of incentives to be in balance, having a currency can actually be harmful to the, it’s a kind of a sociologist insight, but like, so I think there’s some of, I think you have to be really, I think for platforms where that are truly transactional and economic currencies are always good.

Ben: And usually that currency becomes money. ‘cause it’s gonna have an exchange rate with real money

Seth: Right.

Ben: Um,

Seth: Love one price.

Ben: yeah, but for, but I think for, for. It is, I think it’s an interesting phenomenon that needs to be thought about more. Why it’s not, why it’s really generally not a successful route for social for internal markets. I, I’m very, I I believe that some of the obstacles to internal markets are just frictions having to do with like, basically contracting frictions. Um, and one thought that I have had for a long time actually discussed with, we had some there. Let me just, I, you guys will edit. Let me just say that again. One thought I’ve been thinking about for a long time is just as contracting intermediaries. Um, and

Seth: This is a big theme of the

Ben: Andrey

Seth: Coasian Singularity Dude.

Ben: Yeah. This is Andrey’s paper.

Andrey: Yeah. So what, what is your thought about this? Yeah.

Ben: I’m very curious, so I’m very curious for your take on it since you’ve thought about it much more seriously now, but it just, yeah, I think I feel like. A lot of the details were just like implementation details, that if it became your job to implement it at a company, you would, you would decide that it’s, you’d have to really have a high valuation of the marginal allocated efficiency of that currency. And it’s arguable that it’ll, it’ll be, it, I think experiment experimenting with it has just become way more valuable once we reach the LLM, capability of being trustworthy to like, negotiate a contract, which I think honestly is not right now, but yeah.

Ben: I, I see that as a potential, a big organizational impact. I’m very curious what you think.

Andrey: I mean, surely the contracting aspect would be hard. but I also think there’s a social aspect to it as well, right? You’re the CEO, you create an internal Coasean internal market for GPU resources, then you suddenly see a team that you don’t want using the GPUs, using a lot of the GPUs. Now, what do you do

Seth: The whole point of, yeah, the whole point of having a firm is to have a command DI economy. If you wanted everyone making independent economic decisions, you wouldn’t have a company right.

Andrey: but there’s a sense in which there’s some optimization that you want your teams to be making, like leaving idle GPUs or they’re using them very stupidly for some reason, and you don’t, you want that to be kind of disincentivized and. The way it’s currently done is through these very imperfect monitoring systems and people asking very nicely, can I have, this resource?

Andrey: Right? So yeah, I’m, I’m curious whether the, the AIs can do a better job here.

Ben: Yeah, I mean I guess the, you might shortcut you, they’re also becoming better at being the arbiters of requests. Right? So maybe, maybe rather than, but, but I do think money is, one memory I have of Quora actually is that the engineers, they hadbrilliant young people and I very like. Who were first principles thinkers too.

Ben: And so people would ask me also, I had to just like justify money to the whole, to like the skeptics in the whole company. And so I gave, gave a lot of thought

Ben: Yeah, why don’t we have some more multidimensional expression? Right. And there are good answers to that. It’s like very helpful that money is very legible.

Ben: That, but, but I guess we, yeah, for companies, I’m very much with Seth’s point that if you really believed in the power of the, of monetary incentives to, to do it, you, you wouldn’t have a company, but you may find it a useful tool within the command. I mean, even, even the command the North Korea has has currency, right?

Ben: So like it’s definitely a tool. And I think with the Pareto frontier has changed, but I don’t know how

Closing

Andrey: Very, very cool. So, we’re just about out of time. Uh, is there anything either of you want to add to our conversation?

Seth: Ben, do you have any good eigenvalue jokes for us?

Ben: oh man, I should have prepared.

Seth: Alright. We had Ben Golub today who’s made tremendous strides in automated paper reviewing and still has a lot of progress to be achieved on automated Eigenvalue joke, doing, thanks for tuning into this episode of Justified Posteriors. Please like, share, and subscribe. We now have a hoppin’ Discord community for now by invite only DM us on substack Twitter or LinkedIn for your personalized invite code.

Seth: And why don’t you keep your posteriors justified?

Andrey: Thanks, Ben.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit empiricrafting.substack.com