How to Monetize an Algorithm

In this week’s episode of The Mixtape with Scott, I had the pleasure of interviewing Kyle Kretschman, Head of Economics at Spotify. It was a great opportunity for me because Kyle is one of the first economists I have spoken to who didn’t enter tech as a senior economist (e.g., John List, Susan Athey, Michael Schwarz, Steve Tadelis). Kyle entered tech straight out of graduate school. He spent much of his career at Amazon, a firm that has more PhD economists than can be easily counted. Under Pat Bajari’s leadership there, Kyle grew and his success was noticed such that he was then hired away by Spotify to lead up their economics team. At the end of the interview, I asked Kyle an economics article that has haunted his memories and he said “BLP”, which is affectionate shorthand that “Automobile Prices in Market Equilibrium” by Berry, Levinsohn and Pakes 1995 Econometrica goes by. I really enjoyed this interview, and despite the less than ideal sound quality at times, I hope you will too.

But before I conclude, I wanted to share some more of my thoughts. This series I’ve been doing on “economists in tech”, which has included interviews with John List, Susan Athey, Michael Schwarz and Steve Tadelis, comes from a complex place inside me. First there is the sheer curiosity I have about it as a part of the labor market for PhD economists. As I have said before on here, the tech sector has exploded in the last decade and the demand for PhD economists has grown steadily year over year. Tech demand selects on PhD economists with promising academic style research inclinations. There is substantial positive selection in this market as firms seek out strong candidates can be produce value for them. This is reflected in both junior market salaries, but also senior. Job market candidates are economists with technical skills in econometrics and economic theory, not to mention possess competent computer programming skills in at least one but often several popular coding languages. They are also candidates who were often entertaining careers within academia at the time they entered tech, and in those academic careers, they envisioned themselves writing academic articles about research they found personally and scientifically important and meaningful. Going into tech, therefore, would at least seem to involve choice that may go far beyond merely that of taking one job over another. It may involve a choice between a career in academia and a career outside it, which for many of us can feel permanent, as though we are leaving academia. And for many economists, it may be the first time they have ever contemplated such a thing. If they do internalize the story that way, if they do see taking a job in tech as “leaving academia”, then I can imagine that for at least some economists, that may be complicated, at least.

But there’s another reason I have been wanting to talk to economists in tech and that is I am very concerned about the welfare of our PhD students. In a recent article published in the Journal of Economic Literature, economists interviewed graduate students in top economics programs. They found there incredibly high rates of depression, anxiety, loneliness and even suicidality. This is a common feature of graduate studies, but it is interesting that PhD economists have incredibly good employment opportunities and yet the depression and anxiety plague there too. One of the things that struck me in that study was the disconnect between what graduate students felt about their work and what their advisors felt about their own work. Many students, for instance, do not feel they are properly supported by advisers, do not believe their advisers care about their research success and do not even care about them as a person. Whereas most Americans (and faculty) feel that their work has a positive impact on society, only 20% of PhD students in economics feel that way. (I discussed the article as well as my own research on the mental health of PhD students here.)

I suppose part of me feels a great sigh of relief to see the labor market for PhD economists expanding in light of those troubling statistics. If students know that life is full of infinite possibilities, then perhaps they can begin to process earlier what they want to do in the short years they have on this small spinning ball of rock we call Earth. If students do not in the end want to become professors, if they do not have the opportunities to become one, they should know that there is no “failure” involved there. Careers are just that — careers. They do not tell us who we are. The sooner a student can detach from the unhelpful story that our value is linked to a vita listing our accomplishments, the sooner they can begin their own life work of choosing their meaning. Can having more labor market opportunities with more employers competing for them help do that? Well no, not really. At least, not exactly. It can disrupt certain equilibrium, but then the new equilibrium can just as easily cover that up too. Still, I do like the idea that to keep students in academia, universities and departments must fight harder for them, pay attention to them, and invest in them as people. I like the idea that students have more options and that the options are diverse. Will it help their depression? Well, that’s another matter, as that’s complex. And presumably the economists in the survey I mentioned were themselves well aware of the career options they had since they were coming from the nation’s top 10 PhD programs in economics.

I suppose my point is that ultimately, the burden of life really cannot be resolved with money or career. We are trained to look there because we have boundless appetites. But ultimately the hard work of navigating life can only be helped so much by a job. We must still decide for ourselves what meaning we will choose for ourselves. But one thing I know, and one thing which I think our profession is profoundly bad at saying out loud, is that if we make our identity connected to vitas, we will not just be miserable, we will be hopeless, and probably poisoned. Such a mindset leads to endless laps on a brutalizing treadmill of meaningless performance in which a person chases for first place in a race they don’t remember signing up for and which they cannot win. They compare themselves with others running, not knowing that they too are brutalized by their own treadmill, not realizing that it is impossible to catch up with someone else as there is always someone else ahead of us. The sooner we learn that the joy we long for will not come when we get a top 5, the sooner we can look elsewhere. It has taken me many years to relearn a lesson I learned decades ago — I am whole now. I am complete now. I still run, and I still chase, but I am not chasing completeness. I am not chasing my own wholeness. Being whole and complete has nothing to do with a career. Careers are ultimately orthogonal to hope, which does not mean they do not matter — they absolutely matter. But if asked to deliver meaning, we will find that our jobs are as weak as wet spaghetti at such a task as that.

So, I suppose in some ways I simply want to announce — there are incredible opportunities for economists inside government, commerce and academia. But the weight of this life is not likely to be lighter in any one of them, for the weight we feel in life is largely self imposed, inside us, in the stories we tell about who we are and for many of us who we are not. Those stories are real, because we feel them and because we believe them, but they are not true. All stories are wrong, but some are useful, and the story that our lives can only matter if we have certain types of jobs or certain types of success, while it may be useful to getting a paper out or accomplishing something important, in a much bigger sense it is hollow at best and pure poison at worst.

TRANSCRIPT

This transcript will be updated once the more complete transcript is finished; for now it was transcribed using voice-to-text machine learning.

Kyle Kretschman:Might not have prepared myself well enough to be attractive for some of the most pop most top tier schools. Scott Cunningham:In this week's episode of the mix tape with Scott, I had the pleasure of interviewing Kyle kretchma the head of economics at the streaming platform. Spotify. Before I dive into the interview, though, I wanted to give you a bit of a heads up about the sound quality. Unfortunately, the sound quality in the interview on Kaza side is a bit muffled. We discussed refilming. It tried to find a way to tweak it, but there were certain constraints on the actual sound itself that kept us from being able to do it. And we didn't feel that refilming, it would be good because we thought that the interview had a lot of serendipitous kind of spontaneous tangents and things spoken about that. We thought students and people in academia would want to know, would need maybe even need to know. And I doubted that I could recreate it, cuz I don't even know why it happened. Scott Cunningham:So I'm gonna post a video version of this at my subs, for those who feel that a video version would help them kind of follow it in so far as the audio might be at times challenging. So check out the subst for those of you that wanna watch, watch it instead of just listen to it, hopefully that'll help. I won't say much here by way of introduction, except to say a few things about Kyle, because I wanted to let Kyle tell you his story in his own words, cuz it's his story to tell. And it's an interesting story. Kyle's a PhD economist though from the university of Texas Austin, which is down the road from where I live and work at Baylor, where he wrote on topics in graduate school and applied econometrics, empirical industrial organization or empirical IO and public choice after graduating, Kyle went to Amazon, not academia. Scott Cunningham:In fact, given we might start the boom of tech hiring PhD economists in the early to mid 20 2010s. You could say Kyle maybe was sort of one of the earlier hires among that second wave of PhD economists that went there. He worked for several years at Amazon before being hired away by Spotify to head up and lead a new economics team there, perhaps this is part of a broader trend of tech firms building up more internal teams, not just of data scientists, but like Amazon departments of economists who knows recall though from an earlier interview with Susan athe where, when I asked Susan why she said pat Maja had done something amazing at Amazon, she said he made economists productive. And in time he made many of them productive and very in productive from what I've been able to follow. And Kyle is from what I can gather someone whose skills matured and deepened under the leadership of Papa jar at Amazon and other leaders at and other economists at Amazon. Scott Cunningham:And he was ultimately hunted down by a major tech term to create an economics team there I'm by no means an expert on the labor market for PhD economists. I just have been very intrigued and curious by the, the, the Mar the labor market for PhD economists in tech, because well, partly because of realizing first that cause of inference was really valued in tech, but then to sort of realize that there was just this very large community of economists there, but I don't think it's controversial to say over the last 10 to 15 years, the tech industry really has been disruptive in the labor market for PhD economists. They continue to hire at the junior and senior market in larger and larger volume selecting more and more on people who likely would've gone into academia into tenure track or tenured positions. They pay very high wages, some of the very, some of the highest wages in the country, both at the junior level and especially at the, at the higher end at the, at the more advanced levels, people can earn compensation packages by the, in the, by the time they're in their thirties, that many of us didn't know were possible. Scott Cunningham:It's in my mind, historically novel, and I might be wrong about this, but it, it seems historically novel that the PhD economists who likely would've produced academic research papers in tenured and tenure track jobs have begun to branch out of academia, but maintain those skills and maintain that research output. It's partly driven best. I can tell, buy Amazon, I might be wrong, but by Amazon and paja, as well as Jeff Bezos own view, that economists are what I guess we would just say value added for many firms. Therefore I'm continuing to wanna speak with economists in tech to help better trace out the story. This interview with Kyle follows on the back of earlier interviews with people in tech like John list, you know, a, a distinguished professor of economics at the university of Chicago, but also the former chief economist that Lyft and Uber now Walmart Michael Schwartz, former professor of economics at Harvard. Now, chief economist at Microsoft and Susan athe former chief economist at Microsoft professor at Stanford and now chief economist at the DOJ. I hope you find this to be an interesting dive into the industry. Learn a little bit more about economists there, but by, by learning the about one particular important economist, there a, a young man named Kyle crutch, head of economics at Spotify, my name's Scott Cunningham. And this is the mix tape with Scott. Scott Cunningham:Well, it's my pleasure today to have, as my guest on the mix tape with Scott, Kyle crutch, Kyle, thanks so much for being on the call. Kyle Kretschman:Hey Scott, thanks for having me really appreciate the time to talk Scott Cunningham:Well before we get started with your career and, and everything. I was wondering if you could just tell us your name and your title and where you work. Kyle Kretschman:Sure. Yeah. As you said, I'm Kyle kretchma, I'm the head of economics at Spotify, Scott Cunningham:Head of economics at Spotify. Awesome. Okay. I can't wait to talk. So let me, let me, let's get started. I was wondering if you could just tell me where you grew up. Kyle Kretschman:Sure. So most of the time I grew up in outside of Pittsburgh, Pennsylvania, about an hour north of the city, real real small town probably had one stop light. And maybe the, the funny story that I can share is what I took my wife there. She asked where's the Starbucks. And I said, no Starbucks here. There's no Scott Cunningham:Starbucks. Kyle Kretschman:Yeah. So pretty small town called Chippewa township in Pennsylvania. Scott Cunningham:Oh, okay. Is that near like Amish stuff or anything like that? Kyle Kretschman:No, that's the other side of the state. So this would be Western Pennsylvania about near the end of the turnpike, about five minutes from the Ohio border. Scott Cunningham:Oh, okay. Okay. You said, but you, did you mention, you kind of grew up in different places? Kyle Kretschman:Yeah. So before that, my father worked in civil engineering and so would do build roads and bridges basically across every, across the nation. So I was actually born in Louisiana, lived there with, I think for a whole two, three weeks. I don't quite remember. Cause I was pretty young obviously, but then Michigan and then spent some time in Philadelphia before moving out to Pittsburgh around second grade. Scott Cunningham:Oh, that's kinda like, that's like when people described their parents being in the military, just kind of moving around a lot. Kyle Kretschman:Yeah. A little bit. So, but Scott Cunningham:Then you settled in the second grade Kyle Kretschman:That's right. Yeah. So outside of Pittsburgh and then stayed in Pittsburgh through high school and even through undergrad. Scott Cunningham:Oh, okay. Oh, you went to undergrad in Pennsylvania. Kyle Kretschman:Yeah, I did. So I went to undergrad at the university of Pittsburgh. Oh, okay. It was, yeah. If, I guess maybe continuing the story growing up in a town with no Starbucks. I was, I was pretty intrigued by going to a city. Yeah. And find out that lifestyle and yeah, we might have lived pretty close, like an hour away, but we didn't go down to the city very much. So Pittsburgh was just really, really enticing for a city to, for, to go to undergrad in. And so I basically looked at all schools that were in cities and so the proximity plus then the, the ability to just spread my wings and explore what it's like to be in a city was really, really enticing. Scott Cunningham:Did any of your friends go to pit with you? Kyle Kretschman:Yeah, so there's probably, I grew, I graduated from a class of about a little over 200 people in high school and I think there was like five or six people from high school that went to pit for my class. So definitely had some really good friends who went and kept in touch with, through undergrad. Scott Cunningham:Mm. Yeah. So it wasn't, were you sort of an early generation or you weren't, were you a first generation college student in your family or did your parents go to college Kyle Kretschman:Combination? So my dad went to Penn state civil engineer, as I mentioned, me and my mom actually graduated from undergrad the same week. So my mom went back to school later in life after me, after we went to school. And so yeah, we, we were able to celebrate graduation cuz she went to a small private school right outside of the city also. Scott Cunningham:Oh, okay. Okay. Yeah. Well, so what did you like to do in high school? Kyle Kretschman:So I played a lot of sports before high school and then I kind of switched into, and this was a traditional sports of football, basketball, baseball, but then I switched into tennis in high school. And so that kept me busy, but along with a lot of academics and really, really liked computer science. So played a lot of video games growing up, really enjoyed like that aspect in combination. Scott Cunningham:What games were your, were you, did you play on a, on a video game, plat platform? Like an Nintendo or did you play? Kyle Kretschman:Yeah, no, we played a lot of plays very much into like role playing games. Some of the arcade games like Marvel versus Capcom. So yeah. Yeah. Very, very interested in gaming. Yeah. Maybe I was a little too early for that. Cause you know, every, everybody in the 1990s was like, oh, I could make pu money playing video games, which wasn't true back, which wasn't true back then, but that's right. You know, nowadays Scott Cunningham:You can that's right. Yeah. You know, that's right. You can do it. There's all kinds of ways you can make money doing things today that nobody knew was possible 10, 10 or 15 years ago. Even Kyle Kretschman:My Scott Cunningham:That's cool. Yeah. I, I, it's funny, you know, computer games can keep a, keep a kid in high school going, you know, like especially I think they're kind of misunderstood. I, I had a lot of friends that, well, I mean, I, I, I had, when I didn't have a lot of, we moved from a small town in Mississippi to Memphis and I, those, those that first year when I didn't have a friends, I did bulletin boards and played Sierra online games like Kings quest. And it's like, it's like, you know, not intertemporal smoothing, but like inner temporal socializing, smoothing, you know, so that you just kind of get through some periods that would otherwise be a little lonelier. Kyle Kretschman:Yeah, for sure. And I mean, I mean for this audience, like most video games are some sort of form of constrained optimization. So there was, there was the inkling that I, I liked understanding how economies worked in high school through this and yeah. Going back to my mom, my mom always said like she encouraged it and she encouraged education. And there was actually kind of like that nexus, whenever I took economics in high school, it was like, oh, you know, some of these games really are full economies that are constrained and constrained in a way that you can understand and complete in, you know, under a hundred hours. Right. But there was that combination that was kind of showing itself of computer science, computer gains and economics of putting itself together. Scott Cunningham:So you were kind of thinking even in high school about economics in that kind of like, you know, optimizing something and like, like almost that modern theory that we get in graduate school. Kyle Kretschman:I think more, I had the intuition when I didn't have know how to say what it was in high school because my high school was pretty forward and that it offered both advanced computer science courses that could get you through definitely through first year of undergrad, maybe even through second year with advanced placement. And then they also offered advanced placement economics. And so I, I ended up taking advanced place in economics my junior year when most people took senior year. And so whenever I was going small Scott Cunningham:Town, even in that small town, they had, you had good your high school. Good econ. Kyle Kretschman:Yeah. It was a real, it was a really good high school that would put together good curriculum that did a lot of college preparatory work though. They, wow. They really leaned into the advanced placement, the AP courses to get students ready to go to school. Scott Cunningham:Wow. Wow. So even at, as a junior, you're taking AP econ, you know, you don't have to take AP econ. That kind of is say that, that sounds like somebody that was kind of interested in it. Kyle Kretschman:Yeah, very much. Yeah. And again, as soon as I, I definitely didn't get to the graduate level of understanding, like, you know, LaGrange multipliers, but the, the micro and macro sequence just made intuitive sense to me. It was like, it was kind of where I was like, yeah, this fit. And this is how I think. And some people might criticize me now that I think too much like an economist. Right. Like, but at the same time, it just like, it started to put together that language and even more so some of the frameworks that really kind of drew me into it. Scott Cunningham:Well, did you, did you, did you notice that you had this interest in computer science and this interest in economics and that they might be one, did you get a feeling that they could be in conversation with each other? Kyle Kretschman:Not Scott Cunningham:At first, our ancestors a hundred years ago. Didn't, you know, those economists didn't think that way, but now it's just so natural for this generation of economists to be almost one half, you know, one third mathematician, one third economist, one third computer scientist. Kyle Kretschman:Yeah. So not at first, but I, I feel like I made have like lucked into it, honestly, because whenever I chose to go to Pitt, I chose to start as computer science because I knew what that pass was. I was inspired by my older brother, the great teacher in high school. And like, I was definitely like, okay, a software software development engineer career is great. It's cutting edge. It's there. But after probably like the first year, it just didn't feel that end state didn't feel right. And so I made kind of the hard decision to choose, honestly, to switch into economics as a major, because I wasn't sure what the end state would be, where I was going with it. Cuz it was definitely felt more amorphous, you know, it's a social science, so yeah. It didn't feel like it was gonna be as clear cut and as, and have as much certainty. But pretty quickly, like after a year was like, oh, well we're doing, we're using E views at the time. All right, this is coding. I know how to do this. This is great. Right. And starting, starting to see some of that in undergrad was like the, kind of the aha moment that like, yeah, this is, this is a place where I can apply this love of coding and problem solving, but problems and solutions that I find really, really hard and interesting. Scott Cunningham:It was because of econometrics though. It was in that. Kyle Kretschman:Exactly. Yeah, yeah. Scott Cunningham:Yeah. Wow. That's, that's really interesting because you know, I think it's still the case that, you know, you can easily end up with an econometrics class that remains purely theoretical and doesn't end up, you know, exposing the student with a lot of actual coding, but it sounds like your professors were, were getting you into working with data. Kyle Kretschman:That's correct. Yeah. Both. Both within the class. So like I said, we used E views at the time. Yeah. And again, kind of like learning as a go, I, I don't think I really knew what I was doing whenever we were typing commands and E views, but the computer scientist in me was like, okay, well this is a function. I know functions. Didn't put outputs, but definitely didn't understand necessarily things that were going under the hoods or you know, all of the theory that goes with it. Oh, right, right, right. So it was, you Scott Cunningham:Knew the coding part, you knew you were coding, but you did, but like the, the actual statistical modeling was kind of the new part, but that was a way for you to kind of engage it a little bit. Kyle Kretschman:Yep, exactly. Scott Cunningham:Oh, that's interesting. That's interesting. Well, so what were you gonna have to choose between a computer science and an econ major did or did you end up doing both? Kyle Kretschman:So I chose an econ major, but then I had what I would call basically minors or concentrations in computer science, but then also in statistics and also in math, because once, once I had an internship at a bank and was doing data entry and I was like, eh, I don't think this is what I wanna use my economics degree for. Yeah. I had a couple professors at pit named Steve Houston and Frank Giani who brought me on as a research assistant, an undergrad to start being part of some of like their survey projects and data collection. And even, even one of 'em I don't, Steve was crazy, but he even let me TA classes on undergrad, so oh, wow. But he kinda, I mean, I, I say that jokingly because it was formative for me, it was like, okay, this is great. How do I do more of this? And he was like, well, you go get your econ PhD. And I was like, so I can be a teacher with computer science and doing economics altogether. He goes, yeah, let's do that. And so it was with the help and support of some of these really good professors and education to kind push me on this path consider to get Ancon PhD. Scott Cunningham:Mm. And that's when you were like, so how, how, what, what year would you have been in your program? Kyle Kretschman:Probably. I think I was in my junior year where I was starting to explore this. And then in my senior year is where I was like, okay, I'm actually gonna be doing more more of this and applying to grad school because going back, as I said, I entered with some credits. So my senior year was very, I didn't need a full course load. So I was looking for other things to keep me busy, which maybe, maybe that's one of the themes of this conversation is I kinda kind of like the variety and really have variety seeking behavior too. Yeah, Scott Cunningham:Yeah, yeah. Yeah. So you graduate, was there like a field that you were mostly interested in? Kyle Kretschman:I thought I would be going into macro economics. Macro. Yep. Yeah, because Steve worked on the council of economic advisors and I was really inspired by that and the application of economics within, within policy and just again, always applied economics, not necessarily theoretical. So yeah. Then again was, that would be sort of like labor and macro was like the initial idea, but finally Scott, I didn't do all my homework and like, think about like what grad school looked like or all it looked like. I kind of went a little bit more naive than I think other people with, again, ideas of how I could become like a teacher, an educator with some of these tools versus like how disciplined and single thread you need to be on research to be within an econ PhD program and to see that. Scott Cunningham:So you, so you kind of were like, so when you were thinking about graduate schools, what, how, what, what did you sort of, can you walk me through like what you were thinking and how you went about trying to apply to graduate school and where you ultimately chose? Kyle Kretschman:Yeah, sure. So applied probably the, the top 10 and the top 10 probably said no thanks. But also then was targeting specific schools that we had relationships with that I knew would provide computer science and macros. So university at the Iowa at the time, this was 2000 and had a really strong macro program. And then also at the university of Texas with Dean Corbe there, they also had one in Russ Cooper. And so those were like the two that I was like targeting at outside of what the top schools were. But yeah, as I, I kind of mentioned, I, I might not have prepared myself well enough to be attractive for some of the most pop with top tier schools because kind of, you know, as I said, bounced around and would be yeah, a little bit working on it a little bit different things and have computer science versus being solely focused on like economics and math and things that might be more of what the top tier schools were looking for. Scott Cunningham:Yeah. Yeah. You know, you know, it's like the, I mean, I'm the same way. I didn't ha have any econ classes in college. I was a English major, but the, the, the diff there's so many students that sort of seem to almost for whatever reason, know a lot sooner what they want to do and then like make those choices. And then there's just many of us that are, you know, in a process of search yeah. That when you're in a process of search, well, you, you know, by definition, that's like you're using that time to search. Kyle Kretschman:That's exactly right. As Scott Cunningham:Opposed to saying, I've gotta take, I've gotta become a triple major computer science, math, econ, and have to do like, you know, these set of these set of steps that, you know, there's no way I could even have known to do it unless somebody had told me it's weird. I mean, it's just funny how the little things can have such big repercussions for your whole life, but it's, but it, it worked out great. So you end up, where do you end up going? Kyle Kretschman:I went to the university of Texas at Austin. Scott Cunningham:Yeah. Yeah. What year was that? And Kyle Kretschman:So, so this would've been 2002. Scott Cunningham:Oh, okay. So you go to oh 6 0 7. Kyle Kretschman:Okay. And so ended up working. So I ended up working a lot with Jason, Ava. Yeah. And who came in and became the, the head of the department. Yeah. Applied econometrician who just did an amazing job going back to whenever I said, I didn't know how things worked under the hood, in those formulas. He didn't even let us use those formulas. So anytime we were doing applied econometric econometrics with them, not only we learning to teach, we're learning the theory, but he said, you have to code it yourself. You have to do the matrix algebra, you have to calculate standard errors. You can't really call those functions. So that was probably again, that wasn't until the third year, but yeah, in the first year to go back a little bit, Scott Cunningham:I, that played to your strengths though. I bet that played to your strengths. Yeah. Just at the end of the day, wanting to be someone that, that wrote down the raw code. Kyle Kretschman:That's exactly right. And, but the first year I didn't play my strength. Yeah. Yeah. So the first year I felt, I felt a little bit outta water and I was like, this is, I remember when we were proving what local non association. And I was like, this is, this is one hard, but also like, again, going back to like, that is this actually how I wanna be spending my time and right. I, I was like, yes, I do. But I was like, I, I knew that I needed to get to those applied applications. Yeah. And so that's, again, why I was thankful to be able to work with Jason and Steve Trayo and a few other, they applied econometricians at Texas that really encouraged me to explore starting in the second year. They didn't us like pin it down. And so I, I thought I, at the second year I worked like wrote the first, a paper on school choice and trying to see if I could find some sort of instrument on school selection on public versus private. And again, so that led to like that idea of like applied econometrics was really, really the thing that like, I was like, okay, now this fits again. Once we got into second and third year Scott Cunningham:Was, was picking up that intuition, that kind of like labor style identification, causal inference kind of approach. Was that something you picked up from Jason or was that just like from your labor people? Oh, okay. Kyle Kretschman:Yeah. That's yeah. From Jason and Steve a lot. They did a great job of doing that. And yeah. So then, yeah. Then I, then I threw in, I knew threw a little bit of a switch in there also, and my co-author Nick master and Arti and closest friend and classmate in Texas was very theoretical and very interested in applied empirical IO. And so we started working in that field also together. And so then I got to work with the Han me vet and Ken Hendrix on using empirical IO. So, oh, wow. Yeah. And so again, Scott Cunningham:This is the more structural, more structural econometric. So you've got this like reduced, you've kind of got this like traditional labor reduced form type of, part of your brain. And then you've got this empirical IO structural part of your brain kind of emerging at the same time. Kyle Kretschman:That's right. That's exactly right. Yeah. And then we threw, we threw everybody for a loop. I also saying we wanted to study study politics and how money turns into vote using both using all these tools. So yeah, I can see here kind of saying in hindsight, like it all makes sense in this story that I'm telling you, but at the time it was more of what you were talking about. It was searching. It was, I wanna be working on really interesting applied problems. I love the toolkit that economics provides in framing. And yeah. I have to be coding to be able to utilize these tools that I've had built up in the past. Scott Cunningham:Yeah, yeah. Yeah. So, so matching with Nick was really important Kyle Kretschman:Very much. Scott Cunningham:And why, if you hadn't to match with Nick, I mean, just kind of outta curiosity, if you could articulate the value added of that whole partnership, what was it? Kyle Kretschman:Yes. Sure. So, so we matched basically from math camp going into, going into the first year because Nick came both from the pure math and physics background and also had some experience in the air force. So the air force was sending him to Texas and he, we were, we were definitely, we definitely didn't have a lot of vend overlap on the fact. He's like, well, I would have the intuition and some of the computer skills, Nick would have the theoretical math skills, Scott Cunningham:The theoretical math skills. Yep. Kyle Kretschman:And then we just had, we had the common factor that we wanted to work hard together and learn together and we're willing to, we're willing to intellectually hash out really tough things together. Yeah. So yeah, he huge credit to him through being able to put up with me. And he says, he says the same thing once in a while. But again, matching with somebody that had the, the more real analysis proof based understanding of math was so valuable for me. And especially, Scott Cunningham:I think some empirical IO, especially empirical IO, just being able to, you know, think like an economist in the area of IO is thinking real deep about, you know, a rich set of models and modeling approaches. Kyle Kretschman:That's Scott Cunningham:Exactly right. That's definitely not what you're learning in your econometrics classes, even though they might go together. Kyle Kretschman:Yep. So, so yeah, it was just a, it was a really good match from the beginning. And so we complimented each other and we're, we're able to build a strong enough relationship to be able to be able to hash out, have really long nights yelling at each other, we say in the office, but it never, it was always for educational purposes and lifting each other up. Scott Cunningham:Was that different than what you thought grad school was gonna be like? Kyle Kretschman:Yeah. So I knew the research component a little bit. I just didn't under understand the unstructured research on how that was gonna go and like the cadence and where it was gonna and how that was gonna be so required to develop your own viewpoint. Yeah. I thought it would be more directed cuz as a 22 year old, that was the experience I had generally. So that was the big one was the undirected and I liked it, but it was also very difficult. Scott Cunningham:How would you describe what you're talking about to your college self? Who kind of like, you know, he, he doesn't really, he doesn't even have the vocabulary for what you're describing. What would you say? It was like, Kyle Kretschman:I think you use a good term. You have to be not only wanting to search, you have to be willing to search, but you also, then you have to put in the guardrails yourself to keep it focused because you're not necessarily gonna have those external guardrails that you will have from an alternative path of going to either like a master's program that's gonna be more structured or going in an industry or going to get a job. Right. Like I mentioned at a bank for like a 22 year old where entry level jobs are gonna be more structured. Yeah. So yeah, I just, I, I probably knew it, but I didn't know what it meant to be and what, what it meant to experience it. Scott Cunningham:So how did Jason and, and Steve kind of, and any other faculty, how, how did they, how did they, I, so I did this interview with Susan athe and she was saying that, you know, the amazing thing that pat Maja did at Amazon was he managed to make economists productive, which kind it was kind of a weird, weird way of saying it. And so in a way it could, in a way you could imagine a department that sort of has like a, you know, this idea of like research has got to come. There's like a, there's like a, a journey that a graduate student has to come on to just to basically make a decision to be a researcher. Yeah. You know, and you could imagine that creating the conditions for that is, is involves faculty member, doing stuff that's not necessarily obvious. What, how did they, how do you think they contributed to that for you personally? Kyle Kretschman:For me personally, at the time, again, it goes back to encourage the exploration versus mandating or saying that I need to be on one path. So like even Nick and I at the time explore the idea of a private company and how, what, what that would be into like pinching, pitching a venture capitalist on, on that. So all those things, again, in grad school, they, they were encouraged, but they weren't structured at the time. Yeah. So yeah, I can, I can, I understand Susan's comment because I was, I was one of those economists who started pretty early with pat and we, we have a lot of good mechanisms that we've learned and built at Amazon when I was there at the time through pat, through lay other people who were willing to make the jump into this entrepreneurial space that hit the election and the, of coalesce of economists doing open book, empirical research, along with data science. Right. Just becoming more and more valuable and applicable, but is kind of what Susan piloting that we can, we can talk more about if you Scott Cunningham:Want. Yeah. I do wanna talk about that. I wanna talk about the, the decision though, you know, to, to be, because you, you sort of started off in college, you know, you said things like, oh, you can become an educator and then you've gone in this non-academic direction and you know, it, it, and that's like a, that's a more common story now, you know, right. Of, of top talent, very talented PhDs that you could have easily seen 20 years ago, would've been an academia. Their counterfactuals are, are following you. And so, you know, it's, it's a, it's a big part of our, you know, collective story as economists that this, this new labor market that didn't, that didn't exist historically now exists and draws in so much talent. And I was just curious in a way you're kind of like a, a first generation person like that, you know, when you think about it, right. Cause text's not very old, right. Facebook, Facebook, what it's like 2007. And so, you know, so you've got this, you, you, you've got this, this chance to kind of say like, it must have been, so I don't wanna put words in your mouth, but I guess I was just wondering, what were the feelings like as you considered not taking an academic track and when did it start to be something in your mind that you thought that's gonna be something I'm explore Kyle Kretschman:Probably pretty early, because if you wanna really trace the roots of like tech economists back, it starts obviously with Hal varying at Google and me and Nick, actually, we, we sent an email to Hal, probably 2008 saying, do you have any, have any use for some summer interns who can do some empirical IO? And he said, no, not, not at this time, but so, but he Scott Cunningham:Answered the email. Kyle Kretschman:He did answer the email. Yeah. It was nice, nice of him to answer. Cause we knew he was probably pretty busy, but so it, honestly, when Amazon started hiring economists, I was probably searching for about a year to move into tech. If you wanna move back to the decision point coming outta grad school, honestly it was a challenging labor or a challenging job market for me, somebody who is a lover variety, who is working on empirical IO problems with campaign, policy, campaign, finance reform, policy recognition. That's, that's not fitting a lot of the standard application process. Yeah. Once again, that's so that's probably a theme for me. And again, at the time it was hard. I was, I was in the running for jobs at VA wakes force that I thought would be really good fit because they're the EDU the emphasis would be on education with the research ability to do research and work on problems that were more widely probably policy oriented. Yeah. But neither neither of them came through. So I just always knew that I industry was gonna be an option. And so Scott Cunningham:What year is this? What, Kyle Kretschman:What, what this would've been in this would've been in Scott Cunningham:20 11, 20 11. Okay. Oh, so you moved through the, you moved through the program or kind of relatively quickly. Oh 7, 4, 4, 5 years. Okay. Kyle Kretschman:Five years. Yeah. Five years. Yeah. Oh six to 11. Okay. But so for about a year, about six. Yeah. Yeah. And so starting in 2013 is whenever I started applying to the first tech job as a data scientist and got it went great until I talked to the VP who was a business part, like pure business person. When I was talking to the hiring manager at the time, it was a company who was providing college counseling as a software service. And so they would do this at their, their clients were both for profit and not for profit companies. And we were talking like, we'd get into details about treatment effects models and how we could measure the impact of their intervention. It went great. But then I had the flyout scheduled, but then the interview with the VP, he said, well, how am I gonna monetize your algorithm? Right. And I was like, I'm not sure I know what algorithm means, but right. I, I wasn't prepared for that language and that application and how you turn econometric modeling and measurement into, into business impact at the time. Yes. Right. So spent another year looking around with different opportunities like that and honestly learning again. So, so whenever Amazon, so this would've been in 2014 and then Amazon was hiring its first big cohort with pat. So this was a cohort that was about, I think there was about 13 of us. It was a no brainer. Kyle Kretschman:Whenever, whenever we did the interview, it just was like, all right, this is exactly right for me. I was hop. I was hoping it was right on the other side. And I could probably tell you some funny stories about the interview process, but I was like, this is, this is what's meant to be. Yeah. So it, it, it was like a 10 year journey from 2004 when I switched outta computer science into 2014 being like this, just this fit. Scott Cunningham:Right. Right. Right. So outta curiosity, you know, is, is there, is there something that you think is supposed to be learned by the fact that when you were on the job market and you had that interview with that, that gig and the, and you get to the VP and he articulates questions that are not traditional econ questions, or even econometrics questions like business profitability to act, it's kind of ironic, isn't it like to everybody? That's not an economist. That's actually what we, they think we do, you know, is like, they think we do all that stuff. And then they don't know that we're like, like you said, you know, trying to set up a Lara and solve, solve it, like what's a Lara, but do you think your competition at that time did know how to answer questions like that? Like non-economists in those positions Kyle Kretschman:Probably at an inflection point. Yeah. Because this is the same time. Wherever machine learning is becoming more common toolkit with an industry. So there would be like machine learning algorithms that are designed for, you know, prediction, problem sequencing, anything like that that are specifically designed to be used in a business setting to monitor. Scott Cunningham:So they, they not only know machine learning, it's like, they also can kind of immediately articulate why this would be profitable. Kyle Kretschman:I think so. Yeah, because again, the computer, so it's like in learning the language and this is the language that would probably be more understood within a machine learning computer science version is okay, well, I'm gonna use this to change the recommendation engine right. Is very common one. Yeah. That's obviously gonna be, so how are you gonna monetize it? I'm gonna improve the match and the recommendation engine it's gonna have this. So I think at the time there was a little bit of it, but, you know, hopefully I think, I think I learned pretty quick that you can, you can use econometrics in a similar vein. As I said, it's a flavor of data science, Scott Cunningham:Have you had to become a blue collar machine learner? Kyle Kretschman:I've had to understand it, but not, I think you mean by blue collar, you mean like implementing it Scott Cunningham:And yeah, I just, when I, I usually say blue collar in the sense of like, you know, you, don't like, you know, you basically are picking up these skills, but you weren't like, you know, you didn't get a PhD in computer science. You know, Kyle Kretschman:The answer was then that answer is definitely yes. So like as we, as our cohort and as we grew, the economics discipline at Amazon, that was a big part of it is how one could we bring in some machine learning scientist help educate and teach us. Mm. And yeah. So, and even in, sometimes in lecture style, we would do that because it was so important, but then even more so learning to so that you can interact with different stakeholders specifically, like machine learning scientists. Mm. Then understanding when you can actually implement it and marry it within the econometric models was definitely a huge part of the education process. Scott Cunningham:So you go to Amazon, is that right? That's like your first entry into tech Kyle Kretschman:That's Scott Cunningham:Right. Is Amazon, what's your title? Kyle Kretschman:So Scott Scott Cunningham:A scientist or economist. Kyle Kretschman:I, it was something like business intelligence engineer. There wasn't an economist job family. There was, as you said, it was kinda the forefront. I think it was this. Yeah. I think that's what it was, but Scott Cunningham:Cause it is now right. Baja has a that's Kyle Kretschman:Right. Scott Cunningham:He created a job title called economist. Kyle Kretschman:That's right. Yeah. And that got set up about a year in, so like, and I was part of the group. So we would set these, we would set up like these people and process mechanisms that allow economists to be so influential and productive within Amazon. Scott Cunningham:Mm, okay. So how is he doing it? Why, why is Susan saying he performed a miracle by making economist productive? Can you kind of describe, like, if you had to just guess at like the counterfactual, if it hadn't been, you know, pat, it hadn't even been an economist that was hired into Pat's position. Like, what is it that he, what, what is it that he, or Amazon or whatever is making you go transform and become this new version of yourself? Kyle Kretschman:There's, there's a lot of factors and I could probably spend an hour on this, but I'll, I'll try to, I'll try to reduce it down to like some key mechanisms and ideas. The first is that Amazon is probably the most data driven company. I know. Mm. They are so focused on measurement, both of things you can directly measure. And, but they are. So they were very early interested in economic measurements that are UN observables either coming from like coming from econometric models. That, that was whenever pat demonstrated some of those that was like the light bulb went off the, so, because again, it, Amazon was run by and still generally is people with operation science background. And so this over index on measuring as, as coly and as precisely as possible, well that's that's economics. So that, that was part of it. Another part of it is culturally Amazon operates that makes decisions based on six page white papers, you wanna make some economists really productive, have them write a six page white paper instead of giving them a presentation, especially to people like who may be in the background with MBAs or other people who have a comparative advantage, we economists have a care advantage in writing. Kyle Kretschman:So it was little bit of like a surprise, but you might hear these anecdotes where it's true. Like whenever you go into a, a decision making meeting, you come in with your six page white paper that says here's the business decision to be made here is my recommendation. And here's why, and people sit there and it can be a room for five people can be a room of 25 executives. They sit and read the paper and they read the whole thing. Is there an append that can go on forever depending on how big the meeting is. Sure. But that structure of, of data driven decision making, combined with how you're presenting your argument is written seems like, seems like economists should be pretty good at that. Right? Scott Cunningham:Is that a pat thing? He came up with work, the work he made, Kyle Kretschman:What was the six page idea was from Jeff Bezos. And so that was, would Scott Cunningham:Those be circulated throughout the, throughout the, the, the firm, Kyle Kretschman:The stakeholders who needed to be part of the decision making they be circulated. But again, this is every, like everybody's writing six pages. PowerPoint is basically outlawed at, at Amazon. And again, that happened mid 2000. Sometimes people can Google it to find out, but that six page culture and decision making culture, just again, fit economists. Scott Cunningham:So how is a six page paper similar to the kinds of writing that, you know, you sort of associate with economists and how is it different? Kyle Kretschman:So its I'll start with the differences. So one with the six page versus like a 30 page academic, you are not going to be able to share the research process. You are not supposed to share the research process. You're supposed to share the clear recommendation and how you got to that recommendation. Right? So if you think about like a 30 page academic paper XT, be condensed down into those six pages. In my view, they're just, that's just not how the industry operates, but you probably would know better than me on that where, but so again, where it's the same is again, it's a data driven argument. The purpose of this paper, the abstract here is the hypothesis that I have that and here's how I tested it. And here's how I'm making my conclusion. So what I always found really honestly easy was I felt like I was doing the scientific process. Like I felt I, I was with business decision making it generally work within what is the hypothesis? How are we doing this? How are we testing it? What are we think some alternative conclusions could be, but what are we making towards it? So yeah, yeah. Again, it was closer to what I felt like would be a scientific paper in and that hold of day driven mindset is again, that's more, it's very common. Amazon have a common Spotify now Scott Cunningham:Has that been influential throughout, throughout industry? Has that, how have you noticed Amazon influencing Kyle Kretschman:Some Scott Cunningham:Yeah. Like most people don't understand. Kyle Kretschman:Yeah. There there's some companies who definitely have completely adopted it. There's some companies who haven't, but the, the six pager again, that's, this is not a, this isn't a concept just to economist and tech. This is the concept is, is held up as one of the key mechanisms for all of Amazon. Scott Cunningham:Mm mm Hmm. Kyle Kretschman:One other. Scott Cunningham:How often were you writing those? Kyle Kretschman:Depends on what level you were farther in my career. That's the only thing I did was write six page papers and it would be part of like, my team would help, but again, anytime you have a key business decision to be made or an update, like you're gonna be writing the six page. So yeah, it's again, the farther, the more seniority you have though, the more that becomes your job is to communicate side and guide through these business decisions. Scott Cunningham:Do they, to you, Kyle Kretschman:They belong to the team because it's always Scott Cunningham:Put 'em on a, you can't they're like proprietary though to Amazon. Kyle Kretschman:Oh, correct. Yeah. No, they, they're not publicly available. They're Scott Cunningham:Proprietary. Like it must is it what's that feel like to do something? What's it, what's it feel like to, to do something that creative in that kind of like scientific that's siloed within the firm? Does that feel strange? Kyle Kretschman:No, it didn't. Because what it enables is to be able to work on some of the hardest questions without having to worry about without having to worry about com communication strategies or right. For press release. So no, it felt like we were able, and this is going back to like some of the things that pat and we did at Amazon make successful. We worked on some of the hardest problems at Amazon from a very early stage because we said that it wouldn't be publicly available. Right. So that's gonna do that. And Scott Cunningham:That's been a key part. Yeah. Because okay. I get it. Okay. That, that makes a lot of sense. Yeah. So who did you discover? You were, go ahead. Sorry, Kyle. Kyle Kretschman:No, I was gonna say maybe the last me to highlight. Cause again, I, I, we could probably spend this whole interview on this, but the, the other key mechanism that pat pioneered was the proliferation of economists as a job family was not pat saying and us saying, go do this. And I can give through my own personal example. It was the other business executives, seeing the measurement, seeing the results on product, just saying, okay, I want that. So it really was a demand, AKA demand, internal demand for more economists, that was gonna say, I want this with my business decision making process and want these people who can do this and collaborate across the difference. It was not a, oh, we're gonna put economist in the siloed function that everybody's gonna come here. And that was, that was my story. But the very first year I worked on projects directly for the consumer CFO, basically the whole year. It wasn't necessarily by design, but it was what happened. And at the end of the year, year and a half, the, the VP of finance said, come over here and do this with me and come build, come build an economics team and an economics function here within my organization. And that's really is again, that's the real key was it was business decision makers, demanding the ability to understand this and demanding the skill set, just like they would data science, machine learning because of demonstrated value. Scott Cunningham:What were they witnessing with their own eyes that was so compelling that they would Inc that it would increase demand. Kyle Kretschman:So both I'll call it like ad hoc economic analysis on maybe big strategy projects, but also then the introduction of econometric systems into product. Scott Cunningham:Mm. What does that mean? Introduction of econometric systems into products. Kyle Kretschman:So say you have a product that is gonna, let's go back to the recommended system. And I use that again as an abstract, but within there you might make a change to it and you might make a change with the recommender system. That's gonna cause a treatment effect. Right. So, okay. So we can do that one off to estimate that, but you could also then build an economic system. That's gonna measure those treatment effects and changes like an AB platform or things like that. So maybe people might be more common and familiar with like experimental platforms. This would also be then econom. This would be sub out the AB part of it and sub in an economic model, that's going to be doing always on measurement sometimes at a, you know, service level. So sometimes within like individual pages, sometimes it's gonna be at a monthly level, but the integration of econometric models into the product. Scott Cunningham:Right, right. Wow. So how are you a different economist because of that experience at Amazon, if you had to guess, what was it the treatment effect? Kyle Kretschman:Oh, it mean it was, it was incredibly formative because it to tie like it put the fit together with the application to where I could understand and really to where it is, my job is to take a business question, turn it into a scientific process that can be solved with econometrics. And then also be thinking about, is this a problem that needs a scalable solution? Right. So, so Amazon taught me business integration taught me so many different languages, taught me leadership and management taught me how to work with stakeholders in collaborative ways, but then even more so how to deliver the value through econometric measurement, both again, as I said, not only, not only just in ad hoc research papers or one off analysis, but also then where does this fit directly within the products that we build in tech? Scott Cunningham:Yeah. So where'd you go, seems like people don't stay very long in tech. That's like normal. Whereas like, is, is that right? People kind of like, it, it's less normal to stay your whole career at Amazon unless is that wrong or, Kyle Kretschman:I mean, it's got it still do. So it's probably tough to say that because really the, the field started, like you said, really proliferated in 2012. So I stayed at Amazon for six years and I thought I'd be staying even longer. But Spotify came with the opportunity to one work on something I care very deeply about, which is the music industry. I'm a huge music fan. They also came with the idea to build again. So, you know, that was the part that really enticed me was Spotify did not have any PhD economists who were in an and, and economist roles. They had like one in a data science role, but they didn't have the structured economic discipline that they were seeing that Amazon was proliferating. And also then going into like Uber, Airbnb and the other tech companies. And so they said, can you build again? Kyle Kretschman:And I said, yeah, I'm, I'm excited to build. And then last one, all these there's definitely personal considerations here too. And Spotify just really did a great job showing how the company as a whole has Swedish cultures and values. And at the time I had a nine month old and they said, this is a great place to come be a father with the balance and that, and I said, all right, let's make the jump and come to Spotify. And so now I've been here about two years. So cuz I, I actually went to Spotify in may of 2020. Scott Cunningham:So remind me again, your job title at Spotify. Kyle Kretschman:So I'm head of economics. Scott Cunningham:Is, is that the, is that, is that like chief economist? I, I feel like I see different, different job titles and I don't know exactly what, what everything, Kyle Kretschman:Yeah. It, it it's on the path to it. So I'm, I'm the highest ranking PhD economist at Spotify. Scott Cunningham:I see. Okay. I've been there for two years. Okay, go ahead. Sorry. Kyle Kretschman:Yeah. Cause again, that's what I was brought into build was to build, like we did at Amazon was overall integration of PhD economists within the different business units. Scott Cunningham:So this is the part I'm, I'm having some hard time, like, you know, putting, visualizing or putting in my own words. What exactly will it look like if you have been successful in five years at that goal and what would it look like if you had been a complete, complete bust? What are the two things that are like empirical that I would be able to, to observe? Kyle Kretschman:Yeah. A complete bust is probably that an economics discipline is not, is not part of Spotify and there's not, there's not a job family. So a complete bus would've been, I, I moved to Spotify, an economics discipline. I either in, or I'm working data science job, what success looks like is actually what we put first from a, so I'll talk about the people in process, discipline success. We, I came into was Scott Cunningham:Real quick. So Kyle Kretschman:Foundation on basically. Yeah. Scott Cunningham:So, so failure actually would mean that the economist community within Spotify just never materialized, is that what you're saying? And that, and that means like this, having groups of economists that, that think and use the kinds of training we had in graduate school, but in a way that is actually productive in the firm is, is that, is that right? Kyle Kretschman:So, so yeah, and again, that's, Scott Cunningham:The job is successful if you're able to actually create internal demand for economists. Kyle Kretschman:Yep. That's right. And that's, that's what I would say against from the process side. And then from the product side, that's using econometric research in the ways that I've been talking about it's using it both not only for individual analysis, but also then building econometric measurement systems that improve the product to get towards Spotify's mission of, of billion listeners and fans who can connect with over a million creative artists who are making a living. So that's, so it's a combination, it's the combination people process. Do we have the people set up? Do we have this integrated system of economists working alongside all these different types of stakeholders along with the product side of, do we have these measurement techniques that we're applying in a way that is important to Spotify's not only Spotify's business, but all the stakeholders that have an interest in Bon life. Scott Cunningham:So I feel like, you know, I think to academics that, that, and, and maybe even to some degree students, maybe I'm, maybe I'm completely an outlier here and I'm wrong, but you know, I think there's this like really shallow is a negative word. It, I mean, shallow, literally more and just like, it's just the thinnest knowledge possible of what exactly, you know, the, the, the core skillset of a successful economist is in tech. You know, and for many people they think, I think they, they think it's such a primitive level. They're like, it needs to be somebody that can code, you know, it's a data scientist, but, but it, but it, but that's not what I associate with economics. Right. So what would you, what would you articulate? It is, Kyle Kretschman:So it's the ability to do econom applied econometric research. That's applied to business problems. Mm. So within that is coding. Yes. Scott Cunningham:Right, right. Within that is coding. Kyle Kretschman:I, the vast majority, I won't say everyone, but the vast majority of tech economists are gonna have some level of coding and maybe they're not coding anymore. Like I'm not doing any coding anymore, but like they, they have that ability. So that's just again, that's, that's a skillset, but the real ability is doing long-term economic research. Because the questions that we get asked are very hard and difficult, and they are maybe in the academic setting, maybe they are publication worthy, takes that take three years, four years to actually solve with the right model. Yeah. But it's the ability to take that three year research roadmap and make it progress. So when you're doing that, you need to have your summary statistics that the business can see, understand, give feedback on because that accelerates the research process and also accelerate the business impact. So, one, I guess one comparative skillset that I've learned is what I call research with an open book. You shouldn't really go a month without talking to your stakeholders. You should be showing where the research is. You need to be the person who owns that three year measurement roadmap, but you're the person who's gonna be having to take the feedback at a consistent basis. And that's, that's really the different part. So, but again, that goes back to the applied econometrics part Scott Cunningham:Of it. So what is the, so walk me through your, walk me through typical days in your job now. Kyle Kretschman:So my typical day is much more generally now on the management people side. So it's definitely going to be, as I said, building this discipline because we have the creator economics team who's focused on the supply side. We spun up the ads economics team that has a completely separate unit again. So that's like part of the people process, but then it's creator economics team has a hybrid data science and economics team. That's doing all these things that I'm talking about. So with that right now, you know, we're working on long term roadmaps for both product and research and being there. And then also giving I say, honestly, translating a lot of the work that the team is doing into the actual business decision making. So, so I kind of work if you think of the research process. And I think this is kind of generally true, most managers or at least tech economist managers, they're working at that hypothesis generation and then communication of results where then the team itself is working with the data analysis, the statistical models along that research loop to integrate. So it's more of an, my, my role really is an integration of being a translator of economic ability and measurement into the business. Scott Cunningham:So, you know, so you've clearly have a comparative advantage evidence by, you know, your, your successful Mo creating of a career. Right. And, but I was curious, what do you think your, what do you think that is? You know, how would you explain that to like, you know, like on a subreddit, explain to me like I'm five, how, how would you like explain, explain to me what you see as your compared to advantage now in life, you know, that you, that you, that you have the, that your presence as this value added for the firm. And, and was that now that you look back, when you think about who you were as a young person back at UT, or even at pet, do you see signs of that and what, what were the signs Kyle Kretschman:The, yes I do, because really my comparative advantage now is as I mentioned, I seek variety and specifically I seek variety on really hard, interesting problems. Mm. So that's, that's gonna be it, you can abstract away from the economist role. You can abstract away anything else, but I like to work on things that are hard to solve, which is that's gonna be valuable in so many different contexts. And then specifically in the economist frame, again, we're, we're probably comparing to other economists, I'm much more of a variety speaker. So I wanna work on a variety of problems. And I mean, you're Scott Cunningham:People that are not like that. Kyle Kretschman:I mean, right. So there would be people who are gonna be much more focused on changing the methodology for, I mean, Scott, you work in, cause inference, you tell me, what are you, what are you working on? That's gonna be changing some sort of estimator. Right, right. That's that just doesn't hold that much appeal to me, I'll say it. What does appeal to me is this entangling a problem, making it less, making a super ambiguous problem solvable. Right. And then having people dive in and solve that. So, yeah. Yeah. I guess the Reddit theme variety speaking on ambiguous problems, Scott Cunningham:It's funny, you know, like maybe you heard at some point in your career, I'm just guessing someone, or I could imagine someone saying you're not focused. Kyle Kretschman:Yeah, I Scott Cunningham:Did. And, and in fact that might have been true and really not relevant. Right. Well, Kyle Kretschman:It was probably, it was probably relevant feedback to certain end goals. But again, once I, once I realized what the end goals were, it's all of this to tied them all together, all of this led up to where and economi and tech is exactly where yeah, I was, I was meant to be helping quote. Scott Cunningham:Right. Right. Well, you know, I think like one thing that I, I feel like is really positive about the, the growth of the, of tech. And in many ways I see now at universities are now in direct competition with tech and I, and I wouldn't, I don't think that was true 50 years ago that maybe, maybe universities were in competition with government. I don't even know if that was exactly like it is now, you know? And it, it seems like what I see with you is, is a person who, who, who took ownership of their life. Right. And just sort of said, I don't wanna put words in your mouth, but it, but it seems like one of the things that it's like, you know, I think young people can be encouraged by or should hear is that, you know, here's a person that took ownership of their life. This was their career. It's the only life they had. This was their career. And they, they, they had the fortitude and the resilience and the, you know, to, to be themselves and make a career. Is that true? You think, Kyle Kretschman:I appreciate you saying that. And it's probably true again, I kind of like hesitate because again, during, in my twenties and the searching phase, it definitely didn't feel that Scott Cunningham:Way. Didn't feel that way. Kyle Kretschman:Right. But now in the decade of my thirties, yeah, it did. Because, but I was, I was more than willing to explore different avenues. Yeah. Because that's what, because I wasn't, I wasn't willing to settle, I guess I'll say that, like, I wasn't willing to settle for something that I didn't feel was the right fit for me. Yep. And so that's, that's what a big driver that's been able to help me get here for Scott Cunningham:Sure. Yep. Yep. I think that, I think the, the truth is economics is a really valuable PhD and it supports a, it, it supports many personalities with it and it, but you know, like I do think sometimes one of the skills of that, of people that are in that stage of searching is just to be able to be, to wait. Yeah. You know, you're like, so you don't always know that you're planting seeds and, and you know, you can't harvest seeds when they're seeds, you know, you can't harvest plants when they're seeds, you have to wait. Yeah. But you have to be disciplined and continue to water it too, though. Kyle Kretschman:Yeah. And also be, be willing to take some risks once in a while Scott Cunningham:And willing to take some calculated Kyle Kretschman:Risks. But yeah. Specifically like for tech economists, now there is a lot more information. And I will say probably five years ago, I probably cautioned some people from going to get their PhD in economics if they wanted to be a tech economist, because it was still, you know, the majority of people who jumped into it were risk takers, willing to do that. There wasn't this. But now I believe the head of NABE has said that there's over 1500 economists in tech. Now we have a great conference that we spun up while we're, while, while I was at Amazon, that I was a part of that now NABE tech in November. So if people wanna learn, now, the ability is there. So in grad school, like come to NA tech, come see, come look at the program, come see exactly. Cuz it was, it's a conference that's designed by tech economists for tech Scott Cunningham:Economist. Yeah. Susan mentioned it too. A B E national association, business economics. I actually, it's a huge conference and I actually don't think in academia, a lot of people know about it. Kyle Kretschman:Yeah. That's cause Scott Cunningham:We just know about the as SSA. Kyle Kretschman:Right. And that's one of the reasons I kind of wanted to plug it here, honestly. Cause again, it, it, it started out probably 50 people. I forget what year, but now it's up to, I think they're expecting over 500. And so that is, so some of this discovery is searching that if people, if people who are in their PhD wanna learn more about, that's gonna be the best place. Scott Cunningham:I only have two more questions and then I'm gonna let you go. One of 'em is kind of touchy, feely, but okay. So, so, and I, and I I'll just admit, as I ask this question, you're gonna immediately know, I must think this way. And so that does not mean that you think this way, but you know, I love being an economist. I love economics. I love being an economist. And if I was to go get a job and the job title said, didn't say economist on it. I would be, I would feel, I, I can already tell that would be something I would have to work through. Cuz I feel so connected to the tribe. And I was just wondering, you know, do you think that economists and tech feel connected to the tribe? Kyle Kretschman:It, it gets to which tribe Scott Cunningham:Do you think they feel like a part of the broader American economics association? Kyle Kretschman:My I'll say my personal view is probably not because that's not the colleagues that we've grown up with over the past 10 years. Do we feel part of the tech economist community? Yes. There is one there very much is one. And do we feel part of that tribe very much. Do I feel part of the academic economist tribe? Probably not. Yeah. But to tie it all together, like do I feel like I'm an economist? Yes. Yeah. A hundred percent. Yes. And again, we're making differentiations probably for your audience, but in the grander scheme of things, if you look at, if you look at anybody in tech, who's an economist they're gonna to, to non they're, they're economists Scott Cunningham:They're economist. Right? They, they, they, they, they think different. Kyle Kretschman:Yeah. I mean, I'll give a quick anecdote, like one of my, one of my good friends and colleagues at Amazon Neil go, she's not there anymore. His wife and my wife just would always be like, they just respond to the, the question in the exact same way. Scott Cunningham:Yeah. Right. Kyle Kretschman:So that's like, they very much are just so rational with their responses in general, you know, there's exceptions, but they, they would compare notes and we'd have the same general view and outlook on personal finances. Let's say so. Yeah. But I think what you're heading at is, is their different tribes that are growing up. Yeah. Because I think there's probably always been different tribes. And so tech economists is a tribe that is, I mean is more than growing up as again, we think there's probably hook going on 2000 of them and there's very strong connections within that. Scott Cunningham:Yeah. I guess I wonder sometimes I'm like, what's the point? The point is community. The point is not for the, the goal is not for the AA to have everybody. The goal is for, you know, I think the goal is for people to thrive and feel connected to their broader, to a broader community. And if that's grown and ly in tech, that's good. I just always kind of think to myself, like, but at the end of the day, the universities are staffed by faculty in the AEA and it, it just seems like, and so it's like, it, it seems like at the graduate training, you know, many people can feel job dissatisfaction because they know they're not gonna become an academic, you know? And that, that can be a, that might be a source of some of the struggles. There's a new paper in the general economic literature about mental health struggles amongst PhD students and economics. Scott Cunningham:And there would, you could, if you read between the lines, if you read between the lines, there was a real disconnect between them and their advisor. They, the, the, you know, you don't know causality, cuz it was just a survey, but like, you know, the, the students were like, they just seemed to have job dissatisfaction. They didn't think their work mattered. You know? And, and I just kept thinking in my mind, you know, do they know about all their options? Are any of those options stigmatized? You know, because like, if you are on, if you've held it up to students that the only meaningful career that you could possibly have is at a, is tenured at a, at a university writing papers that get into top fives and you, and you sense inside your heart, you know, that just, I don't think that that's, that's not doing it. Kyle Kretschman:Yeah. I think it ties together what you said earlier that economics welcomes a whole bunch of different viewpoints of world views, but maybe doesn't reward the diversity of the way it should in academic. And so like from a diversity viewpoint, these are different career paths, but even more shows should welcome more ideas. So like that's probably where the economics field needs to continue to lean in is with the diversity to enable people, to not to know different paths, to raise up people on different paths and have those opportunities. And I think tech is providing a huge pathway for Scott Cunningham:That. Yep. Yep. I do too. I am very grateful for it. It seems really exciting. It seems like a world of just like working on I'm sure that the way we think about, you know, gossip at Guinness coming up with, you know, students tea of this story of a, of a person in industry making these major contributions, we're just at the beginning of, of that, you know, with just the sheer volume of, of social scientists in tech, working on fascinating important topics. My last question, what is your favorite paper in economics or no, let me say it this way. What is a paper in that has, that has, for some reason stuck in that has stuck around in your head Kyle Kretschman:BLP so yeah, it, it has, because that was probably the first time that I felt like the wow moment of we can, we can estimate substitution effects with observational data. Don't need experiments. We have this method that obviously has proliferated into so many different ways and that stuck with me because throughout all my stuff, I've generally been super in substitution effects. So yeah. Yeah. I would put that in that research thread. Scott Cunningham:Yeah. I, I, I don't think that that's that you having that empirical IO structural, you know, having written in your head. Kyle Kretschman:Yeah. Right, Scott Cunningham:Right. That's great. Well, it is a real pleasure to, to get to walk through. I appreciate you walking me through your, your life and tell me about your, your career. It's really nice pleasure to meet. Kyle Kretschman:Yes. Thanks Scott. I really appreciate the.