Sam Rossoff: Data Centers Inside Data Centers

In this episode, we cover:

• 00:00:00 - Intro
• 00:02:23 - Iwata is the best, rest in peace
• 00:06:45 - Sam sneaks some SNES emulators/Engineer prep
• 00:08:20 - AWS, incidents, and China
• 00:16:40 - Understanding the big picture and moving from project to product
• 00:19:18 - Sam’s time at Snacphat
• 00:26:40 - Sam’s work at Gremlin, and culture changes
• 00:34:15 - Pokémon Go and Outro

Transcript

Sam: It’s like anything else: You can have good people and bad people. But I wouldn’t advocate for no people.

Julie: [laugh].

Sam: You kind of need humans involved.

Julie: Welcome to the Break Things on Purpose podcast, a show about people, culture, and reliability. In this episode, we talk with Sam Rossoff, principal software engineer at Gremlin, about legendary programmers, data center disasters at AWS, going from 15 to 3000 engineers at Snapchat, and of course, Pokémon.

Julie: Welcome to Break Things on Purpose. Today, Jason Yee and I are joined by Sam Rossoff, principal software engineer at Gremlin, and max level 100. Pokémon trainer. So Sam, why don’t you tell us real quick who you are.

Sam: So, I’m Sam Rossoff. I’m an engineer here at Gremlin. I’ve been in engineering here for two years. It’s a good time. I certainly enjoyed it. And before that, I was at Snapchat for six years, and prior to that at Amazon for four years. And actually, before I was at Amazon, I was at Nokia Research Center in Palo Alto, and prior to that, I was at Activision. This was before they merged with Blizzard, all the way back in 2002. I worked in QA.

Julie: And do you have any of those Nokia phones that are holding up your desk, or computer, or anything?

Sam: I think I’ve been N95 around here somewhere. It’s, like, a phone circa 2009. Probably. I remember, it was like a really nice, expensive phone at the time and they just gave it to us. And I was like, “ oh, this is really nice.”

And then the iPhone came out. And I was like [laugh], “I don’t know why I have this.” Also, I need to find a new job. That was my primary—I remember I was sitting in a meeting—this was lunch. It wasn’t a meeting.

I was sitting at lunch with some other engineers at Nokia Research, and they were telling me the story about this app—because the App Store was brand new in those days—it was called iRich, and it was $10,000. It didn’t do anything. It was, like, a glowing—it was, like, NFTs, before NFTs—and it was just, like, a glowing thing on your phone. And you just, like, bought it to show you could waste $10,000 an app. And that was the moment where I was like, “I need to get out of this company. I need a new job.” It’s depressing at the time, I guess.

Julie: So. Sam, you’re the best.

Sam: No. False. Let me tell you story. There’s a guy, his name is Iwata, right? He’s a software developer. He works at a company called HAL Laboratories. You may recall, he built a game called Kirby. Very famous game; very popular.

HAL Laboratories gets acquired by Nintendo. And Nintendo is like, “Hey, can you”—but Iwata, by the way, is the president of HAL Laboratories. Which is like, you know, ten people, so not—and they’re like, “Hey, can you, like, send someone over? We’re having trouble with this game we’re making.” Right, the game question, at the time they called it Pokémon 2, now we call it Gold and Silver, and Iwata just goes over himself because he’s a programmer in addition to be president of HAL Laboratories.

And so he goes over there and he’s like, “How can I help?” And they’re like, “We’re over time. We’re over budget. We can’t fit all the data on the cart. We’re just, like, cutting features left and right.” He’s like, “Don’t worry. I got this.”

And he comes up with this crazy compression algorithm, so they have so much space left, they put a second game inside of the game. They add back in features that weren’t there originally. And they released on time. And they called this guy the legendary programmer. As a kid, he was my hero.

Also famous for building Super Smash Brothers, becoming the president of all of Nintendo later on in his life. And he died a couple years ago, of cancer, if I recall correctly. But he did this motion when he was president of Nintendo. So, you ever see somebody in Nintendo go like this, that’s a reference to Iwata, the legendary programmer.

Jason: And since this is a podcast, Sam is two hands up, or just search YouTube for—

Sam: Iwata.

Jason: That’s the lesson. [laugh].

Sam: [laugh]. His big console design after he became President of Nintendo was the Nintendo Wii, as you may recall, with the nunchucks and everything. Yeah. That’s Iwata. Crazy.

Julie: We were actually just playing the Nintendo Wii the other day. It is still a high-quality game.

Sam: Yeah.

Jason: The original Wii? Not like the… whatever?

Julie: Yeah. Like, the original Wii.

Jason: Since you brought up the Wii, the Wii was the first console I ever owned because I grew up with parents that made it important to do schoolwork, and their entire argument was, if you get a Nintendo, you’ll stop doing your homework and school stuff, and your grades will suffer, and just play it all the time. And so they refuse to let me get a Nintendo. Until at one point I, like, hounded them enough-I was probably, like, eight or nine years old, and I’m like, “Can I borrow a friend’s Nintendo?” And they were like, sure you can borrow it for the weekend. So, of course, I borrowed it and I played it the whole weekend because, like, limited time. And then they used that as the proof of like, “See? All you did this weekend was play Nintendo. This is why we won’t get you one.” [laugh].

Sam: So, I had the exact same problem growing up. My parents are also very strict. And firm believers in corporal punishment. And so no video games was very clear. And especially, you know, after Columbine, which was when I was in high school.

That was like a hard line they held. But I had friends. I would go to their houses, I would play at their houses. And so I didn’t have any of those consoles growing up, but I did eventually get, like, my dad’s old hand-me-down computer for, like, schoolwork and stuff, and I remember—first of all, figuring out how to program, but also figuring out how to run SNES emulators on [laugh] on those machines. And, like, a lot of my experience playing video games was waking up at 2 a.m. in the morning, getting on emulators, playing that until about, you know, five, then turning it off and pretending to go back to bed.

Julie: So see, you were just preparing to be an engineer who would get woken up at 2 a.m. with a page. I feel like you were just training yourself for incidents.

Sam: What I did learn—which has been very useful—is I learned how to fall asleep very quickly. I can fall asleep anywhere, anytime, on, like, a moment’s notice. And that’s a fantastic skill to have, let me tell you. Especially when [crosstalk 00:07:53]—

Julie: That’s a magic skill.

Sam: Yeah.

Julie: That is a magic skill. I’m so jealous of people that can just fall asleep when they want to. For me, it’s probably some Benadryl, maybe add in some melatonin. So, I’m very jealous of y...