Chris Martello: Day of Darkness

In this episode, we cover:

• Introduction (00:00)
• How Chris got into the world of chaos and teaching middle school science (02:11)
• The Cengage seasonal model and preparing for the (5:56)
• How Cengage schedules the chaos and the “day of darkness” (11:10)
• Scaling and migration and “the inches we need” (15:28)
• Communicating with different teams and the customers (18:18)
• Chris’s biggest lesson from practicing chaos engineering (24:30)
• Chris and working at Cengage/Outro (27:40)

Links Referenced:

• Cengage: https://www.cengagegroup.com/
• Chris Martello on LinkedIn: https://www.linkedin.com/in/christophermartello/

Transcript

Julie: Wait, I got it. You probably don’t know this one, Chris. It’s not from you. How does the Dalai Lama order a hot dog?

Chris: He orders one with everything.

Julie: [laugh]. So far, I have not been able to stump Chris on—[laugh].

Chris: [laugh]. Then the follow-up to that one for a QA is how many engineers does it take to change a light bulb? The answer is, none; that’s a hardware problem.

Julie: Welcome to Break Things on Purpose, a podcast about reliability, quality, and ways to focus on the user experience. In this episode, we talk with Chris Martello, manager of application performance at Cengage, about the importance of Chaos Engineering in service of quality.

Julie: Welcome to Break Things on Purpose. We are joined today by Chris Martello from Cengage. Chris, do you want to tell us a little bit about yourself?

Chris: Hey, thanks for having me today, Julie, Jason. It’s nice to be here and chat with you folks about Chaos Engineering, Chaos Testing, Gremlin. As Julie mentioned I’m a performance manager at Cengage Learning Group, and we do a fair amount of performance testing, both individual platforms, and coordinated load testing. I’ve been a software manager at Cengage for about five years, total of nine altogether there at Cengage, and worn quite a few of the testing hats, as you can imagine, from automation engineer, performance engineer, and now QA manager. So, with that, yeah, my team is about—we have ten people that coordinate and test our [unintelligible 00:01:52] platforms. I’m on the higher-ed side. We have Gale Research Library, as well as soft skills with our WebAssign and ed2go offerings. So, I’m just one of a few, but my claim to fame—or at least one of my passions—is definitely chaos testing and breaking things on purpose.

Julie: I love that, Chris. And before we hear why that’s your passion, when you and I chatted last week, you mentioned how you got into the world of QA, and I think you started with a little bit of different type of chaos. You want to tell us what you did before?

Chris: Sure, even before a 20-year career, now, in software testing, I managed chaos every day. If you know anything about teaching middle school, seventh and eighth-grade science, those folks have lots of energy and combine that with their curiosity for life and, you know, their propensity to expend energy and play basketball and run track and do things, I had a good time for a number of years corralling that energy and focusing that energy into certain directions. And you know back, kind of, with the jokes, it was a way to engage with kids in the classroom was humor. And so there was a lot of science jokes and things like that. But generally speaking, that evolved into I had a passion for computers, being self-taught with programming skills, project management, and things like that. It just evolved into a different career that has been very rewarding.

And that’s what brings me to Cengage and why I come to work every day with those folks is because instead of now teaching seventh and eighth-grade science to young, impressionable minds, nowadays I teach adults how to test websites and how to test platforms and services. And the coaching is still the same; the mentoring is still the same. The aptitude of my students is a lot different, you know? We have adults, they’re people, they require things. And you know, the subject matter is also different. But the skills in the coaching and teaching is still the same.

Jason: If you were, like, anything like my seventh-grade science teacher, then another common thing that you would have with Chaos Engineering and teaching science is blowing a lot of things up.

Chris: Indeed. Playing with phosphorus and raw metal sodium was always a fun time in the chemistry class. [laugh].

Julie: Well, one of the things that I love, there are so many parallels between being a science teacher and Chaos Engineering. I mean, we talk about this all the time with following the scientific process, right? You’re creating a hypothesis; you’re testing that. And so have you seen those parallels now with what you’re doing with Chaos Engineering over there at Cengage?

Chris: Oh, absolutely. It is definitely the basis for almost any testing we do. You have to have your controlled variables, your environment, your settings, your test scripts, and things that you’re working on, setting up that experiment, the design of course, and then your uncontrolled variables, the manipulated ones that you’re looking for to give you information to tell you something new about the system that you didn’t know, after you conducted your experiment. So, working with teams, almost half of the learning occurs in just the design phase in terms of, “Hey, I think this system is supposed to do X, it’s designed in a certain way.” And if we run a test to demonstrate that, either it’s going to work or it’s not. Or it’s going to give us some new information that we didn’t know about it before we ran our experiment.

Julie: But you also have a very, like, cyclical reliabilities schedule that’s important to you, right? You have your very important peak traffic windows. And what is that? Is that around the summertime? What does that look like for you?

Chris: That’s right, Julie. So, our business model, or at least our seasonal model, runs off of typical college semesters. So, you can imagine that August and September are really big traffic months for us, as well as January and part of February. It does take a little extra planning in order to mimic that traffic. Traffic and transactions at the beginning of the semester are a lot different than they are at the middle and even at the end of the semester.

So, we see our secondary higher education platforms as courseware. We have our instructors doing course building. They’re taking a textbook, a digitized textbook, they’re building a course on it, they’re adding their activities to it, and they’re setting it up. At the same time that’s going along, the students are registering, they are signing up to use the course, they’re signing up to their course key for Cengage products, and they’re logging into the course. The middle section looks a lot like taking activities and tests and quizzes, reading the textbook, flipping pages, and maybe even making some notes off to the side.

And then at the end of the semester, when the time is up, quite literally on the course—you know, my course semester starts from this day to this day, in 15th of December. Computers being as precise as they are, when 15th of December at 11:59 p.m. rolls off the clock, ...