Arrested DevOps

Trevor and Bridget chat with Courtney Nash (O'Reilly Media) and Lindsay Holmwood (Australian Government Digital Transformation Office) about cognitive neuroscience. We'll talk about recognizing cognitive fallacies, the psychology of alert design, how to make your conference proposals their most appealing, and about empathy from a scientific point of view.

Recording Live from DevOpsDays Chicago! ChatOps is used by many teams and companies as the main communication tool for day to day chat, and their most important activities. In fact, ChatOps may be taking the place of email in the workplace for internal communication for tech teams as it helps communication during DevOps activities like deploys, code pushes, etc. This episode discusses best practices (if there are any) of ChatOps and how to make sure you are getting the most from your team communication tools. Asynchronous vs. Synchronous Communication 25% of the work week is spent managing your inbox. You can actually increase productivity by moving to Sync communication…we think. Sasha: ChatOps creates an enormous amount of noise while at the same time makes communication grouped and searchable. Discussion suggests it is up to the user to mediate that noise, but is it the user, the culture, or the conversation itself that dictates the role of chat? Tivo is given as an example of user mediation: you recorded a shit ton of stuff, and watched only what you wanted. The panel comes to the conclusion that important decisions should lean away from ChatOps, and into a more formal, permanent form of communication. “Important things will ‘re-bubble’ again,” but the chatroom is not the place if a team consensus is needed, especially if the team is remote. Create a culture where ChatOps is used in the way you need. Risky to go “Super Pendulum Swing” in one direction or the other. What is ChatOps good at? Solving the communication problem. Brings everybody into the same experience. Even if you are across Europe, or accross the room, you are having the same experience. Great for in the moment Q & A. Even with one on one questions, if the answer is shared in a public channel, the information is given to all on the team which moderates the need for repeated questions, and increases team efficiency. You need to be constantly pairing. If you direct message someone, you are keeping that information from the team. “If you are not working in your chat tool, you are not collaborating.” Shared History Makes communication searchable, and organized by topic, or at least team. Rooms should be broken down to their smallest parts. Topics, Meetings, Projects, they should all be open spaces for all departments. Getting messages/alerts from integrated tools is perhaps one of the most important features of ChatOps in DevOps: Jenkins, Github, Travis, etc. What’s the Problem? There are just too many messages. But they are necessary messages. Internal ChatOps tool is almost useless when you are a consultant and you are all working on different clients. Problems with Adoption? In your organization, if you are considering chat tools for different purposes, use benchmarking and measurements to monitor your usage and data in each tool (in this case, chat vs. email) Matt uses RescueTime (https://www.rescuetime.com/) religiously. His current rate of email vs Slack: 4 Hours in Slack, 48 minutes in email. Sales, Support, etc. prefer email, but that will not change until their tools are integrated with Slack as well. How to make use for permanent communication? Have you adopted ChatOps for sustained messages and conversations that need to be kept? Pinned Items are like the refrigerator…it’s emptied at the end of the week. If it should live longer than a week, then it gets moved to the wiki, google doc, or the most appropriate space for the info. How do you practice Chat-Zero? (Comment with your answer) How do you go through every message in your chat? How do you know what is important? Last Thoughts: Peter can’t wait for computers to be smart enough to interrupt us only when appropriate. “Who is the least invested in their work right now? Let’s notify that person.” If you are considering it, do it. But be careful what you use, and how you use it. Jason: It’s new tech, but its the old problem. ChatOps is just the newest efficiency on the line. Sign up for the Banana Stand for the latest ADO news.

Matt, Trevor, and Bridget catch up with Kyle Kingsbury about his research on failure in distributed systems, lighting rigs controlled by code, consent and representation, and more. Call Me Maybe: an exploration of failure in distributed systems using the Jepsen tool. Monitorama 2015 talk on Riemann Bridget announces that she’s joined Pivotal; Matt claims this is to diversify the podcast so it’s no longer Chef employee, Chef partner, Chef customer. Community & Events Lots of open CFPS on devopsdays.org Chef Community Summit is Oct 14 and 15 in Seattle. Matt & Trevor will be there. Matt & Trevor: at DevOpsDays Chicago August 25 & 26! Bridget: at VMWorld the first week in September. Check Outs Kyle: Subnautica! Bridget: Sense8 on Netflix Bosh tutorial Trevor: Alphabet Announcement. Google pointing themselves towards the evil Silicon Valley “Hooli” company. Dragonball Z Resurrection F Herkimer NY, and herkimer diamonds Matt: Amazon Echo Helicarrier LEGO set

Charity Majors (Parse/Facebook) is an “Accidental Computerer.” She was the first infrastructure hire at Parse, which was acquired by Facebook in April 2013. Charity handles all of the backend operations and DBA work, and manages a team of 7 engineers. Patrick McDonnell is a web operations manager at Etsy. He made the transition from individual contributor to management a few years ago. Mike Rembetsy (MCR) is the VP of Technical Operations at Etsy. He was one of the first ops people at Etsy when he joined in 2008, and has helped grow the team to over 50 engineers since then. Charity points out that your first question should be whether or not you actually need an ops team. She says, “There are a lot of places out there that think they need traditional operations engineers, when all they really need is someone to really care about their infrastructure… You should have genuinely hard operations problems before you even start looking to hire engineers.” Once you do start down the path of building an ops team, MCR notes that you have to maintain it. You’ll need to grow the team, grow the individuals, grow the culture, which, if your company is in a startup phase, can be distracting to the overall goals. “The glue that holds a team together is how well people interact with one another, how well they respect each other,” says MCR. “That’s what you build good ops teams on, and frankly, that’s what you build any good team on.” As a growing company, we suggest you follow this process: understand/establish your mission communicate that mission clearly to the interviewing team find people who can fulfill that mission understand the fact that technical skills are great, but at the end of the day, it’s about the chemistry and culture of your team Trevor agrees that cultural concerns are absolutely an issue, and asks for suggestions on how to interview for that. MCR explains that at Etsy, they split their interview questions into technical questions and cultural questions about how the interviewees handled particular situations, as well as other skills. Charity notes that good ops engineers are good at learning things, but some people freeze up when they’re put on the spot. She suggests providing interviewees with at least 50% of the questions beforehand, so they have a chance to prepare the preliminary information before interacting with her in a formal interview. “You want people to bring you the self that you’re going to be working with on a day-to-day basis,” she says, “not the self that is freaking out and wondering how they’re being perceived.” Transitioning into the topic of management, Charity brings up the point that if, as a manager, you’re still responsible for key pieces of the infrastructure, you’re holding your team back in their technical development. One of the jobs of a manager, MCR says, is to help motivate your team, and then step aside and let people get things done. In doing this, you let them succeed, as well as fail, and grow as a result of those experiences. He references Daniel Pink’s book, Drive, and the three pieces of science that motivate human beings: autonomy, mastery, and purpose. “A manager’s role is a facilitator,” says Patrick. “Everyone should be doing what they think they need to do. It’s my job to remove obstacles that come in their way and make sure I can smooth things out if need be, but really, it’s to encourage and allow people to reach their full potential.” Etsy has two separate career paths: one for individual contributors and another for management, which allows for the honing of particular skills specific to the end goal. Charity agrees wholeheartedly with this approach, and says emphatically, “Management isn’t a promotion. It’s a career change.” In addition, manager and leader is not synonymous. Some people are much better leaders than they are managers, and vice versa. It is also possible to be a leader while continuing as an individual contributor. In fact, Patrick points out that in some circumstances, you might actually lose influence when moving into a management role if you’re already a leader amongst your peers. Bridget asks the panel, “What’s your best advice for someone in the position of building an ops team?” Patrick: “Focus on hiring good people. Hire people that you like. Hire people that you trust. Hire people that, maybe they need to do a little more research, maybe spend a little more time on StackExchange than the next person, but you know that they’re going to get the job done in the way that you need to get the job done.” Charity: “Build your networks. Go to meetups, talk to people. Don’t just talk to the popular kids. Reach out to diverse communities and diverse crowds, and go meet people who are doing cool and exciting things, who are slightly off the beaten path. The more people you know, the better you’re going to be at hiring.” MCR: “Make sure that you address conflict. Make sure you create a safe place for the people you do hire, to have open, honest conversations with one another. There’s nothing more toxic to a team than people chatting behind other people’s backs. Check Outs MCR: – Open Source Utility for ELK – 2012 Velocity Talk from Dr. Richard Cook: How Complex Systems Fail Charity: – Guided Meditation for Adults: “Breathe in strength, breathe out bullshit.” Patrick: – Dr. Christina Maslov’s talk at Velocity: Burnout in Tech – 5-minute burnout self-assessment Bridget: – DevOpsDays Minneapolis talks Trevor: – Batman Arkham Knight – GitKraken Upcoming Events: DevOpsDays at #alltheplaces

Transcript Andrew Clay Shafer (@littleidea). Coming to you live from DevOpsDays Minneapolis, Matt and Bridget sit down with Andrew Clay Shafer in front of a live audience to talk about the growth of DevOps, explain some commonly heard but not always understood terms, and more (after a brief detour on why episode numbers on podcasts are obnoxious, and why this episode is titled “Eating Sushi with Andrew Clay Shafer”). Don’t know who Andrew is? He suggests you Google him, but then goes on to give a little bit of his background: he’s been involved in software development and technology for almost 20 years. After rooming with Luke Kanies, founder of Puppet Labs in college, Andrew got interested in operations and system administration. O’Reilly’s Velocity Conference was also influential in Andrew’s growth, and Andrew reiminisces about his presentation on Agile Infrastructure in 2009. John Willis speaks up from the audience, and strongly suggests going to look at the slides. Matt takes a moment to explain how Arrested DevOps started: “I started listening to John and Damon on DevOps Cafe and understood about 5% of what they were talking about… There are these things that we, as part of this community, tend to know, and what we try to do with this show is break it down for the people who don’t have the tenacity or stubbornness that I did.” Along that vein, Matt asks Andrew to expand upon the “wall of confusion” idea that was referenced in his 2009 talk, and has become a commonly-used (but not always understood) term in DevOps lingo. “It’s a jargony way to talk about the different incentives that exist between developers and operations,” says Andrew. “There’s a transition that happened as software became service-oriented, versus shipped on CDs, where the servers now become this critical part of the value chain, and if you deemphasize the system administration and operation of those servers, then you don’t actually have software. In the middle of these two worlds, where in one, systems administrators were for keeping the printers and the mail server up, to where they’re a critical part of the value chain in the new world, there are broken IT practices that don’t make sense when you’re trying to manage a service. It means recognizing that the best way to optimize a system isn’t to just throw random stuff onto production servers, and then make it ops problem, but to recognize that the infrastructure itself has become an application, and that you can manage these things as an application.” Andrew points out that as much as he enjoys the attention (and who wouldn’t?), he was simply in the right place at the right time, and the right people listened to him. He connected dots to take advantage of tools and practices that were already used to manage software process, and bring that into the infrastructure and operations works. Bridget asks, “You mentioned Agile, and you mentioned Scrum. I’ve heard you say that Scrum is a disease. Can you give us your thoughts on where that sort of stuff is going?” “My personal opinion is that Scrum’s impact on software development is net negative,” Andrew says. “I think it’s particularly bad when people try to adopt it in operations. It’s really susceptible to problems when you have any interrupt-driven work whatsoever.” He suggests pursuing kanban and chatops, making work explicit and visible – tools that allow both you and your management to understand the full context and value of the situation. The conversation transitions into talking about how to make actual changes to your operations and infrastructure teams, rather than always jumping through hoops to make the necessary changes to keep the pages up or the apps running smoothly. The answer isn’t to simply communicate how difficult it is to manage a system – upper management won’t understand the pain, and therefore won’t listen to the complaints. You have to involve the rest of the team so that they understand what you’re going through first-hand. You get empathy from suffering. This all plays back to Conway’s Law, as Andrew points out: “If you believe Conway’s Law is true (as I do), then you understand that your org structure (who communicaties with whom, who reports to whom, etc.) determines the outcome of any decision.” Bridget brings up the point that this is the essence of dogfooding: requiring not only your engineers to be in the code, but your employees to be using the products that you’re creating, so that there’s a general understanding of why things work the way that they do, and a buyin for the necessary changes. Bridget asks Andrew to expand more on what he thinks we are (and should be) optimizing for, which he touched on briefly during his talk at DevOpsDays. Andrew counters that in order to do that properly, we need to first frame the context, which is a problem that plauges DevOps, Agile, and many other systems with which people are trying to transform their companies – you can’t do something prescriptive until you have enough context to understand where you’re starting from. For example, the diet and exercise program you’d give to someone who’s relatively healthy and active is very different than someone with a different set of circumstances. By the same token, you can’t prescribe a solution to an infrastructure problem without first investigating the roots causes and understanding what the foundation is. However, if you model the world as everything is an agent trying to maximize some function, then the basic premise of your decisions is cause and effect. “Looking at the way people behave, and how this plays out within organizations, you might have very different patterns of interactions and patterns of health.” Andrew continues, “Therefore, what you’ll tell people do is very different from context to context.” Despite all of the different scenarios, Andrew argues that there are three things you can always do: Understand the incentives that people are motivated by Align the incentives with behaviors Radiate information to help people make different choices Wondering what the Nash equilibrium and Pareto efficiency game theories are? Here are a few links: Nash equilibrium: Wikipedia Pareto efficiency: Wikipedia Andrew’s slides on Leading a Learning Organization

Jeff’s background includes 30 years of experience working in people management. Specifically in developing the management skills, and careers of Software Engineering teams. He now works at Chef as the Director of Learning Experiences. What is Career Development Just thinking about career development in the workplace is not very common. Jeff discusses statistics around the specifics of Career Development and implementation strategies. For example, in one study, more than 60% of respondents said that Career Development was of “little to no” importance to their current employer. Less than 5% of employees at some organizations surveyed receive any career feedback from their current bosses. Jeff: There are different approaches people usually take as they grow into their careers. The first is a “go with the flow” approach which can cause issues when leadership skills are not developed in response. A second approach is more proactive in which you plan out and describe your career and where you would like to go in the future. What do you do? The panel discusses their own careers and their attempts to summarize “what you do,” their weaknesses, and their strengths. Especially if the description is to be seen by the entire organization. Can you brag? Should you be vulnerable? For some, listing strengths as an engineer can actually be more difficult than listing weaknesses. Jeff: “It’s a huge act of vulnerability to say who you want to be in an organization” Jeff: “Every Engineer is responsible for their own career development […] and you are 100% responsible for the career development for every engineer on your team” Can you be the Director of Flowers? The group discusses the merit of titles and how relevant and useful they are within an organization. Jeff: “Titles don’t matter, and they absolutely matter.” Ultimately, titles should be descriptive of what you want to do in that position and are indicative of positional authority more than anything else. Management is not for everybody. “It’s Not a Promotion - It’s a Career Change” — Lindsay Holmwood (http://fractio.nl/2014/09/19/not-a-promotion-a-career-change/) Matt: “I don’t like managing people that’s not my thing.” Jeff: Performance management is not the same as Career Development. Often, when Software Engineers get promoted to managers, coding becomes a secondary responsibility and People Management becomes a priority. For some, this is not the trajectory they want for their careers, and that’s ok. Providing context for feedback and guidance is a great way to identify these mismatches between current position and where someone wants to be. Jeff: Using analytics “Project Oxygen” from Google describes 8 qualities of a good manager. ( http://www.nytimes.com/2011/03/13/business/13hire.html ) Jeff: “The Beatle Book” - Ken Schwaber ( http://www.amazon.com/Ken-Schwaber/e/B001H6ODMC ) Trevor: Doing an emotional check-in to describe your current emotional state is a powerful management and communication tool (from the Core Protocols by Jim McCarthy - http://www.mccarthyshow.com/online/) Jeff discusses the usefulness of check-ins on every level within an organization. You know you have a good manager when: 1. They expresses real concern with your career development, separate from quality conversations. 2. They create opportunities for you to realize your goals. (or at least get closer) 3. They have your best interest at heart. …and then there’s the checkouts:

Stephanie Van Dyk @sevandyk is an SRE at Google, and has also worked on healthcare.gov. Mark Imbriaco @markimbriaco is co-founder and CEO of OperableInc. He’s worked previously at DigitalOcean, GitHub, LivingSocial, Heroku, and 37signals. Bridget starts by asking Mark what Operable is all about. Mark explains that Operable is trying to help people who are on the “pointy end” of incidents. They’re trying to build tools that help people collaboratively fix problems. “There’s a lot of tools these days that do things like wake you up and alert when you when there is a problem,” says Mark, “but we think there’s a lot of room to help people actually solve problems.” Stephanie briefly goes through some of the history of healthcare.gov, and how she first learned about it. Her position was unique, she points out. “We worked very hard, and very long hours… We were also in the fortunate position of having a lot of authority, which is important if you’re trying to fix a disaster. There’s a lot of problems to solve, and you don’t want any of your additional problems to be ‘Well, who gave you permission to do that?’” “That’s a really good point – not all disasters are created equal,” Bridget notes, “and maybe we should take a step back and think, what are the ingredients that make something a disaster?” Mark: I’m used to catestrophic problems that last for a few hours at most, or in the really bad case, mabe it lasts for two or three or four days, not something that goes on for weeks or months, so that’s a different perspective from what I have, so I’m super interested to hear about [healthcare.gov]. Matt: I think there’s the disasters where there’s a thing that happens, that’s maybe localized to one type of scenario; then there is what happens in an episode of This American Life, where it’s just one thing after another and everything unravels. There’s a quote from the episode that I like to think about when we think about these bigger disasters that are more than just an outage that may be far reaching: “One ingredient of many fiascos is that great, massive, heart-wrenching chaos and failure, are more likely to fail, when great ambition has come into play, when plans are big, expectations are great, and hopes are at their highest.” “I think you’re certainly right,” agrees Stephanie. “In order for something to be a disaster, the stakes have to be quite high… An outage that you find, and fix, and write a postmortem, and everyone learns something, and the users all get over their hurt, that’s not a disaster. That’s just life. At times, there are incidents that leave scar tissue in their wake, making people wonder for years to come if they truly want to use certain products, or trust their data to a certain company. Mark reminisces: “The gutwrenching terror is, are we going to get our customers’ data back, or is it just gone? As an ops person, there’s almost nothing more terrifying than losing data.” This provides a perfect segue into some of the non-obvious issues that arise with disasters. Stephanie brings up the point that you have to be prepared to regain the trust of your users. “How people think about your service is going to determine the fallout of it, and the impact. It’s interesting – it’s not something engineers like to think about very much, because they simply fix the problem. But someone has to be the one to reassure people that it’ll be ok.” Mark agrees: It’s really, really hard to be in the middle of responding to a serious problem, and also have to be the person who needs to communicate about that externally. There’s so much good will that can be gained from being as transparent and public as you can about what’s going on, without pulling punches or hiding, even if things are really bad. This is all well and good, but Bridget brings up a good point: “How do you know exactly how and what to communicate to people?” Stephanie: There are definitely rings of communication. You have to be able to talk to the other engineers who are working on the problem, and those conversations are going to be very different than how you talk to your customers, even if you’re trying to be super open and honest. Your customers don’t care about where in the logs you found that tiny error. They care about when it’s going to be fixed, and whether you’re actually working on it… Also, the person who’s in charge of solving the outage should not be the same person who’s in charge of communicating about the outage. You should have different roles for that. Mark agrees emphatically, and also noted that wording is incredibly important – not only what you say, but how you say it, and the words that you use. “There are three things I want to get across. The first thing is, I want to apologize to people. It has to be a sincere apology. The other thing I need to do is make sure people feel confident that I understand what happened. I need to display confidence, and a really firm grasp about the problem. The last thing I need to do is tell them what I’ll do to try to reduce the likelihood of something like this happening again.” The conversation turns a corner as Matt asks how you plan for outages and prevent disasters. Stephanie jumps in, and reminds us all: “If you don’t test your backups, you don’t really have backups. Similarly, if you don’t test your outage plan, you don’t have an outage plan.” She suggests setting up brainstorming sessions with a handful of people from your team, appointing a “DM” (dubbed “Disaster Master” rather than “Dungeon Master” by Tyler), and running through possible scenarios. Keep an eye out for a Kickstarter in the near future ;) There are definitely advantages to documenting incident reports along the way, but how do you balance the speed of talking through a solution out loud, and the value of face-to-face communication to build trust vs. the need to document things for posterity? Mark: How you interact on a day-to-day basis is also how you should communicate during an outage. The last thing you want to do is change your mode of communication when everything is falling apart and you’ve got high stress. Matt posits that sometimes what’s a disaster for one company isn’t for another, because of their size, their logistical capabilities, etc., but also, sometimes what is being presented as a disaster isn’t actually all that bad. Stephanie identifies the first benchmark as determining whether or not your users are hurting. “If they’re not hurting yet, you might have a disaster coming, but I don’t think it qualifies. But if your users are hurting, that’s when you really need to jump on board and get focused.” Mark agrees, and adds that being able to quantify how many users are affected, and in what way they’re affected, is hugely important. “That’s different than monitoring. Monitoring may tell you that the server’s down, but it doesn’t tell you how many users that impacts.” He reminds us that when you’re working at scale, “services are down for somebody literally all of the time. “What is the threshold where it becomes a disaster? When do you need to start talking about it publicly and in status? Those are questions you really need to answer up front.” Checkouts Stephanie catehuston.com, Accidentally in Code USDS Mark Pre-Accident Podcast Destiny (the Game) Bridget seedsavers.org - Organic heirloom seeds “Common Ground and Coordination in Joint Activity” by David Woods et al Trevor Pocket chainsaw Dragonball Z XENOVERSE MSOffice for Android Smartphones (Excel, Powerpoint, Word) Preview   Matt Does Not Commute - iOS/Android game iOS | Android Recreating highlights of Breaking Bad in GTA5 editor I’m also obsessed with Hearthstone on iOS now (World of Warcraft card game thingy)

Chris Read, Kevin Hubbard, and Yvo van Doorn are reformed BOFH’s (Basterd Operator’s From Hell). Chris is back on the podcast again, this time talking about his expereince as a SysAdmin in past lives. Kevin is currently the DevOps Engineer for BCycle at Trek Bicycle Corporation, and was a SysAdmin for 15+ years. Yvo previously worked at classmates.com and McGraw Hill Corporation as a SysAdmin, and is now at Chef Software. Before we get started, you’ll need to understand the origin story of BOFH. There were stories posted on Usenet back in the 90’s, supposedly authored by a computer operator named Simon whose sole purpose was to terrorize the users of his systems. The phrase “How great would my job be if it weren’t for the f***ing users!” resonated with many SysAdmins (and still does!). Matt starts the episode off asking what it was like as everyone was starting out in this field. Kevin: To me, it was sort of operating in a scarcity model. You had limited resources, and it seemed like anytime there was a new ask for an application, I immediately went to ‘How am I going to ask for the capacity to run this?’ and I would just get so frustrated. It boiled down to ‘How are we going to support this?’ That was my standard line when someone would bring up something new, and I wish I had trophies to give those people for all of their good ideas, because we just couldn’t get it off the ground with the resources that we had. It wasn’t a fun way to operate, but it was the most realistic view. Chris: When in high school and university I was the System Administrator for the school systems. It was astounding seeing what damage to the system can be done – how people trying to do something could affect shared resources, and the after-effects of that. Most of the time it wasn’t malicious; it was due to ignorance, but it built up this mental attitude of ‘All users are just there to break things. We need to constrain them as much as possible, because when things break, we’re the ones that get shouted at.’ Matt: There was a belief that devs are stupid! All they’re going to do is break things, because they don’t care about the systems like a SysAdmin does, because they’re ours. Yvo: Our devs were incentivized not to care because they were paid based on the amount of code they shipped. I’ve had some nightmare evenings trying to fix all of the problems. Bridget then brought the conversation back around to incentives – are there situations when the incentives are diametrically opposed (or at least not aligend well) between the SysAdmins and the developers? Matt brings up the point that developers are incentivized to build features, while SysAdmins are incentivized to bring stability, which at its most basic level is maintained by things not changing. The viewpoint of ‘developers don’t know what they’re asking for’ is also a problem, Kevin reminds us. SysAdmins will often call the developer and explain why things work the way that they do, but won’t take the time to listen to the actual problem. In reality, there’s a perception of other people touching “our stuff” and things will go wrong, but let’s face it: “there are all sorts of things that can go wrong that are often not a specific person’s action,” says Bridget. Given that all of us here are supposedly reformed BOFH’s here, let’s chat about how things have changed, and what that process was. Chris: I finally realized that my interactions with the developers were better if I went to them without the ‘clue stick’ and simply spoke to them, asking them if they realized the impact of their code. It finally clicked for me when I had to work together with the client-side SysAdmins as well as the developers at Thoughtworks. Our whole purpose was to get code written by two different development teams out into production, and it was only through being an advocate for both teams that I was able to build up a relationship with both teams and understand the value. “It seems like DevOps has formalized the relationship between SysAdmins and developers,” says Kevin. “It seems like a much more natural, iterative process working with devs.” Because we’re working side by side, there’s much less going back and forth with having to figure out the direction and purpose behind projects, and simply getting to collaborate. The “handing down stone tablets” philosophy not only no longer works… it has never worked! In Yvo’s case, the change started to happen when the project management team was dissolved. Suddenly, a SysAdmin had to be a part of the development meetings, because there was no longer an intermediary passing information from one team to another. It immediately became more collaborative, and there was visibility into what was happening early on rather than being notified after everything was finished. We’ve shifted from a mentality of ‘protection’ – teaching our “PFY’s” (pimply-faced youths) to protect their systems against the evil developers – to giving history lessons about how we got to the stage that we’re at now where we need to talk to all of the involved parties, and as Chris said, “having everyone focused on the goals, trying to see things from each other’s angles rather than antagonistcally.” This is how we, as a group, move forward. “It takes a deliberate decision to shift,” Bridget observes. We have to be dedicated to teaching our PFY’s this new, collaborative way so that in the future, fewer people will start out with this BOFH mentality. From there, we shift into How can we do better? Matt asks, “We used to be this way – we’re better now – but what are some of the ways that we can still improve?” “I want to be able to maintain this new flexibility that comes with DevOps, but I feel like there’s some decision-making that needs to be made as far as tools and standards go,” says Kevin. It’s a matter of balancing the old playbook of limited resources and mixing in the new cohesion and collaboration efforts. We’ve done a great job of bringing in the greater teams of operations and developers, but most companies still have the one or two lone SysAdmins who are struggling on a daily basis to keep their heads above water. Yvo cautions that we need to bring them into the circle as well. “If we can make their lives easier, they’re going to eventually go to another shop with the perception that being alone and supporting developers is not a bad thing, but right now they really don’t like life.” Bridget’s money-back guarantee: “If you’re less BOFH-y, I promise you’ll be happier, or else we’ll pay for your therapy.” Checkouts Chris Liz Keogh - Perverse Incentives GOTO Conference Chicago 2015 Closing Keynote by Anita Sengupta from NASA JPL was awesome James Lewis - “How I finally stopped worrying and learnt to love Conway’s Law” DRW is hiring! Kevin Stache - new Trek mountain bike Yvo Hop Venom Double IPA - Boneyard</a Bridget Organic heirloom seeds Common Ground and Coordination in Joint Activity by David Woods et al Matt Effective DevOps by Jennifer Davis and Ryn Daniels asciinema

Shannon Smith is the marketing manager for 10th Magnitude, a consulting firm based in Chicago. She’s a self-described “liberal arts nerd” and fell into the tech world through her job at 10th Magnitude. She has a love for well-crafted sentences, song parodies, and general cleverness, which has translated into a love for marketing and the challenges that come with getting an effective message delivered to the “great abyss.” Jason Hand is a DevOps Evangelist at VictorOps. He’s been in the IT space since college, but it wasn’t until joining VictorOps that he’s ventured into a wider role, working closely with both marketing and product. Matt starts off this unique episode by posing an important question: “Why is DevOps something that people in marketing would even care about?” Jason jumps in right away: “Obviously, the VictorOps tool is something we feel falls in line with a lot of the DevOps topics, in terms of the things we try to help engineers with. Being able to speak intelligently to the different subjects that fall within DevOps is very important, especially when it comes to marketing. As most of us know, we’re very averse to traditional sales and marketing tactics; we can sniff out those types of conversations before they even happen, and we’ll pivot and walk away.” Shannon originally broached the topic during Open Spaces at DevOpsDays Chicago 2014. She was interested in the focus on community, but also intrigued by the jokes about how terrible traditional sales and marketing is. She wanted to know what she could do to combat the negative feelings surrounding these departments, as well as gain insight into what her team could do differently. The group starts throwing out common pitfalls: Using #devops on Twitter and thinking that in doing so, your company is part of the conversation Packaging DevOps as a quick fix to deep-rooted problems Trying to follow traditional marketing and sales “rules” “Even for those of us who are heavily invested in this DevOps thing,” says Jason, “who talk about it on a daily basis, sometimes we don’t even understand everything – all the ins and outs of what is DevOps – because it’s not just a thing. It’s a way of getting things done. It’s much bigger than just one thing. We are very patient and empathetic to that thought process,” he continues, “because we deal with it all the time. That’s part of the challenge within marketing, is how to turn that conversation into a succinct way of explaining it.” Shannon explains that part of the problem with having a succinct explanation for DevOps, as well as how to sell it, is that there are multiple target audiences. “While we do use more traditional marketing for ‘decision makers’ – people who are busy and don’t have the time to talk about these abstract concepts, we’ve recognized that there are two or three very separate audiences, and we have to make multiple different messages work.” The point is, your message is going to be very different depending on whether your audience is practitioner, decision maker, or champion. It could be that your company caters to all three, but then you must have separate messages catered to each of these groups. Otherwise you’ll be trying to mandate change from the top-down rather than getting the buy-in from people doing the actual work, or vice versa. In a conversation with Matt earlier in the week, Steve Pereira said, “I’m in favor of having marketers speak to what they’re actually selling, which is never actually ‘DevOps.’ DevOps is bathwater in which all of these things are babies.” Given all of these different audiences, Bridget asks the “elephant in the room” question: “When you’re talking about the target markets, how do you build an understanding of DevOps among people who aren’t engineers?” Jason mentions that at VictorOps, they have an emphasis on moving agiley across the company. Even the marketing team works in sprints, getting things done on a project basis and having team standups. DevOps isn’t just for engineers – it’s applicable for everyone across the company. But how do we make that happen in a practical sense? Companies that are doing this successfully often exhibit certain qualities: Empathy to others, which allows silos to come down Transparency between teams via chat clients and in person Cross-company knowledge of what the top priorities are and what other departments are working on This cross-company communication can be difficult at times, but ultimately it’s everyone’s responsibility to speak up if they see problems with the messaging. Bridget offers a solution: “If you’re going to have people external-facing in your organization, going out and evangelizing, they need to also be talking to the people inside the organization, who are developing the product or providing the professional services.” With Jason’s role as DevOps Evangelist, he’s often out learning from the best. It’s his responsibility to bring the feedback directly back to the company and educate them on what he’s hearing from the community, and likewise it’s marketing and product’s responsibility to be open and willing to hear the feedback. Collaboration and accessibility is what drives all of this, as Shannon pointed out. Being able to talk to people face-to-face is one thing, but using chat software to check in with the full company, or somehow checking in frequently with multiple teams is key. For our listeners: If you work in a larger organization, we’d love to hear about how you’ve solved this problem of communication throughout the company. What’s resonated, and what hasn’t? Tweet your answers to us at @arresteddevops. Main takeaways: Jason - Spend some time researching DevOps. Get to the DevOpsDays events that are happening in your area. That’s where I’ve learned the most. I’ve made some great friends and contacts. Also, these events have helped me have the sense of empathy not only toward the challenges of the marketing and sales teams, but also the conversations that are taking place among all of the business units, no matter what size company. I can now take what I’ve learned, both online and offline, and take those insights and thoughts back to my team. For the time-being, I’m the one internally who’s teaching DevOps best practices. It’s really a matter of absorption. You can’t sit behind your desk and understand DevOps. You really have to get involved with others. Shannon - Truly the biggest thing that I’ve taken away in the past 9 months is really trying to build up our grassroots messaging and the way that we’re reaching out to people. I think the best way to do that is go to these community events – be involved, listen, share, have natural conversations, and be genuine. Having a presence in the community says a lot more than any little tagline. Checkouts *Shannon* [Graze](https://www.graze.com/us) & [Naturebox](https://naturebox.com/homev2/) - when you need a 3pm snack [Canva](https://www.canva.com/) - graphic design online tool /// insert picture of Trevor in beret w/a baguette here [Unbreakable Kimmy Schmidt](http://www.imdb.com/title/tt3339966/) Jason [Trello](https://trello.com/) - light-weight Kanban boards [Buffer](https://buffer.com/) - tweet scheduler [Mention](https://mention.com/en/) - keep a pulse on what topics people are engaging in [DevOpsDays](https://www.devopsdays.org/) - local DevOps events Bridget [DevOps: A Brief History](http://peterjshan.com/posts/2015/04/devops-a-brief-history/) Trevor [David Bowie's Exhibition](http://davidbowieis.philharmoniedeparis.fr/en) [Agent Carter](http://www.imdb.com/title/tt3475734/) Matt [Mac ID](https://macid.co/) [Octotree Chrome Plugin](https://chrome.google.com/webstore/detail/octotree/bkhaagjahfmjljalopjnoealnfndnagc) [PonyHoof](http://ponyhoof.little.my/)

Seth Falcon is the Engineering General Manager for Chef Delivery. Julian Dunn is the Product Manager for Chef Analytics. Adam Edwards is the Engineering General Manager for the main Chef product. Matt, Trevor, and Bridget gather at the Chef Newsroom at ChefConf 2015 to talk about the latest and greatest developments at Chef. They are joined by Seth Falcon, Julian Dunn, and Adam Edwards. One of the biggest announcements at ChefConf 2015 was Chef Delivery. Matt asks what is Chef Delivery, and why is it interesting? “Chef Delivery is a solution for continuously delivering infrastructure and applications, and it’s built on top of Chef,” says Seth, who’s heading up this new program within Chef. “We’re really excited about Chef Delivery. Successful software organizations use patterns to deliver software at high velocity collaboratively and safely. We’ve been able to distill some of those patterns into a workflow that we think will be easier for folks to adopt and learn.” Seth continues: “The workflow is one that we’ve seen work, by working with customers to build these types of pipelines over and over again, and find those successful patterns. The overall workflow begins on a developer’s workstation, where they make a change, they do some local testing, and then submit it to the system, where some automated verification tests run. The job of those verification tests is to determine whether it’s worth the time of a human to do some code review on that change. Someone can then do some code review to approve the change. At that point, Delivery will build an asset for us that we could release. The workflow for building that asset is simple: do a merge onto the target branch (usually the master), rerun the same verification tests, which usually consists of unit tests, lint testing, and syntax checks, build that asset, and publish it into a repository where it can be fetched later. Then Delivery will provision an acceptance environment, should you need one, and deploy that asset into that acceptance environment, and run some tests to make sure that the deploy was successful. If it was, it waits there for further instruction. The last step is clicking the ‘Deliver’ button. That sets the system in motion to get the code all the way out. It first goes into a ‘Union’ stage, where if you had a number of projects that had some interactions, you’d be testing them together at their latest version, and making sure they’re good. If those tests succeed, Delivery rolls automatically for the rest of the stages – into a rehearsal environment, and then into a delivery environment.” Interested in a more detailed description of the workflow? Seth continues to talk through some of the logistics in the podcast as Bridget and Matt ask questions about particulars. You can also check out this video about Delivery: Seth concludes with, “A lot of what we’re providing here is an accelerant to teams, to give them that system that will allow them to move quickly and learn how to move quickly in that way.” We transition into asking Adam Edwards, Engineering GM for the Chef product, what’s new in his world. “There’s a lot of new stuff just coming out over the last few weeks,” says Adam. He goes on to detail just a few new features: Policyfiles - solves problems with workflow and provides a way to specify in a specific file which cookbooks you want Test Kitchen on Windows PowerShell DSC Direct Integration Chef Provisioning (neé Chef Metal) update - does more with containers and Azure Chef Nightlies - releasing Chef Server nightly on apt and yum repos From here, we move into Chef Analytics, and turn the mic over to Product Manager Julian Dunn. He gives us a quick background on Analytics, explaining that it “gives you a way to visualize, query, and report on the events stream that’s coming from your Chef data. Analytics lets you not only visualize that data, but track events in that stream, and then handle them in various ways.” Matt brings up his favorite part of Analytics, and brings up how it’s closely affiliated to security: “You can take those same audit rules that you write in Analytics, and apply them through your pipeline to test them there. What’s nice is that it’s only one thing to write.” It keeps things simple and straightforward, rather than needing to search through a huge amount of rules and regulations to ensure that everything is accounted for. Julian agrees: “You can think of security as just another aspect of quality. We understand that you can’t get quality if you try to bolt it on to the back of a system. How many of you have worked with applications that didn’t have tests originally, and the software is just poor quality? We tend to treat security in this backwards way, where we think if we don’t build it into the system, down the road we can just do an audit and we’ll magically get compliance and security. If it’s not a characteristic that’s already there, it’s very difficult to achieve those directives.” In addition, Analytics allows customers to report metrics and characteristics about your infrastructure to business owners. “One of the things we often hear from customers,” Julian says, “is how do I measure how successful I am at Chef?” If you’re able to use Chef Analytics, it provides you with a direct way to illustrate the ROI of investing in this type of technology. This business aspect of Chef Analytics was one of the core announcements regarding the Analytics Product Suite at ChefConf, specifically highlighting the integration with Splunk. Julian explains the rationale behind this integration: Many enterprise customers are already using Splunk Splunk makes it very easy to draw visualizations and make inferences from data Matt segues into an overview of ChefConf 2015 and asks everyone for their insights on this year’s event. Bridget: One thing that stood out for me at ChefConf is that there’s a very unified community feeling. At a lot of conferences (that are very wonderful conferences!), there are very specific tracks, or specific groups of people who end up mixing more than with others, and at ChefConf, it definitely seems like there’s a very broad community who are all interacting with each other. Trevor: I’ve been rocking the hallway track… and like Bridget said, everyone’s been fantastic. I’ve spoken with so many people here who I thought would never want to give me the time of day outside of our show, where we kind of have them pinned in a corner (laughs). The Open Spaces in particular were fantastic. Brandon Burton brought up a very sensitive topic: mental health, burnout, suicide… and it was such a breathtaking and emotional experience to hear everyone share personal experiences around that. Julian: It’s really great that even as we’ve grown as a company and a community, we’ve retained certain attributes about that community. I think one of those is that there’s a little bit of quirkiness, and a little bit of uniqueness, and fun. Backend automation and IT has not necessarily been the most fun arena to work in, and I think this is a breath of fresh air to that sector. I hope we’re able to retain those attributes even as we continue to grow. Matt: Some of this experience is really hard to explain. You can’t explain what the Las Vegas strip looks like at night, even if you’ve seen it in movies, until you’re there. I’m not saying this is like going to Vegas, but just like no one can describe the Matrix to you, you have to experience ChefConf for yourself. More information on ChefConf: Watch keynotes and highlight videos Read wrap-up blogpost ChefConf 2016 Links: ChefConf Unikitten #cheffriends