Arrested DevOps

How to Change the Culture of an Organization “Change management rests with the how” – Bill Joy DevOps revolves a lot around what an organization and culture should look like. We talk about it on just about every episode of this podcast. Something we tend to skate around though is the how. How do you change the culture of an organization? Matt got to sit down and have an incredible conversation with Bill Joy of the Joy Group about the how of influencing change. We didn’t talk about what changes companies needed to make, we talked about how to get companies to make the changes. Bill shared the often overlooked fact that change influence isn’t restricted to top levels of power in a company. If I were to ask you right now, “What would make your company better?” I’m pretty sure your mind goes into overload with all of the things that you would change if you could. The thing that most people fail to realize is that they can make the changes, or at least influence them. It doesn’t matter what level of the executive chain you sit at, you have the ability to influence change. Any senior leader can mandate any change. If you’re a senior level executive, you can walk into the office and implement new strategies, create a new company wide rule, begin rapid movement events, or whatever you see fit to accomplish your ‘what’. But it can’t stop there. In mandating a change, you have to be very aware of how it will affect the company itself; the infrastructure, the culture, anticipate any resistance, who are the key stakeholders are, etc. If you don’t allow for these, your change could be very well unsuccessful and you lose a great deal of credibility as a leader. Non Executives Can Influence Change Too You might remember back when we had John Allspaw on our Etsy episode. (Give it a listen, if you missed it.) When asked “How do you implement developments in an organization when I’m not on the top; I’m coming from the bottom of the chain.” Allspaw answered: “I don’t know. I’ve always been in charge.” What a great position to be in. However, not everyone is a high executive with the capability to mandate change. This doesn’t mean that you can’t influence change. The question arises then: How do you influence change in your organization from the bottom? Ask yourself: Who are my key influencers? Change­ management is the influence of authority. That said, as you talk to the person you have deemed to be your authority, you have to remember the power of empathy. Patrick Debois once said “We should stop calling it DevOps and call it common sense.” Here’s the problem with that, not everyone is going to see it your way. Common sense is a relative term. If during any of your influencing meetings you find yourself thinking “This is crystal clear to me, why aren’t they seeing it?” that’s more about you than it is about them. When you make the decision to mandate, or in this case, influence change, it is your responsibility to present your ideas so that people can see them clearly. If they aren’t seeing them clearly, there is a good chance that you aren’t communicating them effectively and didn’t take into consideration who your audience is. Identify Your Audience: It’s all in the personality One of the most popular personality tests around is the MBIT (Myers Briggs) which identifies the strengths and weaknesses of a person’s general personality. There are a bunch of alternatives to the test, but one that works really well with change management is the DiSC Profile, which you can take for free here. The test measures your strengths in four areas: Dominance; Influencing; Steadiness; and Compliance. All of which have different traits associated with them.   Before you attempt to influence another person, you need to have a firm grasp on who you are. For example, if you’re heavily dominant, your natural approach isn’t going to be effective on someone who is compliant. You have to adjust your approach and presentation to your audience. Once you understand yourself, know your triggers. As a dominant go-getter, someone who takes a while to process things or lacks energy may really test your nerves in a social situation. Realize this upfront, prepare for it. Read; Assess; Adapt & Tweak Make it a point to put some time into reading your key influencer. Take special note of the tone in their emails, the speed of their decision making, their general demeanor, etc. These are going to give you clear indications of what personality traits they have, and which you need to play on in order to effectively communicate with them. From here, you learn to adapt. Adapting is essentially wrapping the package up. Maybe you thought at first that you would have to have a presentation based on making things less structured to allow for more success. But, you realized you’re in a room with the key influencer who is very compliant; he/she likes configuration and organization. You then have to be ready tweak your strategy to make the presentation more effective and geared towards them.. Taking On The Role of Consultant Regardless of whether you're a hired expert brought on to assess the state of a company or if you're attempting to influence change internally, you're going to be taking on the role of the consultant. As the consultant, it's important that you remain committed to your views and intentions regardless of the initial feedback that you may get particularly if you're trying internally influence change. The key is to look at your key influencer as your client instead of your boss for the purpose of this project. If your client (who happens to be your boss) shoots down your ideas and offers a counter plan, you have to be ready to channel some boldness and stand firm in your beliefs. Instead of arguing back and forth or worse, completely backing down because of his/her position above you in the company, you could say something like “I see what you’re saying, but if we do it your way, this is what is going to happen”, and then lay out how their plan is flawed. It's Not The Size Of The Company That Matters, It's the Agility When considering the speed and likelihood of a company to change, size shouldn't be the deciding factor. It's seems logical that a larger company would be more difficult to change than a smaller one, but often times it's the smaller company that shows more resistance.  Don't let size lead you to any assumptions.  Instead, there are 5 key areas that you should take a look at: Risk Tolerance: Regardless of the size, does the company encourage risk? Do they punish risk-takers? Speed of decision making: Is the company bound by politics? Hierarchy? Take notice to how the new hire process happens. Levels of authority: Watch the style of how many people need to weigh in on a decision? When you go to your key influencer, is he/she going to refer the proposed changes to a higher level? Will that level then refer is even higher? Empowerment: There is usually a tendency to refer empowerment higher. empowerment to the next level up.  Can you start changing the direction of empowerment down in an organization? What you want to do as a consultant is be able to go in and say something like "Can't this be solved by level 2 instead of going to level 6?" Voice of the customer: Does the organization's customer base see the company as helpful? Are they happy with the customer service? How do they give products? How do they handle product returns? Are they satisfied in general? What's often found is that organizations are more flexible with their external relations than their internal relations. Compliance VS Commitment Whether you’re doing the influencer or the implementer of change, you’re going to come across two types of people who “sign on” to your changes. The Compliant One: The compliant person will sign the documents, put their time in, “do their TPS Reports”, as Matt said in the podcast, punch the clock day in and day out. They’ll do it because they need a job, or are too lazy to leave. Whatever the reason, they’ll make the changes, but won’t really care much about it. The Commitment One: The commitment person will simply, “Believe in the TPS Reports”. They’re dedicated to the idea of change and are committed to the success of the company. As you have more commitment, the success of your company will be exponential. As Bill said; “Find satisfaction in the pursuit of commitment.” The best change influencers are those who don’t see people as something they ‘have to deal with’. They’re authentically interested in them, and enjoy the pursuit of the how of change more than the change itself. Checkouts Bill The Martian by David Weir The Bone Clocks by David Mitchell Managing Transitions by William Bridges Matt HabitRPG

Tonight’s episode featured a special guest host, Julian Dunn of Chef. Our panel: Ryn Daniels - @rynchantress - a web operations engineer from Etsy. Jake Champlin - @grubernaut - an operations engineer from Minted. (Younger than Trevor!) Matt asks Ryn what it was like going to a place like Etsy - the gold standard! - what they expected, and what it was like when they got there. Ryn says that years of reading codeascraft and seeing etsy engineers speak at conferences meant that they were excited to start but also felt a little impostory. “Everyone knows that devops is the internet and the internet is cats - I show up my first day and my desk is covered in cat pictures.” Jake talks about how devopsdaysPGH was his first tech conference, where he got connected to the community and met his future boss Alex Nobert. Julian asks how Jake got connected to Minted, and Jake talks about Bridget introducing him to Alex and how Minted seemed to have a great culture. Trevor asks what stands out about Minted’s culture, and Jake mentions that it’s woman-owned and has a strong focus on the product, and how everyone’s motivated to make a quality product and it’s reflected in your infrastructure. Bridget suggests this might be the inverse of Conway’s Law, while Matt asserts that most “inverse Conway’s Law” discussions just end up proving it. Jake mentions how the community of Minted votes on some products, and the engineering team tries to bring that same excellence throughout. Bridget asks Ryn how Etsy’s engineering practices influence or are informed by the culture of the company. Ryn mentions Etsy’s B-Corporation status, which means they are focused on social good, transparency, giving back to the environment - and that’s reflected in a lot of what they do, from codeascraft to blog posts to involving sellers in the community. Both guests give a short version of what their company does; Ryn says that aside from the monitoring and sending everyone to Velocity, Etsy is the largest marketplace for handmade and vintage goods. When Jake explains Minted, it’s revealed that as “lazy podcasters”, we didn’t realize that Minted was as artist-focused as it is, providing artists a platform and community - Bridget thought it was primarily for paper goods, while Matt thought it had something to do with money. Bridget asks Ryn about their decision-making process to decide that Etsy was right for them. They said that even though Etsy folks they met at Velocity didn’t know them, they were welcoming and never condescended to them. (Also, pink-haired thought leadership.) They also admired how Ian Malpass was mentioning at devopsdaysMSP how he wanted to put together a class for effective male allies inside Etsy. Similarly, when Jake went to devopsdaysPGH, the way people were open and welcoming is what made him know he was in the right place. (A restaurant-related digression led to a shout-out to Jon Cowie’s knife-spork.) Matt mentions having the feels and being excited about getting a chance to work with Julian Dunn! and how when starting a new job, when you feel intimidated, people in a good culture are going to reach out to you and make you feel welcome. Julian asks our guests what the experience of starting at one of these devops gigs was like. Ryn talks about how starting at Etsy is less-stressful than smaller jobs in the past (since payroll isn’t an open question) and then mentioned the engineering rotations that allow new people to “bootcamp” with other teams, to increase understanding across the site. They also mentioned the “first push” program, where everyone (even non-engineers) learns to deploy a change to the site. Jake’s working remote, and his onboarding process was like being thrown into the deep end of the pool. He felt impostor syndrome at that point. Bridget agrees that it can be an intimidating feeling, and asks Jake how he deals with that. He says that he’s working with really smart people and is happy to learn new things every day. Trevor asks about cultural cues. Ryn says that they have a chat-heavy culture, and because there are so many remotes, even the in-person team will communicate as if they are remote, so you get to know what’s going on with the team and what their favorite cat gifs are. Trevor mentions introducing hipchat to a client and how it caught on quickly. Jake mentions that although only ops and qa are remote, everyone at Minted uses hipchat. Matt believes it’s frustrating when there are people in an org aren’t on chat, and Jake agrees that having the whole org there (HR/payroll, etc) makes communicating easy, even as a remote employee. Ryn: “I do get to sit ten feet away from John Allspaw, so I’ve got that going for me.” Matt: “If you’re playing the Arrested DevOps drinking game, that’s a drink.” Julian asks what are some of the downsides of a chat-heavy culture. Bridget mentions that at DramaFever, people have sometimes found chat to be distracting, if people are wanting your attention all day, and Ryn points out that at Etsy, it’s considered okay if someone needs to be heads-down working on something, either turning off notifications or signing out of chat. Trevor mentions that working with a global team can mean unintended awakenings from 3am @-mentions, while Ryn and Jake are both in cultures that encourage not having chat on their phones. Bridget says that at DramaFever, the solution is spaces in someone’s name so as to not alert them during off hours. Jake talks about how being in an oncall rotation instead of being oncall 24/7/365 is great, and after a week of oncall, at Minted they get the next Friday off (and their co-workers will kick them out of chat if they join). Bridget asks Ryn about their blog post on work-life balance, and Ryn says they wrote that about disconnecting, and Etsy also has a culture of asking people to take care of themselves (where they threatened to take away their VPN access because they were working while sick). Julian, who reveals he is stranded in Chicago and that’s why he is on the show, turns the conversation back to finding good roles and asking what role a recruiter plays. Ryn says that an Etsy person being oncall and troubleshooting during a Sysdrink meetup in New York attracted a number of applicants. That, codeascraft, speaking at conferences - showing what an org does, instead of just saying “we’re hiring” - works better. Bridget asks them what makes a job a place they know is right for them. Jake says the culture, and working on interesting things with smart people. Ryn agrees, and says that their first week at Etsy they got to start contributing right away. Bridget points out that new people have a power that people who have been there longer can never get back - the power of not knowing how things are “supposed” to be (and the ability to write documentation to better serve a “don’t know the answer” POV). Ryn says that breaking Nagios Herald their first week (and then they also got to experience Etsy’s culture around blamelessness.) Jake is in a more greenfield situation, and so he was able to start contributing immediately out of necessity. Bridget asks for the panel’s best advice for people who are interested in a “devops” job, want to find such a job, etc. Apparently the answer is having Bridget introduce you to people and/or convince you that you’re totally good enough to work somewhere. [Note from Bridget: this may not scale.] Both Jake and Ryn point out that connecting with the community on Twitter is a really valuable place to start making connections. Matt points out that liking your co-workers isn’t so much “nepotism” - you don’t have to party with them - but you spend a lot of time with your co-workers, so you’re going to want them to be people you don’t hate. Matt says, “It’s not know the ‘right’ people, it’s just ‘know people’.” Jake points out that at PGH, Mark Imbriaco and Ben Rockwood spoke to him as if he was a friend, even though they are prominent in the community. Ryn encourages everyone to use Twitter. Checkouts Ryn git troubleshooting tmate - terminal sharing Jake Running Linux on a Mac Helena Nelson-Smith on Mental Overload Julian http://usersknow.blogspot.com/2015/02/your-job-is-not-to-write-code.html http://www.isaacchansky.me/days-since-last-new-js-framework/ Trevor Laphroaig PX Cask Triple Matured Scotch The Book of Mormon How we upgrade a live data center - http://blog.serverfault.com/2015/03/05/how-we-upgrade-a-live-data-center/ Bridget Fat Bike Birkie - http://www.birkie.com/bike/events/fat-bike-birkie/ Bike camping in Napa 30 Days of Biking: http://30daysofbiking.com Matt The Goat Farm - Enterprise DevOps podcast by Michael Ducy and Ross Clanton. DevOps Checklist - from our buddy Steve Peirerra

James Turnbull describes Docker as “a solution that is built by people to be usable by people, as opposed to some of the previous containerized solutions which were built by engineers to be usable by a very small subset of other engineers”. When you might want to fly less… “when you start to recognize the airport lounge staff and the flight attendants on the New York-San Francisco route and they start to recognize you.” James wrote The Docker Book to allow people of varying skill levels to quickly understand how to use Docker and what the practical applications could be for them. It’s intended to be a practical how-to guide. At Kickstarter, developers have a dockerized replica of production on their laptops. Matt asks if Docker can be only used in completely new deployments designed for Docker from the ground up. James points out that if you have existing infrastructure tools, it’s simple to create Dockerfiles from them. The night before 1.0 launched at the first Docker conference in mid-2014, James removed all references to “don’t use this in production” from docker.com. James mentions that Fig (soon to be renamed to Compose) helps with modeling multi-tier architectures locally. James says, “People kinda forget the past and go, “oh my god Docker’s a pain in the ass to use”, and I’m like “compared to what, exactly? Compared to your previous build, or compared to you shipping around 10 ISO files and running Vagrant and 20 VMs on your local machine?” He continues, “It wasn’t that long ago that the dark ages were real. I’m not suggesting that Docker’s a panacea, but it’s certainly a step in the right direction.” James points out that something like Elasticsearch does well in Docker, since “it’s a bit of a fiddly thing to build, with the right version of the JVM, right version of Elasticsearch, prepping all the data, etc”. James highlights continuous integration as a “sensational combination” with Docker. On the controversy, James points out there will always be hype and people claiming “this is a revolutionary technology that will cure world hunger”. He says, “I’m fond of saying that Docker is a powerful tool to help you in your development life cycle […] not every workload in your data center is well-suited to Docker.” James doesn’t make technical architecture decisions based on the writing of tech journalists or blog posts, but rather by testing and evaluating the relative merits of a given solution. In the case of Graphite, James would run carbon-relay and carbon-cache inside Docker containers, but he’d point them at a physical machine with SSDs to actually write the whisper files. Matt read a blog post and reactions on reddit and wanted to see what James thought of the concerns around security and operability. James points out that empathy for developers is something sysadmins need to cultivate, because you don’t manage infrastructure for infrastructure’s sake. James points out that the main reason developers ship code that doesn’t work in production is that they have no fucking idea what production looks like because there’s this grumpy asshole that manages production and they’re terrified to go ask them a question. Bridget says that as such a former grumpy asshole, she’s much happier when the devs aren’t afraid to talk to her. James mentions that Docker containers are not virtual machines and should not be used to separate security concerns, and you should secure the host the containers are running on. Matt: “I’m not suggesting that this [security concerns] is why DOCKER BAD…” Bridget: “Race conditions with devicemapper is why DOCKER BAD.” James: “[PCI/DSS] is a low bar. If you followed simply the regulations for the compliance stuff that related to PCI/DSS, you would be running a massively insecure system.” James points out that “owning” the standard gives one access to the marketing around an ecosystem. He also thinks that even if Rocket is a better technical solution, Docker has more traction. Bridget: “So when I feel ranty about Docker and devicemapper, I should submit some pull requests.” James: “You should talk to Michael Crosby… Michael Crosby is currently in San Francisco somewhere going you motherfucker.” James sees Amazon and Microsoft’s embracing of Docker as a great driver of revenue towards these cloud providers, if it gets developer code to production faster. They aren’t following hype; there are transparently obvious business reasons to do it. In terms of skating to where the puck is going to be, James suggests looking at orchestration, software-defined networking, software-defined data centers - people building that sort of thing with Docker components. Docker Compose, Docker Swarm, people moving up the stack to manage different levels of abstraction. James: “I challenge you to find a LAMP stack site where 80-90% of the configuration files aren’t identical - our secret knowledge of what to tweak isn’t as valuable as we think it is.” Check outs James Jason Dixon’s Monitoring with Graphite book The Art of Monitoring - James's upcoming book Bridget Spent a week with my Philly & NY co-workers, went to the 3rd annual DramaFever Awards show, sang more off-key karaoke. Docker to Ducy Chrome plugin Matt Was in PDX for the first time last week for Agile Open Northwest. Led an improv open space. Got a tattoo. Met the founder of Voodoo donuts winding a clock. Such hipster. Very Portland. kitty gem Better Call Saul - new AMC show

What are some of the challenges traditional Microsoft IT Pro’s deal with moving to a more automated DevOps pattern? Jessica: Hard to tell which tools are really going to make their lives easier. Are the cultures of the companies benefiting the human side of the IT Pro? Jeffrey: Because Microsoft has great GUI tools, they become the biggest strength and weakness of the DevOps/IT-Pro The process of using a GUI is much harder to replicate in documentation. Because most of the community uses powershell commands, Microsoft IT-Pros really need to get on board. IT-Pros are never done with learning. If you don’t want to learn anything new, get into the lumber business. Microsoft has been making more open source integration moves, and changing philosophies to accept the Open Source community. “What up with that?” Jeffrey: “The body follows the head” It helps when you have a leader with a fresh approach who focuses on customer service and helping users within the community Jessica: It is really exciting to get behind a leader that is welcoming to the communities. It is refreshing to see Microsoft becoming a software company, not a “Windows software company” Microsoft wants you to be successful. Tools such as RESTful APIs are becoming available across all OSs. What is the acceptance level of the OpenSource movement within Microsoft? Jessica: Whatever you’re running, we can host it for you Traditional Configuration Management in Microsoft has been difficult. What are the plans? Steve Morowski (http://stevenmurawski.com/) has good info for those interested in DevOpsing with Windows. Current Microsoft tools are really good for enterprise, client management. Not so good for data center management. We need something different, that is simple, and usable. The problem is, everyone wants to do configuration their way. They want to be the CTO of their servers. Jeffrey describes the creation, and idea conception of a Microsoft Configuration Management platform that takes into account the deep differences between Linux, Unix, Windows. Describing different tools currently available, their faults, and how they might be able to connect them for modern, DevOps oriented, Configuration Management. The ability of chef and puppet, etc. are beneficial because of the ability of devs to pick it up, version it, and insert small parts of just what they need into the configuration. Jessica: We are getting to the point where Microsoft DevOps engineers are adapting the powershell. Until the powershell is adopted by IT-pros, modern DevOps tools will be a difficult push. You should already love powershell. How can people get more comfortable with powershell? Matt: It is not a scripting language. It is the way you interact with a system. Don’t write scripts in bash, write commands in bash that emulate the scripts. Jeffrey: Don Jones: Powershell in a Month of Lunches (http://morelunches.com/2011/04/01/learn-windows-powershell-in-a-month-of-lunches-1st-ed/) Step by step people get it, or they don’t. Managers really need to promote and reward the people giving you the IT that you want. Poweshell makes your environment repeatable, automatable, stable, etc. It is the future of the IT pro, and people must adopt it. Are we automating ourselves out of jobs? Jeff: The cloud is a great, cheap place to offer undifferentiated IT, however, if you can provide differentiated IT you are practically printing money vs. the cloud. Jessica: We do need a healthy fear. Not of automation though. Be scared of more interesting things. You need to learn automation. How do we work with Microsoft when its just not the best for DevOps-ing? As more people us Microsoft, the more Microsoft changes. Jeff discusses the many ways in which Microsoft is using flexible R&D to make a push for DevOps tooling, as well as some tools coming down the pipeline. Jessica: When choosing a tool, the longevity of the tool and the community around it is critical. Why can’t I copy a file to a server using WinRM? Jeff: Come talk to me at ‘Build and Ignite’. Transcript Checkouts Jessica The SoCal Linux Expo - Scale13 - a DevOps day Feb 20th - she has a discount code The Field Guide to Understanding Human Error (Dekker) Lean Enterprise book (Jez Humble) Jeffrey Hardcore History podcast -  I’m in love with this podcast. Dan Carlin is an awesome storyteller. Brain Science Podcast - (Cool podcast about the brain. I was just telling someone about this today) http://www.microsoftvirtualacademy.com/training-courses/getting-started-with-powershell-3-0-jump-start This is the start of a 2 day training session on using PowerShell. It is one of the most widely viewed jumpstarts ever. http://www.leeholmes.com/blog/2011/04/01/powershell-and-html5/ One of my all time favorite PowerShell scripts. Trevor FCC Ruling on broadband Windows 10 on Raspberry Pi 2 Matt Kitchen-windows is almost a thing! If you want to play with it, check out the Windows cookbook at http://github.com/opscode-cookbooks/windows BitTorrent Sync - http://www.getsync.com/ Yes, Please by Amy Poehler

Transcript Check-Outs Jill Beer: http://www.beeradvocate.com/beer/profile/45/680/ Newest obsession: Rowing class: http://www.cityrow.com/ (there are probably many others outside of NYC) Josh I’m officially the SysAdmin for Caustic Soda, which is a podcast about horrible things. It’s like Mythbuster for all that is gross and horrible. Now that I think about it, I can’t recommend it. Influxdb is an open-source, distributed, time series database with no external dependencies. Currently in Alpha (v0.8.8) it's going through a major refactor, the "production ready" version v0.9.0 is due out this month. The ease of installation and setup is what sold me on it. I think it has a lot of potential, so we'll see how v0.9.0 looks when it's released. The Bruery:  Briefly mentioned. Small batch, high quality beers out of Placentia, Ca. Their barrel aged and sour beers are outstanding. I recommend Loakal Red as the gateway beer for those that like IPAs. Mike http://www.opsschool.org Tubes: A Journey to the Center of the Internet, Andrew Blum The No Asshole Rule: Building a Civilized Workplace and Surviving One That Isn't Matt Elevate - brain training app Doing the fitbit thing again. We can be buddies. http://mattstratton.com/fitbit Trevor The Decemberists: What a Beautiful World, What a Terrible World. They're back! Microsoft reviews pull requests for C# from github Walking around your neighborhood. Discovered a new spice shop and comic shop by wandering North.

Dave is at Next Big Sound, which does analytics for creative industries, and he’s seen a few orgs handle failure well, and a lot of organizations handle it poorly. He got interested in blameless postmortems and human factors in discussions with John Allspaw of Etsy, and Allspaw influenced him to read the work of David Wood and Sidney Dekker on human factors. He is writing a book for O’Reilly called Being Blameless. MCR works at Etsy now, but has spent a lot of time consulting at various firms where he’s seen failure handled with blame. He points out what Rt. Lieutenant Colonel Scott Snook said in Friendly Fire, a book about when two US helicopters were accidentally shot down, that failure is part of complex systems. MCR: “I work at Etsy, and that’s what we do - we examine failure as a learning opportunity.” Dave is running his next workshop on Awesome Postmortems in NYC on February 12th, in which Dave: “Sidney Dekker’s Field Guide to Understanding Human Error is probably the most important book for people like us, meaning people that are in the IT world - it’s very accessible and gives lots of examples from fields outside of IT, but they’ve very relevant to what we do.” MCR: “Failure is gonna happen. It’s not a matter of if something is going to fail, it’s a matter of when it is going to fail.” MCR mentions the different categories of failures - those that “fail closed”, that are easy to detect, like disk filling up, and “fail open” - the surprises. He mentions some of the techniques Etsy uses - an IRC warroom, Vidyo video chatting, to resolve an immediate issue. After the immediate issue is solved, the learning begins. MCR: “We celebrate failure as much as we celebrate success here. […] The three-armed sweater is given to the person who most spectacularly impacted the website in the year.” On the topic of why to do a blameless postmortem, MCR points out that it’s for learning, and there are both technical and human factors. Dave points out that blaming a person short-circuits the learning. Claiming that a person is the cause of the outage feels like a good story, but it’s not true. Dave discusses root cause and mentions Allspaw’s excellent blog and a specific post about there being no such thing as a root cause, and Dave disagrees. He believes that outages are caused by change, and the systems with which we work are fundamentally changeable. “The impermanence of systems is the reason that they both function and malfunction.” Mike counters by saying, “Is there really a root cause for something that failed? If a hard drive dies, it’s the same hard drive. It hasn’t changed.” They both agree that it’s a philosophical rabbit hole. MCR notes that as Etsy grows, they’ve found that user-impacting, service-degrading issues are when they do postmortems, and even if not user-impacting, if they can learn from a failure it’s worth doing one. Dave says, “The more we learn about the complex systems within which we work, the better we’re able to operate them.” Within a week or two, according to Dave, is common practice of a time in which do the postmortem. MCR mentions that it’s important to write down the timeline almost immediately, definitely within a day or two, but doing it while someone’s amygdala is still triggered (and they are upset) is too soon. Dave points out that the facilitator of a postmortem sets the tone, including reminding people of hindsight bias, and at Next Big Sound they use a specific framework document which Dave will share. He also mentions defusing stress with empathy and humor. On the topic of evaluating anything you do, MCR mentions that Etsy created Morgue because any department across Etsy can apply these techniques to learn. Dave points out they do retrospectives as well as prospective review at Next Big Sound. MCR says Etsy does both an architectural review and an operability review ahead of time. Dave mentions that answers in prospective reviews can be biased in a positive way, whereas in a “premortem” we imagine things going badly, and try to determine what could lead to that: in essence, harnessing hindsight bias to work for us. Bridget forgets what decade it is and claims to have seen a presentation at devopsdays 2003. That would have been a nifty trick, since the first one was in 2009. :) Check Outs Dave: Field Guide to Understanding Human Error, Sidney Dekker - new edition just came out Dec 28th, 2014! Mike: http://www.bsr.org/en/our-network/member-list http://solutions.3m.com/wps/portal/3M/en_US/NA-DataCenters/DataCenters/Solutions/EfficiencySustainability/ImmersionCooling/ Etsy's postmortem tool Morgue Bridget: Did a lot of reading last week! Beside the aforementioned Dekker book, I read Web Operations (edited by John Allspaw and Jesse Robbins), Jon Cowie’s Customizing Chef, and the first three chapters of Jason Dixon’s Graphite book. All good stuff. The Boundary Waters Canoe Area is great in the winter too: I like Gunflint Lodge: http://www.gunflint.com/ Trevor: I was on vacation and delightfully disconnected. It’s been pretty awesome. Got a new Kindle and have been reading Game of Thrones before Matt accidentally (though at this point it’s my fault) spoils something. Set up kegbot at our new office, will be doing it’s grand opening later today :) Metrics about office beer / root beer consumption to come! Matt: Been on vacation, which is great. Doing the Dadops thing although I was sick for most of it, which was not delightful. However, I did see Big Hero 6, and Trevor was right about that. Book I’ve been listening to on audiobook is The Challenger Sale by Matthew Dixon and Brent Adamson http://www.amazon.com/The-Challenger-Sale-Customer-Conversation/dp/1591844355

It’s been a year of ADO, with topics from CI to security, panels of devs and of ops, and even Pete Cheslock. For the last episode of the year, we thought it would be fun to revisit Matt and Trevor chatting, sans guests - which hasn't happened since the first episode!

Data - we can’t have applications or services without it. If software is eating the world, then data is the maitre’d. However, it can be challenging to incorporate database design and release into the world of continuous delivery and devops. Grant Fritchey and Jonathan Hickford of Redgate join Matt and Trevor for a discussion of DevOps and Databases.

Transcript Checkouts Ducy The Goat Farm podcast Ross All the great talks from the Enterprise DevOps Summit, DOES14 Target Tech Blog (post on flashbuilds) Steve ALL THE TWEETS! Debian snapshots Amazon Lambda/CodeDeploy/Containers (in case it’s not covered in the AWS recap) My recap doc of DOES14, pretty rough but shareable Bridget devopsdays.org - for a devops near you, or to be inspired by talks at past ones so you can hold one inside your org Lots of great talks last week at DevOpsDays Vancouver, and one you should definitely watch is Stephanie Van Dyk, a Google SRE who worked on the healthcare.gov rescue. Trevor Worm Robot Big Hero 6 Barbiefail Matt Yak Shaving Expert t-shirt Customizing Chef by Jon Cowie Have you tried turning it on and off again supercut from the IT Crowd

Microsoft is open-sourcing .NET and creating the CLR for Mac and Linux .Net core5 is the new framework for 5, you can ship your own version of the app. There is now a free version of Visual Studio — Visual Studio Community for open source developers and students. In fact, it is all happening out in the open on github. Emma Jane is a long time listener of ADO! She has been teaching version control for many years with specific emphasis on the communication behind version control in teams. She has since switched to distributed version control such as git. Her aim in her teachings is to create resources that make git ”less painful” than it currently is. Distributed vs Centralized Version Control Emma Jane: In distributed VC the DB that contains the changes, exists on the local system and I can have multiple connections to multiple DBs with other versions. Centralized is all in a single DB, locally. How is Distributed VC relevant to DevOps? Matt: Many people hold the theory that you cannot have “The DevOps” without distributed version control. It implies communication through teams, so what is the validity of that statement. Emma Jane: Git is not the only VC option out there, but it is the most popular currently. You need to assess your team and your project, along with the related expertise and community support, and go with the one that fits your needs. As soon as you say “can’t” someone will prove you wrong. Testing Matt: The whole basis of git-flow is based on the fact that you can’t trust your contributors. Especially with open source. It sends a message that says “I don’t have to test my shit, because you’ll do it for me.” Emma Jane: If we are talking about testing, you need to have full coverage testing of whatever your product. Many testing frameworks allows for 99.9% accuracy on the tests, but that .1% causes you not to trust your tests. This makes it really hard to get reliable CI into the dev process. For that matter, Devs shouldn’t trust themselves when it comes to pushing code, you should always rely on testing because everyone is going to make mistakes, and humans might not catch them. Git allows you to have control over the pushed code. Trevor: There should be no permissions. All developers should have the same permissions and the flow should go through QA. How do you learn git? Emma Jane: All kinds of people are interested in learning git. But mainly: Someone who is on subversion and wants to change to git Someone who has been told to use git, but they don’t know how to run command line tools. CTO or management types that know they want to use git, but they’re not really sure where to go from that decision. In order to identify how your team will most efficiently use git, draw out your team flow and identify where efficiency is being blocked. Is re-basing causing problems? Is a PR sitting out there for too long? Use those as discussion points with your coworkers. You cannot introduce creativity when you are just told to memorize commands. Emma Jane: - Use Interactive Add! It allows you to split up your diffs into different commits. So you don’t end up committing a huge chunk of features that should most-definitely not be committed together. How do I get set up? Look for the right git-flow based on the type of deployments you are going to be using to release the software. Are your deployments feature based? or time based? How important is a rollback? Your code should always be deployable in a CD framework. You are only rolling forward, you have one master branch, and feature branches, how can you have correct and fast CD if you have multiple branches before the CD process starts. Your git setup should be directly related to your infrastructure. The git releases and flows of a team of 1 is going to be massively different than the git flow of a large team for a Could Provider. Things you should know (about git): Rebasing: Rebasing allows you to recombine how your commit chunks are strung together. It takes all the commits of a branch and Great for when you are adding too many commits. Git Bisect You can take out commits individually, and assess if the commits are in a working state. However, if you do not have full commits, for example, commits when you are just thinking about something, it will be much harder to assess the state of the commits individually. Source of Emma’s talk about Git: http://github.com/emmajane/gitforteams Post version: http://24ways.org/2013/git-for-grownups/ Recording: http://prague2013.drupal.org/session/git-makes-me-angry-inside Emma’s rant about storing the history of your project: http://gitforteams.com/resources/evolution-social-coding.html GitHub conversations: http://guides.github.com/introduction/flow/ Check-outs Emma Kaleidoscope mergetool -  because of image diffs Sketch training for techies -  because of impostor syndrome for drawing GNOME wins -  go open source! Matt Teamocil  - generator for tmux sessions The NoPhone  - kickstarter for a “technology-free alternative to constant hand-to-phone contact” “It’s not a promotion, it’s a career change”  - blog post by Lindsay Holmwood Trevor: Sandisk SSD Android L: Inbox Rosetta Probe