The Business of Open Source

In this episode of the Business Cloud Native, host Emily Omier talks with Jon Tirsen, who is engineering lead for storage at Cash App. This conversation focuses on Cash App’s cloud native journey, and how they are working to build an application that is more scalable, flexible, and easier to manage.The conversation covers:How the need for hybrid cloud services and uniform program models led Cash App to Kubernetes. Some of the major scaling issues that Cash App was facing. For example, the company needed to increase user capacity, and add new product lines. The process of trying to scale Cash App’s MySQL database, and the decision to split up their dataset into smaller parts that could run on different databases.Cash App’s monolithic application, which contains hundreds of thousands of lines of code — and why it’s becoming increasingly difficult to manage and grow. How Jon’s team is trying to balance product/ business and technical needs, and deliver value while rearchitecting their system to scale their operations.Why Cash App is working to build small, product-oriented teams, and a system where products can be executed and deployed at their own pace through the cloud. Jon also discusses some of the challenges that are preventing this from happening.How Cash App was able to help during the pandemic, by facilitating easy stimulus transfers through their service — and why it wouldn’t have been possible without a cloud native architecture. Links:Cash App: https://cash.app/Square: https://squareup.com/us/enJon on Twitter: https://twitter.com/tirsen?lang=enConnect with Jon on LinkedIn: https://www.linkedin.com/in/tirsen/?originalSubdomain=auThe Business of Cloud Native: http://thebusinessofcloudnative.com TranscriptAnnouncer: Welcome to The Business of Cloud Native podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. My name is Emily Omier, I'm here chatting with Jon Tirsen.Jon: Happy to be here. My name is, as you said, Jon Tirsen, and I work as the engineering lead of storage here at Cash App. I've been at Cash for maybe four or five years now. So, I've been with it from the very early days. And before Cash, I was doing a startup, that failed, for five years. So, it's a travel guide in the mobile phone startup. And before that, I was at Google working on another failed product called the Google Wave, which you might remember, and before that, it was a company called ThoughtWorks, which some of you probably know about as well.Emily: And in case people don't know, the Cash App is part of Square, right?Jon: Yes. Cash App is where we're separating all the different products quite a lot these days. So, it used to be called just Square Cash, but now it has its own branding and its own identity, and its own leadership, and everything. So, we're trying to call it an ecosystem of startups. So, each product line can run its business the way it wants to, to a large degree.Emily: And so, what do you actually spend your day doing?Jon: Most of my days, I'm still code, and doing various operational tasks, and setting up systems, and testing, and that sort of thing. I also, maybe about half my day, I spend on more management tasks, which is reviewing documents, writing documents, and talking to people trying to figure out our strategy and so on. So, maybe about half my time, I do real technical things, and then the other half I do more management stuff.Emily: Where would you say the cloud-native journey started for you?Jon: Well, so a lot of Square used to run on-premises. So, we had our own data centers and things. But especially for Cash App, since we've grown so quickly, it started getting slightly out of control. We were basically outgrowing—we could not physically put more machines into our data centers. So, we've started moving a lot of our services over to Amazon in this case, and we want to have a shared way of building services that would work both in the Cloud and also in our data centers. So, something like Kubernetes and all the tools around that would give us a more uniform programming model that we could use to deploy apps in both of these environments. We started that, two, three years ago. We started looking at moving our workload out of our data centers.Emily: What were the issues that you were encountering? Give me a little bit more details about the scaling issues that we were talking about.Jon: There two dimensions that we needed to scale out the Cash App, sort of, system slash [unintelligible] architecture. So, one thing was that we just grew so quickly that we needed to be able to increase capacity. So, that was across the board. So, from databases to application servers, and bandwidth, everywhere. We need to just be able to increase our capacity of handling more users, but also we were trying to grow our product as well. So, at the same time, we also want to build and be able to add new features at an increased pace. So, we want to be able to add new product lines in the Cash App. So, for example, we built the Cash Card, which is a way you can keep your money in the Cash App bank accounts, and then you can spend that money using a separate card, and then we add a new functionality around that card, and so on. So, we also needed to be able to scale out the team to be able to have more people working on the team to build new products for our users, for our customers. Those are the two dimensions: we needed to scale out the system, but we also needed to have more people be able to work productively. So, that's why we started trying to chop up—we have this big monolith as most companies probably do, which that's I don't know how many hundreds of thousands of lines of code in there. But we also wanted to move things out of that, to be able to have more people contribute productively.Emily: And where are you in that process?Jon: Well, [laughs], we're probably adding still adding code at an exponential rate to the monolith. We're also adding code at an exponential rate outside of the monolith, but it just feels so much easier to just build some code in the monolith than it is outside of it, unfortunately, which something we're trying to fix, but it's very hard. And it is getting a little bit out of hand, this monolith now. So, we have, sort of, a moratorium on adding new code to the monolith now, and I'm not sure how much of an effect that has made. But the monolith is still growing, as well as our non-monolith services as well, of course. Emily: When you were faced with this scaling issue, what were the conversations happening between the technical side and the business owners? And how is this decision made about the best way to solve this problem is x, is the Cloud, is cloud-native architecture?<...

Emily and Dejan cover the following points:8x8’s journey to a leading cloud technology provider.Why 8x8 decided to migrate to Kubernetes, a move that gave them the flexibility to run workloads wherever they want.Dejan’s thoughts on the Kubernetes migration, and how it’s helped the company improve its operations. For example, Kubernetes has helped 8x8 migrate away from several legacy systems.The biggest challenges and surprises that the 8x8 team experienced during their migration journey, such as getting engineering teams to embrace a culture built around monitoring, observability, and documentation.How 8x8 has avoided “feature bloat” and maintained a product that performs at a high level, while staying true to the features that are important for its core customer base. The strategy of obtaining buy-in from stakeholders and fellow executives by focusing on business problems, instead of technical issues. This included cost, velocity of innovation, global scale, and so on.How 8x8’s cloud-native architecture has made it faster and easier to scale. TranscriptAnnouncer: Welcome to The Business of Cloud Native podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. I'm Emily Omier, and I am talking with Dejan Deklich, from 8x8.Dejan: So, I'm the Chief Product Officer at 8x8. To give you an idea, 8x8 is now 16 or 1700 employees worldwide, 450 million in revenue, give or take, offices all over the world, customers all over the world. I'm responsible for all product management, engineering, QA, project management operations for all the products worldwide for 8x8.Emily: Can you give me a little bit of an idea of 8x8’s history in the Cloud?Dejan: So, 8x8 has been around, probably, a lot longer than most companies you're talking about. We've been public 30 years, give or take. We have been in the business of communication and collaboration since early 2000s. As you can imagine, we have gone through so many different tech stacks, architectures, and so on, that it is pretty amazing. We have, in the last several years, done a massive cleanup and rebuild of our software stack. We rebuilt pretty much all of the mobile apps, desktop apps, web apps. We rebuilt the platform starting with billing and provisioning all the way down to how the voice traverses the world. So, it's been a incredible couple of years, incredible journey where I would argue we have gone from the early versions of hosted service to early versions of Cloud, maybe 10 years ago, and we are now what I would like to call a proper cloud technology company. And it's been a very interesting, difficult journey. We learned a lot. We messed up a lot of things, then we learned some more than they did it correctly.Emily: When you first moved to Kubernetes, and the modern public cloud, what was the rationale? What were their business reasons?Dejan: Those multiple steps there. We moved to public cloud I don't know, five, six, seven years ago. We ran a lot of things in Amazon. And to be fair, we still also have data centers around the world. So, let me explain quickly what we actually running because I think it's important. So, we have, I think 16 data centers around the world, and then we run in pretty much every region of Amazon, we use Google Cloud extensively, and we have now shifted a lot of workloads to Oracle Cloud. At the same time, business is threatening me with Alibaba Cloud and Tencent Cloud as something that might be coming our way in the next couple of quarters. So, data centers are there because on the networking layer, the Cloud does not yet give us what we need for the realtime voice and video transmission. We actually are the best voice provider in the industry. We have proven that, and that's where your milliseconds really matter, therefore networking still sits in data centers. As soon as the backbone can be moved into Amazon, and we are told that could happen in the next three to four years, we will move likely everything to the Cloud. So, what we have generally in the Cloud are different applications, and the reason for that is simply the velocity of deploying and scaling them. So, what matters to us is, on one hand, the global reach: we have customers in 150 countries around the world. We have to have data centers close to the customers. And the applications need to be as close to the customer as possible, therefore all the different regions of Amazon, and Google, and whatnot. So, as you can imagine, managing all of that, monitoring all of that is a non-trivial exercise. So, we moved to Kubernetes, in large reason, simply because it is one underlying framework that allows us to run workloads wherever we want. So, to give you an idea, we launched a video meetings product to compete with Zoom. We had, on launch, a couple of hundred thousand users, nothing really. And then, this COVID-19 happened, and within a period of weeks, we now hit 15 million users. The only way you can scale a system like that is if you have a properly built underlying architecture, everything horizontally scalable. I was blown away, everything really worked. People were super busy, but by having proper cloud architecture, we were able to actually scale, and fulfill the demand that we have seen worldwide. Now, the nice thing is, as you put more and more workloads on top of Kubernetes, you can shift them between clouds as you want, or data centers as you want. And I think that's number one reason why we went with Kubernetes. I love Amazon, I love Google, and nothing makes me happier than writing them a million-dollar checks, but I also want to be able to move the workloads wherever I can run them cheaply. And, to me, that's very important. I don't have unlimited budget; I have to be able to play the game and get the most compute and the most bandwidth for the lowest cost that I can, and Kubernetes lets me do that.Emily: And would you say that Kubernetes was a technical decision or a business decision or both?Dejan: That's a good question. I think normally, the way we operate at 8x8, you start with the business problem. The business problem was we don't want to be locked into one cloud. We want to be able to run wherever we want to run, and on top of that, we have customers in Europe who are not very friendly towards Amazon, and want us to run on other clouds. And then, we took a peek: what can we do? What's the fastest and easiest way to do it? Turned out it was Kubernetes, so that's the way we went.Emily: What did the move to Kubernetes, what was it like? What were some of the surprises?Dejan: It was very interesting. It is still very interesting. So, on one hand, the good thing was we have already broken the monoliths in the past God knows how many years, into services. But to get things running properly in Kubernetes, you have to go a bit deeper, you actually have to really clean up your code, and so on, and so on. So, one thing that I thought was incredibly useful was this allowed us to, for the first time in 8x8 history, create a proper template for a service where all yo...

Some of the highlights of the show includeThe diplomacy that’s required between software engineers and management, and why influence is needed to move projects forward to completion.Driving factors behind Ygrene’s Kubernetes migration, which included an infrastructure bottleneck, a need to streamline deployment, and a desire to leverage their internal team of cloud experts.Management’s request to ship code faster, and why it was important to the organization. How the company’s engineers responded to the request to ship code faster, and overcame disconnects with management.How the team obtained executive buy-in for a Kubernetes migration.Key cultural changes that were required to make the migration to Kubernetes successful.How unexpected challenges forced the team to learn the “depths of Kubernetes,” and how it helped with root cause analysis.Why the transition to Kubernetes was a success, enabling the team to ship code faster, deliver more value, secure more customers, and drive more revenue. Links:HerdX: https://www.herdx.com/Ygrene: https://ygrene.com/Austin Twitter: https://twitter.com/_austbotAustin LinkedIn: https://www.linkedin.com/in/austbot/Arnold’s book on publisher site: https://www.packtpub.com/cloud-networking/the-kubernetes-workshop Arnold’s book on Amazon: https://www.amazon.com/Kubernetes-Workshop-Interactive-Approach-Learning/dp/1838820752/TranscriptAnnouncer: Welcome to The Business of Cloud Native podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. My name is Emily Omier, and I am here with Austin Adams and Zack Arnold, and we are here to talk about why companies go cloud-native.Austin: So, I'm currently the CTO of a small Agrotech startup called HerdX. And that means I spend my days designing software, designing architecture for how distributed systems talk, and also leading teams of engineers to build proof-of-concepts and then production systems as they take over the projects that I've designed. Emily: And then, what did you do at Ygrene? Austin: I did the exact same thing, except for without the CTO title. And I also had other higher-level engineers working with me at Ygrene. So, we made a lot of technical decisions together. We all migrated to Kubernetes together, and Zack was a chief proponent of that, especially with the culture change. So, I focused on the designing software that teams of implementation engineers could take over and actually build out for the long run. And I think Zack really focused on—oh, I'll let Zack say what he focused on. [laughs].Emily: Go for it, Zach.Zach: Hello. I'm Zack. I also no longer work for Ygrene, although I have a lot of admiration and respect for the people who do. It was a fantastic company. So, Austin called me up a while back and asked me to think about participating in a DevOps engineering role at Ygrene. And he sort of said at the outset, we don't really know what it looks like, and we're pretty sure that we just created a position out of a culture, but would you be willing to embody it? And up until this point, I'd had cloud experience, and I had had software engineering experience, but I didn't really spend a ton of time focused on the actual movement of software from developer’s laptops to production with as few hiccups, and as many tests, and as much safety as possible in between. So, I always told people the role felt like it was three parts. It was part IT automation expert, part software engineer, and then part diplomat. And the diplomacy was mostly in between people who are more operations focused. So, support engineers, project managers, and people who were on-call day in and day out, and being a go-between higher levels of management and software engineers themselves because there's this awkward, coordinated motion that has to really happen at a fine-grained level in order to get DevOps to really work at a company. What I mean by that is, essentially, Dev and Ops seem to on the surface have opposing goals, the operation staff, it’s job is to maintain stability, and the development side’s job is to introduce change, which invariably introduces instability. So, that dichotomy means that being able to simultaneously satisfy both desires is really a goal of DevOps, but it's difficult to achieve at an organizational level without dealing with some pretty critical cultural components. So, what do I spend my day on? The answer to that question is, yes. It really depends on the day. Sometimes it's cloud engineers. Sometimes it's QA folks, sometimes it's management. Sometimes I'm heads-down writing software for integrations in between tools. And every now and again, I get to contribute to open-source. So, a lot of different actual daily tasks take place in my position.Emily: Tell me a little bit more about this diplomacy between software engineers and management.Zach: [laughs]. Well, I'm not sure who's going to be listening in this amazing audience of ours, but I assume, because people are human, that they have capital O-pinions about how things should work, especially as it pertains to either software development lifecycle, the ITIL process of introducing change into a datacenter, into a cloud environment, compliance, security. There's lots of, I'll call them thought frameworks that have a very narrow focus on how we should be doing something with respect to software. So, diplomacy is the—well, I guess in true statecraft, it's being able to work in between countries. But in this particular case, diplomacy is using relational equity or influence, to be able to have every group achieve a common and shared purpose. At the end of the day, in most companies the goal is actually to be able to produce a product that people would want to pay for, and we can do so as quickly and as efficiently as possible. To do that, though, it again requires a lot of people with differing goals to work together towards that shared purpose. So, the diplomacy looks like, aside from just having way too many meetings, it actually looks like being able to communicate other thought frameworks to different stakeholders and being able to synthesize all of the different narrow-focused frameworks into a common shared, overarching process. So, I'll give you a concrete example because it feels like I just spewed a bunch of buzzwords. A concrete example would be, let's say in the common feature that's being delivered for ABC Company, for this feature it requires X number of hours of software development; X number of hours of testing; X number of hours of preparing, either capacity planning, or fleet size recommendations, or some form of operational pre-work; and then the actual deployment, and running, and monitoring. So, in the company that I currently work for, we just...

Some highlights of the show includeThe challenges of operating digital commerce at scale, including the need for resource pooling and resiliency — and how this caused Ant Financial to re-think their infrastructure. Ant Financial’s former approach to scaling, which was mostly manual, and highly resource-intensive. How Kubernetes is expediting cloud development for Ant Financial.Haojie’s thoughts on the global engineering skills gap, and China’s growing cloud computing market including driving factors and barriers. Why Ant Financial’s migration has largely been a success — and why achieving operational security is now a top priority for the company.  How Ant Financial is managing disconnect between its engineers and business leaders. The company’s ongoing mission to migrate its systems and applications away from legacy architectures.LinksLinkedIn: https://www.linkedin.com/in/haojiehang/https://www.investopedia.com/tech/worlds-top-10-fintech-companies-baba/TranscriptAnnouncer: Welcome to The Business of Cloud Native podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: So, I always start the same way. Can you introduce yourself?Haojie: Hey, my name is Haojie Hang. I'm a product manager in the CTO office at Ant Financial. I work on the product and strategy side for, basically, the CTO and the other executive leaders, as well as leading a small product teams within the org to look at the frontier technology in the cloud and other infrastructure businesses.Emily: And can you tell me a little bit more about what Ant Financial does? And then, also, what do you do on a day to day basis? What do you do when you get into the office?Haojie: Yeah, I'll do a quick introduction about the Ant Financial business. It's not just one business or two business, it's a group of businesses that we innovate and we do, mostly in China, but we're also expanding very rapidly all over the world. So, Ant Financial is basically a group of businesses including credit for both consumers and the enterprise, as well as loan businesses, both consumer and enterprise businesses. We say that the parent organization is basically, we call it Alipay, it’s the earliest business we do since 2004 when the business was basically born from Taobao, which is our parent company. So, in short, the Ant Financial Business has a lot of presence in the business of payments business, remittance, credit card, loans, securities, and many other businesses like intelligent technology, blockchain, pretty much everything you can imagine in the FinTech and financial services, we’re in there.Emily: Tell me a little bit more about the cloud-native journey for Ant Financial. When did it start? Why did it start? What was some of the motivations behind moving to cloud-native?Haojie: Yeah, it's actually quite interesting. I joined Ant Financial in 2008, but actually, the entire company started to look at cloud-native technology quite early, in 2012. So, back then, people were just looking at these technologies around the world, mostly from the US, they look at this open-source community, look at what other companies are doing, how to use the cloud-native technology to help with their business in the peak time, so during event. There’s online promotion event we're doing every year, called Double 11—Shuāng shíyī in Chinese. Every year, so we have a large amount of promotional events happening online, trying to help merchants and the customer is trying to sell and buy stuff in our Tmall and Taobao platform in very, very discounted price. So, for that promotion event online, we have to think about the resilience, the resource pooling, oftentimes the visits has to increase multiple times, sometimes over 100 times the increase compared to the normal time. So in that case, we have to think about how we can be very resilient and efficient infrastructure to support that business needs. So, this is a very large topic. And then, back then, there was a lot of focus and study in our cloud computing department. So, we started looking at this technology called Mesos in 2012. And then, we do a lot of experiments around this technology, but from the business perspective, it's still hard to justify the benefits of moving to Mesos completely. So, we have multiple teams doing a lot of research in Mesos, in Kubernetes, sometimes in our own technology stack, but there's not enough proof or enough confidence for us to move completely over to that technology, until the emergence of Docker container, this Docker technology. Then we started to look at our container infrastructure, really do the investigation around this technology, and understand why this is taking over so quickly over the world, from the business perspective, and from the technology perspective. If you look at the community of Docker, the thing does not really happen until 2015. But we are already in the game for about a year or two. So, we're actually quite happy about our original strategy, but it's just in terms of the research. We're actually a little bit behind in terms of moving to this cloud-native architecture. But as you can see, that I had an interview with CNCF. So, we are very happy about the results that we have right now. Pretty much the entire architecture we run within Ant Financial is, basically, on Kubernetes ecosystem. It's not just using the open-source version of it. We're doing a lot of customization around this open-source framework. Yeah, I can talk more about the details.Emily: Yeah. Well, let's back up just a little bit. I’m curious what you were doing to manage this scaling before? And how did that change? And what about the whole process changed? Like, how stressful is it now, compared to before?Haojie: The process was very manual, I would say. We have extremely large team of engineers, and DevOps, security teams. And oftentimes their responsibility are overlap. So, some engineers are doing security work, some engineers are doing basically operational work. I would say, some people really hated it because they have to be on the computer, look at monitor 24/7, making sure transactions succeeded. When the peak time happens, there's nothing wrong with it. Sometimes they have to keep their phone open 24/7, basically to make sure this thing will not fail, right? And then, just many parts of work has to—so in the previous way, the way we do this operation is quite manual. We don't have a mature system or methodology telling us what we should do first, we should do second, and what's what would you do after this. So, basically the collaboration chain was not there. Therefore, when issue happens, our operation team has to respond very quickly. But then, how can we quickly identify the problem, and make it a problem? That's a problem, right? So, we have to make sure every time we respond, we respond in a very effective manner. That's the problem. In the previous process when something unexpected happen, who had to engage with the entire team from product, engineering, operation, security, everybody has to get up and look at the problem together, which was quite inefficient. So, after we moved to this cloud-native architecture—it's not the standard cloud architect, it's, kind of—we have a lot of innovation on...

Some of the highlights of the show includeHow containerization enabled Nav to spread roughly 250 virtual machines across multiple environments, while drastically reducing infrastructure spendTravis’s thoughts on buying cloud native software tools versus building them, and what engineers should consider during this processThe difficulty of finding security solutions that work inside of a cloud-native ecosystemWhy companies should expect to encounter unique challenges when migrating to KubernetesWhy companies need to understand their end goal, and determine an overall objective before beginning a migrationTravis’s must-have engineering tool, and why he can’t live without it LinksLinkedIn: https://www.linkedin.com/in/stmpy/Twitter: https://twitter.com/stmpy TranscriptAnnouncer: Welcome to The Business of Cloud Native Podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. I’m Emily Omier, your host. And today I’m here with Travis Jeppson. Travis is currently at Kasten, but he’s also going to talk about his time as a director of engineering at Nav.Travis: At Nav, my role shifted quite a bit while I was there. I started as a software developer, writing Ruby back end applications for them, and then shifted into—actually within a month of being there, they shifted me over to the operational side because I had previous experience working with containerization, and also in infrastructure. So, they quickly moved me over into that realm and from there, I worked there for about a year until they told me, go spin up a team and get things moving. Help us move to containerization. Help us move to a more modern infrastructure and stuff. And so, about a year after that I became a director of engineering to where I had our ops team that had spun up, and then I also acquired both our QA team and our IT team that was there. And then, about a year after that, I ended up acquiring a little bit more than that. So, I ended up with a fair amount of our front end and some of our backend teams as well, and where they moved me into the senior director position. So, a day in the life, towards the end of when I was at Nav was a lot of working with the teams, helping them to do a lot of architectural perspective, and changes, and outlook to where we were trying to get as far as the company is concerned. We were building a product that we could address both first-party customers where they would log in to the Nav website directly, as well as working with partners so that we could issue out Nav functionality to those partners that they could incorporate to their pages as well. And so, we worked very hard to try to segment those two pieces together so that what we were building could be dispersed between both first-party customers and our third-party customers. And so, towards the end of my time there, it ended up being a lot of working within all of engineering to help facilitate those purposes. Then, just about six months ago, I ended up shifting my role over to a company called Kasten. And, Kasten is strictly working within the Kubernetes ecosystem. So, we do data management for Kubernetes based applications, and I am the site lead in Utah for Kasten, and so my day in and day out, a lot is, it's, kind of, all over the place. Sometimes it's working with engineering to help figure out some things going on there, sometimes it's working with brokers to help find office space for it. And sometimes it's dealing with insurance. It ended up being quite dynamic. But overall, I'd say most of my time is really spent more on the engineering side, just from the perspective of having worked at Nav and having been a consumer of a lot of these technologies, I think that they really appreciate my insights that I'm able to give there. So, I end up working, a lot, with the engineers to help facilitate what we're doing.Emily: Sounds like you end up serving as a bridge from having been an end-user. But do you think that there is common miscommunications that happen, or what do those conversations sound like? Why is that experience valuable?Travis: Yeah, so I don't know if it's as much as a miscommunication as much as what are customers looking for? And what are they trying to achieve? And why are they purchasing different software solutions? And what makes sense for them, more than anything. And I think that, having been a consumer of those products, I was more or less on the front lines there. When I was building our operational team at Nav, that was basically what I was doing is trying to figure out what things are we going to spend time on? And what things are we going to build ourselves, or what things do we need to just go find a solution for and bring them in-house? And the funny thing is when I was doing that for Nav is actually when I was introduced to Kasten and to the CEO here. And so, that ended up changing the way my career went. But overall, I think what Kasten—what those conversations really end up becoming is what are customers trying to do, and where are they trying to go?Emily: Yeah, and in fact, that is exactly what I want to talk about more on this podcast. So, tell me a little bit about what your experience at Nav was. What were you looking for? What did you want to prioritize? What was the company hoping to get out of moving to containers?Travis: So, I would say maybe the piece that really facilitated a lot of the progress in that sense was starting to understand our infrastructure spend. And then, to couple with that was also trying to become more agile. More agile in the sense of being able to push on demand, where previous to that we were pushing—you know, when we push our code, we did it on a bi-weekly basis—well, every other week, and it was always very cumbersome. If we have pictures of us in the early days of Nav, where there would be 10 engineers around someone’s desk, and they were the one person that was pushing the code into production, just waiting for the other shoe to fall, or waiting for something to happen. And so, when I started doing operational things for Nav, it started addressing those two things. What can we do to help control our infrastructure, and to understand it a little bit better? And how can we also create more of a dynamic infrastructure? Like, Nav is very much a US-based company. And so, the traffic that we're getting onto our website was regional very, very much. And so, there would be periods where it would be very busy, and then there'd be periods where it wasn’t. And the way that our infrastructure was designed, and a lot of times the way that they are designed, especially with virtual machines, is that you're building for capacity. You're building to be able to handle that load, and that has to stay there all the time, regardless of whether that capacity is being used or not. And so, that was one of the biggest questions, and that bill was—we were completely in the clouds. We were completely in AWS, but that bill continued to get more and more expensive every month. To the point of where it warranted the executive team to come down and say, “This needs to be fixed. This is going at an outrageous pace, and we need to be able to figure out how to control this.” And so, that's when they came to me and said, “Okay, get a team spun up, and let's figure out how to control this.” And so, I would say that those wer...

Some of the highlights of the show includeWhy Adform decided to move to a cloud native architecture and Kubernetes specifically Who was the driving force behind the move to Kubernetes?Was the switch purely an engineering decision or did it involve people outside of engineering?Positive and less positive surprises that come with switching to cloud native Organizational and technical problems Edgaras has facedWhat’s next for Adform on their cloud journeyLinksLinkedIn: https://www.linkedin.com/in/apsega/Twitter: https://twitter.com/ApsegaTranscriptAnnouncer: Welcome to The Business of Cloud Native Podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. I’m Emily Omier, your host. And I’m here today with Edgaras Apsega, lead IT systems engineer at AdForm. Edgaras, what I’d like to do is just start out with you introducing yourself.Edgaras: I’m Edgaras. I’m working in the Adform. For anyone that doesn't know, Adform is one of the leading advertising technology companies in the world, and provides the software used by buyers and sellers to automate digital advertising. And, probably one of the most interesting parts of our solution stack is demand-side platform that has real-time bidding. And, what it means is that when that page is loading for some kind of internet users, behind the curtain, there's actually a bidding process that takes place for the placeholders to show ads. So, basically, you're doing low latency stuff. And, in Adform, I'm a lead systems engineer for the cloud services team. Our team consists of eight people, and we are providing private cloud storage, load balancing, CDN, service discovery and Kubernetes platforms for our developers that are in [00:01:36 unintelligible] production services.  So, to better understand the scale that our team is working on, first of all, you can see that we are not using public cloud and we have our own private cloud that has six regions, more than 1500 physical servers, and there are more than 4000 [00:01:55 unintelligible]. And, for Kubernetes, we have seven clusters, more than 50 physical machines and around 300 constantly running [00:02:05 pods].  So, we can say that we prefer bigger clusters with bigger resources sharing pools. And you asked, how do I spend my daily work, right?Emily: Yeah. So, when you get into the office or—right now you're not going into the office—get into your table or your [laughs] home office, what are the first couple things that you do, or…Edgaras: Yeah, so, when I arrive at work, or, like, at these times, just get off the showers straight into work desk, [laughs] actually, I'm most productive in the mornings and evenings. So, in the mornings, when I go to my work desk, I try to do as much as I can. My sprint plan tasks, and then I scroll through the Slacks, emails, and the tickets assigned to me because we have a development team in another region. So, instantly in the mornings, we have some kinds of support tasks that we need to do.Emily: Let's go ahead and talk about what this is all about, the business of cloud native, and tell me a little bit about why Adform decided to move to a cloud native architecture. Why did you decide to use Kubernetes, for example?Edgaras: I'd say, actually, there were two parts. At first, we moved from traditional and, let's say, old-fashioned monitoring solutions to Prometheus, and its integration with service discovery solved lots of operational time for constantly managing and configuring monitoring and alerting for our, quite often, changing infrastructure. And the second part is the adoption of Kubernetes and all of the together coming parts like continuous integration and delivery. So, why we moved to this kind of architecture? It was because the biggest pain points for developers were to maintain actually their virtual machines. And rolling out new software releases in an old-fashioned way, took just lots of time for new software releases to reach production. So, we were looking at the new solutions that were available in the market, and Kubernetes was actually one of them. So, after successful proof of concept, we have selected it as our main application scheduler and orchestration tool.Emily: What would you say was, like, the business value that you were hoping to get out of Kubernetes, out have the ability to release software faster, for example?Edgaras: Yeah.  So, actually, we wanted to remove the operational time from our developers so that they could spend more time coding without taking care of all of the infrastructure surrounding parts, like the application operating system management, [00:04:58 unintelligible] monitoring, alerting, logging, and so on. So, basically what, I'm saying is that the business value was for the developers to be able to ship features faster, and have a more stable platform that scales application [00:05:15 unintelligible] as well. So, in addition to that, we have a big research department, and the research department always wanted us to have a dynamic environment where they could just launch an applications around some research models, and then shut it down.  So, I believe that was the business value.Emily: Who in the organization do you think was motivating, or driving the move to Kubernetes?Edgaras: I'd say, actually, it was more like the operation engineers, because the developers ended taking care of their environment virtual machines. They don't know much about it, but they still have to look after it, and constantly asking us for help. And we wanted to have this operational stuff only in our hands and for the developers to run only the code.  So, I believe, yeah.Emily: To what extent was the move to Kubernetes, or to cloud native in general, just purely an engineering decision? Or did it involve other people outside of engineering?Edgaras: Well, it wasn't only the engineering decision, because we had to take it to the upper levels, just to show this new cloud native, the modern way of developing and running applications. So, the upper management level had to invest time for us to move to microservices oriented architecture and so on. So, basically, we had to show that with a little bit of time investment we can gain lots of benefits, like faster code deploys. So, we are taking the operational work from developers, and developers, when they're releasing their applications, they have full stack monitoring, logging, and they don't need to do any of the operational tasks.Emily: How difficult was it to have this conversation? Do you feel like the upper management, did they understand the value?Edgaras: Yeah, it was kind of hard, because nobody wants to invest time to write the code. And, as we are a software company, we always need to write new features. But, once we showed a good example, when investing not so much time, we have those kinds of benefits, then it was quite easy to change the mindset of upper management.Emily: And, how important do you think this was for Adform?Edgaras: I think it was very im...

About Emily OmierEmily Omier is a content strategy consultant who helps companies leverage content to build thought leadership, increase website traffic, grow their mailing list and book more demos. She has worked with CloudBees, Portworx, Plutora, Armory, and is a regular contributor for The New Stack. She graduated from the Columbia University Graduate School of Journalism and lives in Portland, Oregon.