Cloud Engineering Archives - Software Engineering Daily

Kubernetes is a cluster management tool open sourced by Google. On Software Engineering Daily, we’ve done numerous shows on how Kubernetes works in theory. Today’s episode is a case study in how to deploy Kubernetes to production at a company with existing infrastructure.   GolfNow is a fifteen year-old application written in C# .NET. It is a successful, growing business that is a division of NBC Sports. As GolfNow has grown, it has encountered scalability issues, and the engineering team at GolfNow decided to move its entire monolithic infrastructure to microservices running in Docker containers, managed by Kubernetes. Sheriff Mohamed joins the show today to discuss migrating his company’s application to Kubernetes. It’s a great show for anyone who is moving a large team to Kubernetes, or considering the technology for their application. The post Kubernetes Migration with Sheriff Mohamed appeared first on Software Engineering Daily.

In a microservices architecture, a user request will often make its way through several different services before it returns a result to the end user. If a user experiences a failed request, the root cause could be in any of the services along that request path. Even more problematic is the challenge of debugging latency in this kind of request chain.   Reshmi Krishna joins the show today to discuss distributed tracing, the process of tracking the path of a request through multiple services in order to determine the root cause of latency or errors. A popular tool for distributed tracing is Zipkin, which is largely based off of a paper published by Google called “Dapper”.   Reshmi is also speaking at the upcoming O’Reilly Velocity Conference in New York, September 20-22, so check that out if you are interested in web performance, continuous delivery, or anything else related to web and mobile development. The post Distributed Tracing with Reshmi Krishna appeared first on Software Engineering Daily.

“Serverless” usually refers to an architectural pattern where the server side logic is run in stateless compute containers that are event-triggered and ephemeral. Mike Roberts has written a series of articles about serverless computing, in which he discusses theories and patterns around serverless architecture. In this episode, Mike and I discuss how to reimagine our software architecture using functions-as-a-service. We go into the costs, benefits, and modern limitations of current serverless platforms like AWS Lambda.   The post Serverless Architecture with Mike Roberts appeared first on Software Engineering Daily.

Unbounded data streams create difficult challenges for our application architectures. The data never stops coming, and we are forced to assume that we will never know if or when we have seen all of our data. Some streaming systems give us the tools to deal partially with unbounded data streams, but we have to complement those streaming systems with batch processing, in a technique known as the Lambda Architecture. Apache Beam is a unified model for defining and executing data processing workflows, and Frances Perry joins the show to explain how Beam provides a way for us to model our data processing, agnostic of whether we choose to run those workflows on Spark, Flink, or Google’s Dataflow. Links Apache Beam Streaming 101 Streaming 102 The Dataflow Model Google Cloud Dataflow Fundamentals of Stream Processing with Beam Mobile Gaming Example Dataflow: Beam and Spark Comparison The post Apache Beam with Frances Perry appeared first on Software Engineering Daily.

Prometheus is a tool for monitoring our distributed applications. It allows us to focus on the services we are deploying rather than the individual machines that make up instances of that service.   The monitoring service itself is a portion of a distributed system that is treated differently than the services we are monitoring. We don’t want to use a consensus-based tool like Zookeeper or Consul because we can’t afford the strong consistency.   Brian Brazil’s company Robust Perception is built around Prometheus, and he joins the show to discuss why and how to use Prometheus. The post Prometheus Monitoring with Brian Brazil appeared first on Software Engineering Daily.

Monitoring translates machine data into actionable business metrics, and is a key component of a modern software company. James Turnbull’s new book “The Art of Monitoring” describes how organizations can build their monitoring infrastructure.     James joins the show today to outline the strategies that a company can use to proactively monitor their systems. We talk about pull- vs. push-based monitoring, events, logging, metrics, and James’s experience as CTO of Kickstarter. The post The Art of Monitoring with James Turnbull appeared first on Software Engineering Daily.

Lee Atchison spent seven years at Amazon working in retail, software distribution, and Amazon Web Services. He then moved to New Relic, where he has spent four years scaling the company’s internal architecture. From his decade of experience at fast growing web technology companies, Lee has written the book Architecting for Scale, from O’Reilly. As an application scales, it becomes significantly more complicated while at the same time receiving more traffic. The intersection of these two problems leads to a variety of discussions around availability, risk management, and microservices. Lee and I didn’t have time to get through everything in his book Architecting for Scale, but if you enjoy this episode, check out the book. Lee also spoke recently at the O’Reilly Velocity conference in Santa Clara, so you can check out his talk. We are giving away a free ticket to O’Reilly’s Velocity 2016 conference in New York. If you want to be entered to win that ticket, send a tweet about your favorite SE Daily episode about dev ops or web performance, and tag software_daily, as well as #velocityconf. The post Scalable Architecture with Lee Atchison appeared first on Software Engineering Daily.

Scheduling is the method by which work is assigned to resources to complete that work. At the operating system level, this can mean scheduling of threads and processes. At the data center level, this can mean scheduling Hadoop jobs or other workflows that require the orchestration of a network of computers. Adrian Cockcroft worked on scheduling at Sun Microsystems, eBay, and Netflix. In each of these environments, the nature of what was being scheduled was different, but the goals of the scheduling algorithms were analogous–throughput, response time, and cache affinity are relevant in different ways at each layer of the stack. Adrian is well-known for helping bring Netflix onto Amazon Web Services, and I recommend watching the numerous YouTube videos of Adrian talking about that transformation. The post Schedulers with Adrian Cockcroft appeared first on Software Engineering Daily.

In 1999, it took $50,000 to buy a server. Once you bought that server, you had to know how to operate and maintain it. Today, cloud service providers have changed how we build software. Servers, load balancers, networking, storage–these hardware concerns have been turned into software. Don Pezet joins the show today to discuss the fundamentals of a cloud service provider. These are the basics that you need to know about building and maintaining your application in the cloud. Don is a host of IT Pro TV, a company that makes training resources for engineers and operators. The post Cloud Providers with Don Pezet appeared first on Software Engineering Daily.

Six years ago, Twitter was experiencing outages due to high traffic. Back in 2010 Twitter was built as a monolithic Ruby on Rails application. Twitter migrated to a microservices architecture to fix these problems. During this migration, the engineers at Twitter learned how to build and scale highly distributed microservice architectures. William Morgan was an engineer at Twitter during that time, and he is now the CEO of Buoyant.io, a company building open-source microservices infrastructure. Some of the big problems at Twitter were solved at the communication layer, using an RPC library called Finagle. At Buoyant, those lessons are being applied to a project called Linkerd, an RPC proxy. The post Scaling Twitter with Buoyant.io’s William Morgan appeared first on Software Engineering Daily.