Cloud Engineering Archives - Software Engineering Daily

Rust is a systems programming language with a distinct set of features for safety and concurrency. In previous shows about Rust, we explored how Rust can prevent crashes and eliminate data races through its approach to type safety and memory management. Rust’s focus on efficiency and safety makes it a promising language for networking code. Tokio is a set of networking libraries built on Rust. Tokio enables developers to write asynchronous IO operations by way of its multithreaded scheduler. Tokio’s goal is to make production-ready clients and servers easy to create by focusing on small, reusable components. Carl Lerche is an engineer at Buoyant, a company that makes the popular Linkerd and Conduit service mesh systems. Kubernetes developers deploy service mesh to their distributed applications as sidecar proxies. These proxies need to be low-latency and highly reliable. In that light, it makes sense that Conduit (the more recent service mesh from Buoyant) is built using Rust. Carl joins the show to describe why Rust is useful for building networked services. The post Rust Networking with Carl Lerche appeared first on Software Engineering Daily.

A database stores data to an underlying section of storage. If you are an application developer, you might think of your persistent storage system as being the database itself–but at a lower level, that database is writing to block storage, file storage, or object storage. A container orchestration system manages application containers. If you want to run WordPress (a blogging platform) on Kubernetes, that means you also need to run a database to store your blog posts in a persistent way. To run a database, you need to have an underlying storage medium–which could be a disk that is at your on-prem data center, or block storage on a disk at a cloud provider. Kubernetes is not the only container orchestrator. There’s also Cloud Foundry, Mesos, Docker Swarm, and several others. Each of these container orchestrators needs to be able to run a variety of persistent workloads (such as a MySQL database or a Kafka cluster). Each of these persistent workloads needs to be able to use different types of backing storage. With the range of container orchestrators and the range of backing storage types, a problem arises. Every storage type would have to write custom code to connect to each container orchestrator. The solution to this is the CSI: the container storage interface. The CSI is an interface layer between the container orchestrator and the backing storage system. In today’s episode, Jie Yu from Mesosphere describes the motivation for the CSI, and gives an overview for its design principles. There are great lessons here for anyone working with containers or distributed systems in general. Show Notes Stripe: Designing Robust and Predictable APIs with Idempotency  The post Container Storage with Jie Yu appeared first on Software Engineering Daily.

Containers have improved deployments and resource utilization. Kubernetes created a platform to manage those containers and orchestrate them into distributed applications. In today’s episode, we explore tools that improve the workflow of the application developer who is working with Kubernetes, including Helm, Draft, and Brigade. Helm is a package manager for Kubernetes, which allows users to find, share, and use software that is built for Kubernetes. The unit of installation for Helm users is a Helm Chart. Installing a Helm Chart can simplify the deployment of a database, load balancer, or continuous integration tool. Draft is a tool for simplifying the containerization process. When a developer runs Draft, a Dockerfile is created to containerize the application, and a Helm Chart is created to enable the application to be easily deployed. Brigade is a tool for creating and running Kubernetes workflows. Brigade allows for event-driven scripting on top of Kubernetes. Chatops, continuous integration systems, and complex big data pipelines can all be defined with Brigade. Brigade is exciting because it is a higher level tool on top of Kubernetes–in some ways similar to the “serverless on Kubernetes” systems we have covered in the past. Ralph Squillace is a principal program manager with Microsoft, where he works on containers, Linux, and cloud products. Ralph joins the show to talk about how developing with containers has changed in the last few years, and how it will continue to evolve in the near future. The post Container Native Development with Ralph Squillace appeared first on Software Engineering Daily.

Deploying software to a container presents a different security model than deploying an application to a VM. There is a smaller attack surface per container, but the container is colocated on a node with other containers. Containers are meant to have a shorter lifetime than VMs, so there are generally fewer consequences if a container needs to be destroyed and rebuilt due to a potential security vulnerability. Maya Kaczorowski works on container security at Google. In a recent talk at KubeCon, Maya discussed the runtime security of containers on Kubernetes. Maya joins the show to discuss container security, and what it means to software developers and operators. Maya also gives guidelines for evaluating the security of your own cluster. We talked about the security benefits of a managed Kubernetes provider and also explored how some container security vendor software works. The post Container Security with Maya Kaczorowski appeared first on Software Engineering Daily.

Voice interfaces are a newer form of communicating with computers. Alexa is a voice interface platform from Amazon. Alexa powers the Amazon Echo, as well as Alexa-enabled cars, refrigerators, and dishwashers. Any developer can build a device with a voice interface using a Raspberry Pi. Paul Cutsinger works on Echo and Alexa at Amazon. He’s focused on growing the market of developers who are building voice interfaces. In this episode, Paul describes how to design and implement a voice application for the Amazon Alexa platform. The market for voice powered apps is so new, and there has yet to be a “killer app.” If you like to tinker on new platforms, you will like this episode–and I was surprised by how easy it sounds to build a voice app. Personally, I use voice interfaces all the time–to set timers, to find out how to tell if a cucumber has gone bad, to ask what temperature to cook a potato at. Sometimes, when I am lying in bed trying to get to sleep, I will ask my nearest device to read me a Wikipedia article. These are great use cases, but I’m sure we will see something much more groundbreaking in the future. The post Alexa Voice Design with Paul Cutsinger appeared first on Software Engineering Daily.

Gloo is a function gateway built on top of the popular open source project Envoy. The goal of Gloo is to decouple client-facing APIs from upstream APIs. Gloo is similar to an API gateway, which is a tool that software companies can use to collect all their APIs and one place and impose security, monitoring, and other standards around those APIs. The goal of Gloo is to provide all the tools necessary to glue together traditional and cloud-native applications. Idit Levine is the CEO of Solo.io, a company that is building Gloo and several other projects. The post Gloo: Function Gateway with Idit Levine appeared first on Software Engineering Daily.

Mesos is a system for managing distributed systems. The goal of Mesos is to help engineers orchestrate resources among multi-node applications like Spark. Mesos can also manage lower level schedulers like Kubernetes. A common misconception is that Mesos aims to solve the same problem as Kubernetes, but Mesos is a higher level abstraction. Ben Hindman co-founded Mesosphere to bring the Mesos project to market. Large enterprises like Uber, Netflix, and Yelp use Mesosphere for resource management. Before he started the company, Ben worked in the Berkeley AMP Lab, a research program where the Spark and Tachyon projects were also born. At this point, he has spent significant time in both academia and industry. This conversation spans distributed systems theory, history, and practice. Ben and I spoke at KubeCon 2018 in Copenhagen–which was an amazing conference. We were both amazed at how big the audience for Kubernetes has gotten, and the pace at which the technology is advancing. Today, Kubernetes is mostly used for scheduling containerized applications that engineers have built themselves. But there will be higher level tools that use Kubernetes as a building block. Much like Zookeeper was used as a building block for Hadoop, Kubernetes will be used to build serverless applications and distributed databases. Once you are using a distributed database built on Kubernetes, you don’t want to think about the container orchestration–you want to think about the raw storage and CPU requirements for that database. This is one reason why Mesos is so compelling. Since Kubernetes creates an increased cardinality of distributed systems, it’s good to know that there is a framework built to manage those higher level applications. The post Cluster Schedulers with Ben Hindman appeared first on Software Engineering Daily.

When I buy a mattress online, I pay for it with my credit card. Behind the scenes, a complex series of transactions occur between a payment gateway, the credit card company, and a few banks. There are problems with this process–it is slow, complex, and involves the synchronization of several different parties. Some consumers will not want to purchase the mattress because they do not have cash, and the lending rates they get offered are higher than they are willing to spend. If these consumers were presented with more intelligent loan rates, the lender could still make money, the mattress company could still make money, and the consumer would get a new mattress. It’s a missed opportunity all around. Affirm is a consumer financial services company. Their first product offers loans to consumers making purchases. In today’s episode, Affirm CTO Libor Michalek explains how Affirm decided to build this product, and what they have done to scale it. The conversation took me by surprise. Affirm was started by Max Levchin, who was a co-founder of PayPal. I assumed that when Affirm was created, they already knew exactly what they were going to build–because Affirm is a payments company and Max has had knowledge of the payments industry going back two decades. In reality, Affirm started out with more vague ideas around what they were building. They spent some time running small experiments as they looked for product/market fit–just like a bootstrapped startup would have. It was inspiring to know that even an experienced team is willing to go through the humble process of searching for a product within a space they are deeply familiar with. We didn’t get to all the questions I was planning to explore, but I hope to do another show about Affirm in the future. Show Notes Affirm Careers Page The post Affirm Engineering with Libor Michalek appeared first on Software Engineering Daily.

Superpedestrian is a robotic bicycle wheel that learns how you pedal and personalizes your bicycle ride. The engineering challenges of Superpedestrian are at the intersection of robotics, software, and real-time analytics. The first half of today’s show is about Superpedestrian. Goss Nuzzo Jones and Matt Cole are engineers at Superpedestrian. The slides for their presentation are also in the show notes. The second half of today’s show is about HubSpot, a massive business with lots of infrastructure challenges. Thomas Petr explained how HubSpot’s engineering has matured, and some of the scaling problems they have tackled. Last month, we had three Software Engineering Daily Meetups: in New York, Boston, and Los Angeles. At each of these Meetups, listeners from the SE Daily community got to meet each other and talk about software–what they are building and what they are excited about. I was happy to be in attendance at each of these, and I am posting the talks given by our presenters. The audio quality is not perfect on these, but there are also no ads. Thank you to HubSpot for hosting this Meetup–they have beautiful offices and if you are looking for a job (or if you want to host a technology Meetup in the Boston area) I strongly recommend checking them out. We’d love to have you as part of our community. We will have more Meetups eventually, and you can be notified of these by signing up for our newsletter. Come to SoftwareDaily.com and get involved with the discussion of episodes and software projects. You can also check out our open source projects–the mobile apps, and our website. The post Superpedestrian Robotic Wheel / Infrastructure at HubSpot Meetup Talks appeared first on Software Engineering Daily.

Alexis Le-Quoc started Datadog in 2010, after living through the Internet boom and bust cycle of the late 90s and early 2000s. In 2010, cloud was just starting to become popular. There was a gap in the market for infrastructure monitoring tools, which Alexis helped fill with the first version of Datadog. Since 2010, the number of different cloud infrastructure products has proliferated. There were new databases, queueing systems, virtualization and containerization tools. Web 2.0 took off, and thousands of new Internet businesses got started. Many of these businesses used Datadog to monitor their increasingly wide range of infrastructure configurations–and Datadog began to scale. On today’s show, Alexis tells the story of how Datadog grew from its first product into a variety of tools–infrastructure monitoring, logging, and application performance monitoring. Monitoring is a unique challenge–there is a ton of data, the data is latency sensitive, and the data is operationally important. These engineering constraints provide for a great conversation. Alexis is the CTO of Datadog, and we talked about cloud providers, building a business, infrastructure, and how to scale engineering management. Full disclosure: Datadog is a sponsor of Software Engineering Daily. The post Building Datadog with Alexis Le-Quoc appeared first on Software Engineering Daily.