Cloud Engineering Archives - Software Engineering Daily

“Sometimes there’s a misconception that Genie is a job scheduling platform… Genie really represents our extraction layer, from what our computational resources are, to our end user jobs.” Genie is an open-source tool that provides job and resource management for the Hadoop ecosystem in the cloud. Tom Gianos is a Senior Software Engineer at Netflix focusing on its big data platform. He is one of the core contributors in charge of maintaining and improving Genie. Questions How fast is the pace of Netflix data collection increasing? What are the differences between running Hadoop in the cloud and running it in a data center? Why is job and resource management so important for a Hadoop ecosystem? What does Netflix do with spare computational resources? What is a bonus cluster? Does Netflix collect more data than it can practically make use of? Links Genie Yarn – Resource Scheduling Layer Hadoop Platform as a Service in the Cloud SE Daily Episode on Presto Spring Boot The post Netflix Genie with Tom Gianos appeared first on Software Engineering Daily.

Distributed systems programming will always be a world of tradeoffs — there is no silver bullet in the future. But life can be made easier with tactics such as the actor pattern and the use of conflict-free replicated data types (CRDTs). Caitie McCaffrey is a distributed systems engineer who currently works at Twitter. She previously worked on Halo 4 at Microsoft and 343 Industries. At QCon San Francisco, she will be hosting the track Taming Distributed Architecture. Questions Describe the service problems on Halo 4 launch day. How did a kill switch help the situation? Why are “implicit assumptions the killer of any distributed system”? Why did you use the actor model to build Halo 4 services? What is a conflict-free replicated data type? Links Halo 4: High Demand with Low Latency and High Availability (YouTube) Caitie McCaffrey’s blog Conflict-free replicated data type Actor pattern The post Taming Distributed Architecture with Caitie McCaffrey appeared first on Software Engineering Daily.

“DevOps is not a thing. It is a set of problem statements and solution possibilities that are always growing.” The hosts of DevOps Cafe joined Software Engineering Daily for a conversation about DevOps culture and misconceptions. Questions What do software engineers need to know about DevOps? What are the biggest misconceptions around DevOps? Is DevOps ultimately about continuous integration, or containers, or any specific set of technologies? Why is Netflix a good DevOps case study? What isn’t DevOps? Links DevOps Cafe Netflix Culture SlideShare Ops, DevOps, and Noops at Netflix Release It by Michael Nygard Amazon article by New York Times The post Origin of DevOps with John and Damon from DevOps Cafe appeared first on Software Engineering Daily.

Jenkins is an extensible open source continuous integration server. Kohsuke Kawaguchi is the primary developer of Jenkins CI and the CTO of CloudBees, a provider of enterprise Jenkins. Questions: How does continuous integration affect DevOps? What has changed in the five years since Jenkins was created? In what ways is Jenkins opinionated? What are the synergies between Docker and Jenkins? What are some common continuous integration patterns and antipatterns? Links: Jenkins Homepage CloudBees Continuous Delivery Thoughtworks Continuous Delivery Continuous Integration Slideshare The post Continuous Delivery with Jenkins Creator Kohsuke Kawaguchi appeared first on Software Engineering Daily.

Container infrastructure has benefits of security, scalability and efficiency. Containers are a central component of the DevOps movement. Joyent provides simple, secure deployment of containers with bare metal speed on container-native infrastructure Bryan Cantrill is the CTO of Joyent, the father of DTrace and an OS kernel developer for 20 years. Questions: Why are containers relevant to DevOps? Why are containers advantageous to the developer? What is the benefit of running a container on the principal hardware rather than the VM? Do DevOps employees exist? How is DTrace useful for DevOps? What are the societal benefits of global warming? Links: Docker and the Future of Containers in Production Docker and the Future of Containers in Production (SlideShare) Joyent: Triton Elastic Container Infrastructure Bryan Cantrill AMA The post Containers with Bryan Cantrill from Joyent appeared first on Software Engineering Daily.

Rocana applies big data, advanced analytics, and visualizations to dev ops in order to guide users to the root causes of problems. Eric Sammer is the co-founder and CTO of Rocana. At Cloudera, he served as an Engineering Manager responsible for tools and partner integrations. Within that role, he developed many of Cloudera’s best practices for developing large, distributed, data processing infrastructure. Questions include: Does a nondeterministic data center problem ever sink a company? How do you identify salient differences between competing open-source products like Spark/Storm/Samza? How does a developer plug into Rocana? What is the open-source strategy for Rocana’s product? What is the architecture of Rocana? Links: Product overview Rocana on Twitter Cloudera interviews Eric Sammer The post Hadoop Ops: Rocana CTO Eric Sammer Interview appeared first on Software Engineering Daily.