The New Stack Podcast

Apache Airflow is an open-source platform for building machine learning pipelines. It allows users to author, schedule, and monitor workflows, making it well-suited for tasks such as data management, model training, and deployment. In a discussion on The New Stack Makers, three technologists from Amazon Web Services (AWS) highlighted the improvements and ease of use in Apache Airflow.Dennis Ferruzzi, a software developer at AWS, is working on updating Airflow's logging and metrics backend to the OpenTelemetry standard. This update will provide more granular metrics and better visibility into Airflow environments. Niko Oliveria, a senior software development engineer at AWS, focuses on reviewing and merging pull requests as a committer/maintainer for Apache Airflow. He has worked on making Airflow a more pluggable architecture through the implementation of AIP-51.Raphaël Vandon, also a senior software engineer at AWS, is contributing to performance improvements and leveraging async capabilities in AWS Operators, which enable seamless interactions with AWS. The simplicity of Airflow is attributed to its Python base and the operator ecosystem contributed by companies like AWS, Google, and Databricks. Operators are like building blocks, each designed for a specific task, and can be chained together to create workflows across different cloud providers.The latest version, Airflow 2.6, introduces sensors that wait for specific events and notifiers that act based on workflow success or failure. These additions aim to simplify the user experience. Overall, the growing community of contributors continues to enhance Apache Airflow, making it a popular choice for building machine learning pipelines.Check out the full article on The New Stack:How Apache Airflow Better Manages Machine Learning Pipelines Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

In this episode featuring Nima Negahban, CEO of Kinetica, the potential impact of generative AI tools like ChatGPT on businesses and organizations is discussed. Negahban highlights the transformative potential of generative AI when combined with data analytics. One use case he mentions is an "Alexa for all your data," where real-time queries can be made about store performance or product underperformance in specific weather conditions. This could provide organizations with a new level of visibility into their operations.Negahban identifies two major challenges in the generative AI space. The first is security, especially when using internal data to train AI models. The second challenge is ensuring accuracy in AI outputs to avoid misleading information. However, he emphasizes that generative AI tools, such as GitHub Copilot, can bring a new expectation of efficiency and innovation for developers.The future of generative AI in the enterprise involves discovering how to orchestrate these models effectively and leverage them with organizational data. Negahban mentions the growing interest in vector search and vector database capabilities to generate embeddings and perform embedding search. Kinetica's processing engine, coupled with OpenAI technology, aims to enable ad hoc querying against natural language without extensive data preparation, indexing, or engineering.Check out the episode to hear more about how the integration of generative AI and data analytics presents exciting opportunities for businesses and organizations, providing them with powerful insights and potential for creativity and innovation.Read more about Generative AI on The New StackIs Generative AI Augmenting Our Jobs, or About to Take Them?Generative AI: How to Choose the Optimal DatabaseHow Will Generative AI Change the Tech Job Market?Generative AI: How Companies Are Using and Scaling AI Models Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

In this episode of The New Stack Makers from KubeCon EU 2023, Rob Barnes, a senior developer advocate at HashiCorp, discusses how their networking service, Consul, allows users to incorporate containers or virtual machines into their workflows without imposing container usage. Consul, an early implementation of service mesh technology, offers a full-featured control plane with service discovery, configuration, and segmentation functionalities. It supports various environments, including traditional applications, VMs, containers, and orchestration engines like Nomad and Kubernetes.Barnes explains that Consul can dictate which services can communicate with each other based on rules. By leveraging these capabilities, HashiCorp aims to make users' lives easier and software more secure.Barnes emphasizes that there are misconceptions about service mesh, with some assuming it is exclusively tied to container usage. He clarifies that service mesh adoption should be flexible and meet users wherever they are in their technology stack. The future of service mesh lies in educating people about its role within the broader context and addressing any knowledge gaps.Join Rob Barnes and our host, Alex Williams, in exploring the evolving landscape of service mesh and understanding how it can enhance workflows.Find out more about HashiCorp or the biggest news from KubeCon on The New Stack:HashiCorp Vault Operator Manages Kubernetes SecretsHow HashiCorp Does Site Reliability EngineeringA Boring Kubernetes Release Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

What did software engineers at KubeCon say about how AI is coming up in their work? That's a question we posed Taylor Dolezal, head of ecosystem for the Cloud Native Computing Foundation at KubeCon in Amsterdam. Dolezal said AI did come up in conversation."I think that when it's come to this, typically with KubeCons, and other CNCF and LF events, there's always been one or two topics that have bubbled to the top," Dolezal said.At its core, AI surfaces a data issue for users that correlates to data sharing issues, said Dolezal in this latest episode of The New Stack Makers.Read more about AI and Kubernetes on The New Stack:3 Important AI/ML Tools You Can Deploy on KubernetesFlyte: An Open Source Orchestrator for ML/AI WorkflowsOvercoming the Kubernetes Skills Gap with ChatGPT Assistance Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Kubernetes release 1.27 is boring, says Xander Grzywinski, a senior product manager at Microsoft.It's a stable release, Grzywinski said on this episode of The New Stack Makers from KubeCon Europe in Amsterdam."It's reached a level of stability at this point," said Grzywinski. "The core feature set has become more fleshed out and fully realized.The release has 60 total features, Grzywinski said. The features in 1.27 are solid refinements of features that have been around for a while. It's helping Kubernetes be as stable as it can be.Examples?It has a better developer experience, Grzywinski said. Storage primitives and APIs are more stable. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

The mystery and miracle of flight sparked Ev Kontsevoy’s interest in engineering as a child growing up in the Soviet Union.“When I was a kid, when I saw like airplane flying over, I was having a really hard time not stopping and staring at it until it's gone,” said Kontsevoy, co-founder and CEO of Teleport, said in this episode of the Tech Founders Odyssey podcast series. “I really wanted to figure out how to make it fly.”Inevitably, he said, the engineering path led him to computers, where he was thrilled by the power he could wield through programming. “You're a teenager, no one really listens to you yet, but you tell a computer to go print number 10 ... and then you say, do it a million times. And the stupid computer just prints 10 million. You feel like a magician that just bends like machines to your will.”In this episode of the series, part of The New Stack Makers podcast, Kontsevoy discussed his journey to co-founding Teleport, an infrastructure access platform, with TNS co-hosts Colleen Coll and Heather Joslyn. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Developer tool integration and AI differentiate workflows to achieve that "fluid" state developers strive for in their work.Amazon CodeCatalyst and Amazon CodeWhisperer exemplify how developer workflows are accelerating and helping to create these fluid states. That's a big part of the story we hear from Harry Mower, director AWS DevOps Services, and Doug Seven, director, Software Development, AWS CodeWhisperer, from our recording in Seattle earlier in April for this week's  AWS Developer Innovation Day.CodeCatalyst serves as an end-to-end integrated DevOps toolchain that provides developers with everything they need to go from planning through to deployment, Mower said. CodeWhisperer is an AI coding companion that generates whole-line and full-line function code recommendations in an integrated development environment (IDE).CodeWhisperer is part of the IDE, Seven said. The acceleration is two-fold. CodeCatalyst speeds the end-to-end integration process, and CodeWhisper accelerates writing code through generative AI. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Just as everyone was heading out to the New Year's holidays last year, CTO Rob Zuber got a surprise of a most unwelcome sort. A customer alerted CircleCI to suspicious GitHub OAuth activity. Although the scope of the attack appeared limited, there was still no telling if other customers of the DevOps-friendly continuous integration and continuous delivery platform were impacted.This notification kicked off a deeper review by CircleCI’s security team with GitHub, and they rotated all GitHub OAuth tokens on behalf of their customers. On January 4, the company also made the difficult but necessary decision to alert customers of this “security instance,” asking them to immediately rotate any and all stored secrets and review internal logs for any unauthorized access.In this latest episode of The New Stack Makers podcast, we discuss with Zuber the attack and how CircleCI responded. We also talk about what other companies should do to avoid the same situation, and what to do should it happen again.  Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Feature flags, the toggles in software development that allow you to turn certain features on or off for certain customers or audiences, offer release management at scale, according to Karishma Irani, head of product at LaunchDarkly.But they also help unleash innovation, as she told host Heather Joslyn of The New Stack in this episode of The New Stack Makers podcast. And that points the way to a future where the potential for easy testing can inspire new features and products, Irani said.“We've observed that when the risk of releasing something is lowered, when the risk of introducing bugs in production or breaking, something is reduced, is lowered, our customers feel organically motivated to be more innovative and think about new ideas and take risks,” she said. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Hoi Europe and beyond!Once again it is time for cloud native enthusiasts and professionals to converge and discuss cloud native computing in all its efficiency and complexity. The Cloud Native Computing Foundation's KubeCon+CloudNativeCon 2023 is being held later this month in Amsterdam, April 18 - 21, at the Rai Convention Centre.In this latest edition of The New Stack podcast, we spoke with two of the event's co-chairs who helped define this year's themes for the show, which is expected to draw over 9,000 attendees: Aparna Subramanian, Shopify's Director of Production Engineering for Infrastructure; and Cloud Native Infra and Security Enterprise Architect Frederick Kautz.  Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.