Parker Selbert and Shannon Selbert discuss robust job processing in Elixir, touching on the reliability, consistency, and observability factors. They delve into the creation of Oban, lessons learned, job throughput, distributed architectures, retry algorithms, and the significance of PostgreSQL and Elixir in this process.
Read more
AI Summary
AI Chapters
Episode notes
auto_awesome
Podcast summary created with Snipd AI
Quick takeaways
Oban in Elixir ensures seamless job handling with unique configurations and persistence features.
PostgreSQL plays a vital role in ensuring reliability and consistency in job processing with Oban.
Starting with essential features and expanding gradually is crucial for Oban's successful open-core model.
Deep dives
Robust Job Processing in Oban Pro
Oban Pro offers robust job processing capabilities with features like persistence, unique job configurations, and a focus on side effects. It provides a featureful package ready for deployment with minimal configuration, ensuring seamless job handling without the need for additional tools or scheduled jobs.
Background on Oban and its Evolution
Oban, a background job processor specifically designed for Elixir, has evolved over time to offer essential functionalities like job management and persistence. With a brief history that includes leveraging PostgreSQL for reliable job processing, Oban has become an integral part of Elixir applications for efficient job handling.
Lessons Learned and Core Functionality of Oban
In its five-year journey, Oban has learned valuable lessons, emphasizing the importance of strategic feature selection for an open-core model. The podcast highlights the significance of starting with essential features and gradually expanding, showcasing Oban's open-source roots leading to commercial offerings like Oban Web and OpenPro, designed to address advanced business functionality needs.
Versatile Use Cases and Scalability of Oban
Oban finds application in various industries such as fintech and energy, enabling diverse use cases like sending emails in the background, scheduling tasks, and handling large job volumes efficiently. Its scalability is evident from processing over a million jobs per minute across multiple nodes and supporting workflows with extensive job processing capacities.
Challenges and Solutions in Job Processing with Oban
One of the common challenges for new users adopting Oban is the correct naming and configuration of queues, essential for segregating workloads and ensuring efficient job handling. The podcast emphasizes maintaining up-to-date versions of Oban to leverage improved functionalities and enhance job processing capabilities in Elixir applications.
Shannon Selbert, co-founder of Soren and developer of Oban, and Parker Selbert, creator of the Oban background job framework, chief architect at dscout, and co-founder of Soren, speak with SE Radio host Gavin Henry about robust job processing in Elixir. They explore the reliability, consistency, and observability in relation to job processing, to understand how Oban, Elixir, and PostgreSQL deliver them.
The Selberts describe why Oban was created, its history, which parts of the Elixir ecosystem they use, and why this would not be possible without PostgreSQL and Elixir. They discuss the lessons learned in the 5 years since the first release, as well as use cases, job throughput, the hardest problem to solve so far, workers, queues, CRON, distributed architectures, retry algorithms, just-once methodologies, the reliability the beam brings, consistency across nodes, how PostgreSQL is vital, telemetry data, best use cases for Oban, and the most common issues that new users face. Brought to you by IEEE Computer Society and IEEE Software magazine.
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode