"You've Got a Job to Do" with Joel Meador

Apr 4, 2024

SmartLogic Engineer Joel Meador joins the podcast to discuss the vital role of background jobs in app performance, optimizing user experience, monitoring and debugging strategies, system security, and tools like Oban and Sidekiq. They emphasize best practices for implementing background jobs, CPU utilization, and continuous improvement in job management.

Ask episode

Chapters

Transcript

Episode notes

Introduction

00:00 • 2min

Exploring Common Use Cases for Background Job Tools

01:38 • 23min

Choosing the Right Tools for Job Systems in Programming

24:37 • 2min

Security Considerations for Job Systems

26:38 • 11min

Exploring Community Developments and Future Possibilities in Tech Projects

37:13 • 2min

Exploring Background Job Implementation Options in Web Development

38:50 • 2min

Tips for Efficient Background Job Management and Support for Miami Indians' Fundraising

41:16 • 3min

Today on Elixir Wizards Office Hours, SmartLogic Engineer Joel Meador joins Dan Ivovich to discuss all things background jobs.

The behind-the-scenes heroes of app performance and scalability, background jobs take center stage as we dissect their role in optimizing user experience and managing heavy-lifting tasks away from the main application flow. From syncing with external systems to processing large datasets, background jobs are pivotal to successful application management.

Dan and Joel share their perspectives on monitoring, debugging, and securing background jobs, emphasizing the need for a strategic approach to these hidden workflows.

Key topics discussed in this episode:

The vital role of background jobs in app performance
Optimizing user experience through background processing
Common pitfalls: resource starvation and latency issues
Strategies for effective monitoring and debugging of task runners and job schedulers
Data integrity and system security in open source software
Background job tools like Oban, Sidekiq, Resque, Cron jobs, Redis pub sub
CPU utilization and processing speed
Best practices for implementing background jobs
Keeping jobs small, focused, and well-monitored
Navigating job uniqueness, locking, and deployment orchestration
Leveraging asynctask for asynchronous operations
The art of continuous improvement in background job management

Links mentioned in this episode:

https://redis.io/
Oban job processing library https://hexdocs.pm/oban/Oban.html
Resque Ruby library for background jobs https://github.com/resque
Sidekiq background processing for Ruby https://github.com/sidekiq
Delayed Job priority queue system https://github.com/collectiveidea/delayed_job
RabbitMQ messaging and streaming broker https://www.rabbitmq.com/
Mnesia distributed telecommunications DBMS https://www.erlang.org/doc/man/mnesia.html
Task for Elixir https://hexdocs.pm/elixir/1.12/Task.html
ETS in-memory store for Elixir and Erlang objects https://hexdocs.pm/ets/ETS.html
Cron - https://en.wikipedia.org/wiki/Cron
Donate to Miami Indians of Indiana https://www.miamiindians.org/take-action
Joel Meador on Tumblr https://joelmeador.tumblr.com/

Special Guest: Joel Meador.