Failure Modes, Checkpointing and Backfill Importance

Intro topic: Asymmetric Returns

News/Links:

• NanoChat by Andrej Karpathy
  • https://github.com/karpathy/nanochat
• Pydantic AI
  • https://www.marktechpost.com/2025/03/25/pydanticai-advancing-generative-ai-agent-development-through-intelligent-framework-design/
• 1000th Starlink this year
  • https://spaceflightnow.com/2025/05/16/live-coverage-spacex-plans-morning-launch-of-starlink-satellites-from-california/
• ChatGPT Apps SDK
  • https://openai.com/index/introducing-apps-in-chatgpt/
    
    

Book of the Show

• Patrick
  • The Will of the Many by James Islington
    • https://amzn.to/43IfU8Q
• Jason
  • Interview with DHH (Founder of Ruby on Rails)
    • https://www.youtube.com/watch?v=vagyIcmIGOQ
      
      

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

• Patrick
  • Factorio
    • https://www.factorio.com/ 
• Jason
  • nip.io 

Topic: Workflow Orchestrators

• Why
  • Batch jobs (embarrassingly parallel)
  • Long-running tasks (e.g. transcoding video)
  • Checkpointing/resuming
• How
  • Message Queues
  • Containerization
  • Worker Pools & Autoscaling
  • History & Backfill
• Steps to run workflows:
  • Containerize the workflow definition and send to the cloud
  • Containerize all the individual tasks
  • Submit job(s)
• Examples
  • Airflow
    • Legacy but dominant
  • Dagster
    • Great UX for python developers
  • Temporal: https://temporal.io/
    • The new hotness
  • Ray
    • Low-level but very powerful
  • Kubeflow
    • Designed for ML workflows, integrated dashboard