Running in Production

Nick Janetakis - Full stack developer
undefined
Nov 16, 2020 • 1h 28min

Crowdbotics Is a React Native + Django App Builder / Deployer

In this episode of Running in Production, Anand Kulkarni goes over his app builder and deployer platform using Django. Over 15,000 customers have built and deployed their apps through it. It’s running on 1,000+ Heroku Dynos and has been up and running since early 2017. Anand covers having a mono repo for the 100k+ lines of back-end code, switching to an API driven back-end, using Terraform, making use of Heroku’s eco-system and much more. Topics Include 6:59 – Motivation for using Django and Python 10:36 – Using the Django admin, apps and having a mono repo on the back-end 16:36 – There’s over 110,000 lines of back-end code and a bit of linting 24:38 – Using DJ Stripe for Stripe integration 28:27 – Switching over to React + Vue after having server rendered Django templates 34:20 – It’s hosted on Heroku and uses Redis, Celery, PostgreSQL, Docker and Terraform 38:32 – Taking advantage of using full text search within PostgreSQL 42:03 – Doing a fair bit of code changes (git commits) within Celery workers 50:10 – Picking Heroku, running 1,000+ Dynos and having a low 5 figures monthly bill 54:43 – Making sure customer apps stay up even during traffic spikes 56:28 – A list of Heroku add-ons that are being used and Cloudflare for a few things too 1:03:30 – Using Terraform to provision Heroku resources 1:05:59 – Step by step on how new code gets from development to production 1:12:42 – Dealing with customer secrets securely 1:15:57 – Automated backups for the database and user uploaded files 1:19:34 – Sentry handles exception tracking and notifying developers of errors 1:22:29 – Best tips? For the important parts of your app, use technologies you know 1:27:04 – Check out the Crowdbotics blog Links 📄 References https://yeoman.io/ https://12factor.net/ ⚙️ Tech Stack django → python → react → vue → circle-ci → cloudflare → docker → heroku → papertrail → postgres → redis → s3 → sendgrid → sentry → slack → stripe → terraform → webpack → 🛠 Libraries Used https://github.com/encode/django-rest-framework https://www.pylint.org/ https://eslint.org/ https://github.com/jazzband/django-oauth-toolkit https://github.com/pennersr/django-allauth https://github.com/dj-stripe/dj-stripe https://github.com/celery/celery Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.
undefined
Nov 9, 2020 • 1h 15min

Divio Is a Cloud Platform for Managing Web Applications

In this episode of Running in Production, Daniele Procida walks us through their cloud platform to manage web apps using Django. 10s of millions of web requests flow through their client’s applications. It’s running on multiple cloud providers and has been up and running since 2015. Daniele covers the importance of a stable platform when you’re dealing with customers who have their own customers, writing upwards of a million lines of Python across their services, being vendor neutral with hosting infrastructure and more. Topics Include 3:08 – Publicly available since 2015 but it was in development for years beforehand 8:43 – Motivation for using Django and Python 11:08 – The control panel is mainly a Django app with a React front-end 13:54 – Nearly everything is written in Python and Django (over 500k lines of code) 19:25 – Using Django Rest Framework and a few other Python libraries 24:24 – Using Docker, Celery and PostgreSQL 30:04 – Uncovering a bit more on how Divio works and what it supplies developers 33:39 – Web servers, custom load balancers and SSL certificates with Let’s Encrypt 37:13 – Being vendor neutral when it comes to their hosting infrastructure 40:16 – How their server infrastructure is set up 46:10 – Using various automation tools such as Ansible, Helm and Terraform 50:58 – Deploying to a large amount of servers with a rolling restart strategy 53:18 – The deploy process when they roll out new features to their platform 1:02:09 – Database / user media backups along with alerting and monitoring 1:06:59 – Using Stripe for processing payments 1:10:23 – Best tips? Reduce miscommunication and try to use your own services if you can 1:13:44 – Check out https://www.divio.com/ Links 📄 References https://documentation.divio.com/ ⚙️ Tech Stack django → python → react → ansible → aws → azure → cloudflare → docker → docker → gcp → gitlab-ci → kubernetes → lets-encrypt → nginx → pagerduty → postgres → redis → stripe → terraform → 🛠 Libraries Used https://github.com/encode/django-rest-framework https://hypothesis.readthedocs.io/en/latest/ https://github.com/boto/boto https://github.com/celery/celery Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.
undefined
Nov 2, 2020 • 1h 23min

Daughterly Care Is a CRM to Help Care for the Elderly

In this episode of Running in Production, Barton Ip talks us through building a custom CRM to help care for the elderly using Django. They are running on AWS with ECS and the site has been up and running since early 2019. Barton talks about hiring contractors when needed, splitting logic across 40+ Django apps, heavily using Celery, incrementally improving their infrastructure over time and more. Topics Include 3:51 – Motivation for using Django and Python 8:05 – It’s broken up into 40-60 different Django apps 12:57 – Using Pandas and NumPy to help pair up patients and care givers 17:00 – All of the data points are saved into PostgreSQL and Celery is heavily used too 26:38 – It started with Django templates but now it’s using DRF for the API + Vue 32:49 – They are using Docker and overall their stack is very AWS centric 34:06 – Each developer has their own EC2 instance as a dev environment 40:13 – nginx sits in front of uwsgi and Cloudflare is hosting their DNS records 43:27 – AWS Athena has been wonderful for getting at and querying audit / API logs 46:33 – Scaling from 1 ECS instance (4 CPUs / 16 GB RAM) to however many they need 50:52 – The process to deploy code from development to production 58:22 – Delete prevention, backing up data and testing the restore process 1:05:26 – Migrating to Terraform and Ansible to help configure their infrastructure 1:07:40 – Alarms and sending alerts over Slack and to Sentry 1:09:50 – All of this infrastructure wasn’t planned initially, it grew organically over time 1:15:33 – Best tips? Avoid the sunk cost fallacy (don’t let prior time spent dictate what you do) 1:22:06 – You can find Barton’s software development company at https://initialise.io/ Links 📄 References https://en.wikipedia.org/wiki/JSON_Web_Token https://aws.amazon.com/athena/ https://en.wikipedia.org/wiki/Sunk_cost ⚙️ Tech Stack django → python → vue → acm → cloudflare → cloudwatch → codedeploy → docker → ecs → mandrill → nginx → postgres → rds → s3 → sentry → slack → travis-ci → webpack → 🛠 Libraries Used https://pandas.pydata.org/ https://numpy.org/ https://github.com/celery/celery https://github.com/encode/django-rest-framework https://github.com/unbit/uwsgi https://github.com/psf/black Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.
undefined
Oct 26, 2020 • 1h 2min

Paparazzi Accessories Is an E-commerce Store That Sells Jewelry

In this episode of Running in Production, Dan Purcell goes over building an e-commerce store with Django that processes up to 3,000 orders per second. It’s running on 12+ web servers on DigitalOcean and the site has been up and running since 2014. Dan covers building an API backed app with Vue on the front-end, handling thousands of concurrent websocket connections, using Authorize.Net for payments and deploying to a bunch of web servers that are managed by Ansible with Fabric. Topics Include 3:49 – Motivation for using Django and Python 8:13 – It’s a monolithic app with about a dozen Django apps using Vue on the front-end 11:42 – Challenges of dealing with thousands of concurrent sales 17:01 – Using Celery, Redis and 4 beefy PostgreSQL servers that are replicated 19:50 – Managing suspicious login attempts and using Slack for important notifications 22:02 – Running 3 nginx servers and using Sentry for exception tracking 24:02 – DigitalOcean is the main hosting provider, some servers have ~40 CPUs 27:34 – They’re running Ubuntu 18.04 LTS and the servers are managed by Ansible 32:33 – Using Authorize.Net to handle all payments and why they aren’t using Stripe 36:35 – The deploy process using Fabric from development to production 38:42 – Using feature flags with Django Waffle and deploying with a bit of down time 43:54 – Dealing with secrets by having an encrypted git repo 46:48 – Database backups happen through daily dumps and logging / monitoring 57:35 – Best tips? Start small, deploy early / often and automate as much as possible 1:01:08 – You can learn more about Dan on his site at https://velocitywebworks.com Links 📄 References https://www.starlette.io/ ⚙️ Tech Stack django → python → vue → ansible → authorize-net → digitalocean → gcp → lets-encrypt → nagios → nginx → postgres → redis → sentry → slack → supervisor → ubuntu → 🛠 Libraries Used https://github.com/encode/django-rest-framework https://github.com/celery/celery https://github.com/jazzband/django-axes https://www.fabfile.org/ https://github.com/unbit/uwsgi https://github.com/django-waffle/django-waffle Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.
undefined
Oct 19, 2020 • 35min

An Inventory Management System for an E-commerce Platform

In this episode of Running in Production, Galen Rice walks us through building an internal inventory management system with Django and Python. It’s running on a single bare metal server and has been up and running since 2013. Galen talks about replacing an Excel sheet with a custom app, using 20 Django apps, taking advantage of the Django admin and deploying everything onto a custom server. Topics Include 5:02 – Motivation for using Django and Python 6:49 – Despite being ~6 years old it uses Python 3.x but it also uses Django 1.8 10:23 – It’s 1 monolithic app split into about 20 Django apps across ~15k lines of code 14:39 – Staff workers are using the Django admin and there’s full text search with PostgreSQL 17:44 – It runs nginx on a bare metal server, also bulk CSV imports are done via cron jobs 22:52 – Walking us through the deploy process from development to production 27:11 – Traffic is served internally over HTTP and nginx serves static files 27:49 – Database backups were saved onto S3 and a backup hasn’t had to be used yet 30:37 – Taking the application down means 50 people stand around doing nothing 31:37 – Best tips? Make the smallest possible changes at a time and plan for having a team 34:15 – Galen is on GitHub if you want to see what he’s up to Links 📄 References https://www.manning.com/books/the-mikado-method ⚙️ Tech Stack django → python → centos → nginx → postgres → s3 → systemd → 🛠 Libraries Used https://www.reportlab.com/opensource/ https://github.com/python-amazon-mws/python-amazon-mws https://gunicorn.org/ Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.
undefined
Oct 12, 2020 • 1h 17min

A Video Course Platform Built for a Course Called Modern DevTools

In this episode of Running in Production, Umar Hansa talks about building a video course platform with Express and Node. It’s hosted on a single $10 / month DigitalOcean server. It took about 3 months of part time development and has been up and running since late 2017. Umar talks about keeping it simple, building a server rendered monolith, using SQLite and not being afraid to build something from scratch. He covered a lot of ground in this episode. Topics Include 4:03 – Motivation for using Node and Express 8:21 – A break down of what features the platform currently has 10:26 – Currency exchange rates, PayPal and Stripe integration options 17:46 – It’s a monolithic app with 4,000 lines of code that evolved slowly over time 21:53 – It’s using server rendered templates with progressive JS enhancements 29:29 – TypeScript is being used here and there and Gulp is building assets 32:49 – How asset cache busting works with Gulp and the template language 34:43 – SQLite is being used with the Knex library as a query builder 38:50 – A background worker isn’t being used but adding one is a work in progress 43:18 – Sending emails out with Postmark and dealing with 3rd party APIs being down 46:53 – It’s hosted on a single $10 / month DigitalOcean server running Ubuntu 18.04 LTS 52:13 – Managing the Node process using PM2 in production with minimal down time 58:37 – Setting the server up by hand by following tutorials and then using ServerPilot 1:04:34 – Using GitHub Actions in the future and secret management 1:09:03 – Depending on DigitalOcean’s automated backups and handling logs 1:14:37 – Best tips? Don’t be afraid to build something from scratch 1:16:03 – You can find Umar on Twitter at @umaar and he goes by the same name on multiple sites, also check out his new course at https://learnbrowsertesting.com Links 📄 References https://github.com/nickjj/flask-static-digest https://martinfowler.com/bliki/BlueGreenDeployment.html ⚙️ Tech Stack express → node → cloudflare → digitalocean → gulp → nginx → postmark → server-pilot → sqlite → ubuntu → updownio → vimeo → 🛠 Libraries Used https://www.npmjs.com/package/open-exchange-rates https://github.com/totaljs/node-paypal-express-checkout https://github.com/ericf/express-handlebars https://github.com/mozilla/nunjucks https://github.com/knex/knex https://github.com/Unitech/pm2 https://github.com/motdotla/dotenv Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.
undefined
Oct 5, 2020 • 1h 3min

An E-commerce Site That Lets Folks Buy Online Courses

In this episode of Running in Production, Dalton Polhill goes over building a text based course platform with Django and Python. It’s running on a single EC2 instance and was up and running after 2.5 months of development time. Dalton talks about doing contract work for a client, how the Django admin was a big help, happily using SQLite, the value in using web frameworks and more. Topics Include 2:46 – Motivation for using Django and Python 5:01 – The Django admin helped a lot for building this app 7:47 – Fleshing out what this application does (features, models, etc.) 13:18 – Going from internal testing to live testing for payments is stressful 18:15 – Mostly Django templates with sprinkles of JavaScript 20:41 – Building the core of the app first while learning how it all fits together 23:21 – SQLite is being used as the primary database 29:40 – Not using Docker caused a few hiccups along the way 35:32 – Identifying AWS services by name is hard, and the site is hosted on EC2 41:49 – The EC2 instance is running Ubuntu 18.04 LTS and was set up manually 44:28 – DNS can be confusing at times 46:25 – An old school way of deploying code from development to production 51:46 – Database backups aren’t automated yet 56:08 – Emails are sent out using SES and being aware of delivery limits 59:27 – Best tips? Learn one or more web frameworks 1:02:41 – You can find Dalton on GitHub Links 📄 References https://channels.readthedocs.io/en/latest/asgi.html ⚙️ Tech Stack django → python → apache → aws → lets-encrypt → route53 → ses → sqlite → stripe → ubuntu → Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.
undefined
Sep 28, 2020 • 35min

Tracking Games Played on a Flight Simulator

In this episode of Running in Production, Paul Cappaert talks about building a leader board for a flight simulator that he developed at University. It’s built with Django and Python and runs on the Heroku free tier. It was in development for most of the 2019 school year. Paul goes over using QR codes as a login mechanism and what it was like building an API back-end so that the game server (Unity) running the simulator can report back stats. Topics Include 3:36 – Motivation for using Django and Python 5:26 – It’s broken up into a few Django apps across a few thousand lines of code 6:31 – Using QR codes to identity users from the simulator (it uses Unity) 11:32 – Using Django REST Framework to accept requests from the game server 15:58 – Using SQLite locally but PostgreSQL on Heroku, Docker + Redis too 18:32 – Reasons for picking Heroku over the alternatives 22:46 – The process to develop and deploy the application 24:58 – Secrets are saved outside of GitHub 26:43 – Django templates power the front-end for the website 31:09 – Best tips? Don’t worry about not knowing everything from the get go 32:19 – Also don’t forget to read the manual, aka documentation 34:11 – Paul is on GitHub at https://github.com/paulcappaert Links 📄 References https://unity.com/ ⚙️ Tech Stack django → python → docker → heroku → postgres → redis → sendgrid → sqlite → 🛠 Libraries Used https://github.com/lincolnloop/python-qrcode https://github.com/python-pillow/Pillow https://github.com/encode/django-rest-framework https://github.com/django/daphne Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.
undefined
Sep 21, 2020 • 55min

Collecting and Processing Genomic Data to Help Cure Rare Diseases

In this episode of Running in Production, Dan Kolbman goes over using Django to build an internal tool that helps make sense of ~5 Petabytes of Genomic data that is then made available to clinicians. It’s running across many different AWS resources using ECS Fargate. Dan walks us through what their app does, dealing with loads of data, using GraphQL, getting away from using Serverless and going mostly all-in with AWS. Their apps are open source too. The ones we’ll be talking about are on GitHub here and here. Topics Include 3:55 – Motivation for using Django and Python 6:11 – Using GraphQL and having a few separate apps (micro-service’ish) 11:16 – Querying ~5 Petabytes of Genomic data stored on S3 17:21 – Using both Graphene (GraphQL) and Django REST Framework 22:44 – Docker is being used in dev (Docker Compose) and in production (ECS Fargate) 25:20 – PostgreSQL and Redis are being used too with lots of background tasks 27:29 – Breaking down which AWS resources they use, along with using Terraform 37:02 – Netlify is being used for deploy previews and CloudFront for production 39:34 – Breaking down the work flow for deploying something from dev to prod 46:55 – Planning for disasters and handling backing up data 51:02 – Automated metrics around CPU and memory, along with alerting 52:29 – Best tips? Use tools that a lot of people have thought long and hard about 54:21 – You can find Dan on GitHub and his personal website Links 📄 References https://www.nih.gov/ https://aws.amazon.com/athena/ https://www.nomadproject.io/ ⚙️ Tech Stack django → python → apollo → aws → circle-ci → cloudfront → docker → ecs → elasticache → fargate → graphql → jenkins → netlify → open-source → postgres → rds → redis → s3 → ses → slack → terraform → 🛠 Libraries Used https://github.com/encode/django-rest-framework https://github.com/graphql-python/graphene https://github.com/rq/django-rq Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.
undefined
Sep 14, 2020 • 44min

Social Hiring Platform for the Food and Beverage Industry

In this episode of Running in Production, Colin McFaul goes over using Django to build a social hiring platform for the food and beverage industry. It was in development for 9 months before it went live in early 2020. It’s hosted on Heroku. Colin talks about building 2 separate apps and keeping the web app as a monolithic app with ~42k LOC, but it’s broken up with Django apps. Elasticsearch is used to power an auto-complete search feature. Billing isn’t implemented yet but it’s currently in development with Stripe. Topics Include 3:37 – Motivation for using Django and Python 5:46 – Breaking down what the 2 apps do (sales tool and the main site) 10:34 – The web app is a monolithic app but it’s broken up by Django apps 13:24 – There’s ~42k lines of code, and it’s mainly using server rendered templates 17:43 – Elasticsearch is being used for full text search and PostgreSQL for the rest 20:38 – Redis and Celery are being used for background and periodic tasks 24:27 – Billing is half way implemented with Stripe 27:32 – Thought process around using Heroku (2x $25 / mo Dynos) to host the site 31:39 – Logging (Papertrail), metrics and error reporting (Sentry) 33:52 – The deploy process from development to production (it includes Docker) 36:32 – Doing a PG dump to backup the DB and user uploaded content is saved on S3 38:37 – Best tips? Put in the research before implementing something 43:23 – Check out https://www.ndustrylink.com Links ⚙️ Tech Stack django → python → docker → elasticsearch → heroku → papertrail → postgres → redis → s3 → sendgrid → sentry → stripe → 🛠 Libraries Used https://github.com/pennersr/django-allauth https://github.com/encode/django-rest-framework https://github.com/yourlabs/django-autocomplete-light https://github.com/celery/celery https://github.com/joke2k/django-environ Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

The AI-powered Podcast Player

Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
App store bannerPlay store banner
Get the app