Running in Production

In this episode of Running in Production, Peter Kazarinoff goes over deploying Jupyter Hub on DigitalOcean with Ansible. It’s running on a single $40 / month DigitalOcean server for his 24 students. He’s been doing this since summer 2018. Peter talks us through what Jupyter notebooks are and what Jupyter Hub does, the process of learning DevOps related tasks to deploy a project he didn’t write, his journey of automating the process with Ansible and how it’s ok to make mistakes along the way. Topics Include 1:07 – What exactly is Jupyter and Jupyter Hub? 2:12 – Students can access a fully working Python environment from their browser 2:58 – Setting up a fully working local development environment can be tricky 4:35 – 24 students (the size of the class) access the Jupyter Hub server 5:20 – Comparing Jupyter Hub to other alternative tools and services 6:43 – There’s a number of ways to do student authentication (oauth2) with Jupyter Hub 8:30 – Peter has been using this set up since the summer of 2018 9:08 – With Covid-19, now is a good time to be able to do remote work 9:49 – You don’t need to know much about how Jupyter Hub works to deploy it 10:59 – Sometimes your task is to deploy a code base you didn’t write 11:32 – Why you might want to self host things like Jupyter Hub and GitLab 12:22 – Initially Peter tried to code up his own solution but then found Jupyter Hub 13:39 – You need a Linux server, some type of auth, Python and a database to get going 14:42 – The server is hosted on a $40 / month DigitalOcean droplet 17:22 – You need a pretty beefy server even with relatively few users for Jupyter Hub 17:59 – There’s nginx acting as a reverse proxy in front of a Tornado web server 18:44 – Configuring nginx was Peter’s biggest pain point since he teaches Python not DevOps 19:25 – It was important to set up SSL certificates, he used Let’s Encrypt with Certbot 20:12 – From teaching Python to becoming Mr. DevOps (deployment is hard) 21:15 – The server is using Ubuntu 18.04 LTS and Peter connects using PuTTY on Windows 22:12 – Peter will try the next Ubuntu LTS in fall’s semester 22:42 – The server is running Python 3.7 to keep it compatible with Jupyter Hub 23:32 – Each quarter gets a brand new server that’s set up from scratch 23:54 – From manually configuring everything to Bash scripts to using Ansible for automation 25:25 – The Ansible set up has been used once so far 26:03 – Students never directly SSH into the server but it’s good to keep it locked down 27:05 – Peter can log into the server and check out his student’s work 27:38 – The next step for Peter’s automation project is to scale up the server’s size 29:25 – Other professors are using and planning to use this Jupyter Hub set up 30:24 – Peter usually updates the server’s packages when he turns it off to upgrade it 31:59 – Most instructors are using the college’s built in learning management system 33:04 – Jupyter Hub takes care of presenting logs and errors in its admin dashboard 34:05 – The college’s LMS is still the source of truth for looking at a student’s work 34:54 – Backups are done manually but Peter isn’t too worried about losing the data 35:47 – The Jupyter Hub is accessible on the public internet but no downtime alerts are used 37:13 – Best tips? Deployment is tough but it’s ok to make mistakes as you go 38:13 – Peter learned from many mistakes such as pushing SSH keys to GitHub and more 40:18 – After all is said and done, despite it being tough, it works well for his students 41:16 – You can find Peter on GitHub, Twitter and his blog, he also has an open source book Links 📄 References https://github.com/jupyterhub/jupyterhub https://jupyter.org/ https://numfocus.org/ https://www.putty.org/ https://professorkazarinoff.github.io/jupyterhub-ENGR114-2019Q4/ https://github.com/ProfessorKazarinoff/jupyterhub-ENGR114-2020Q1 ⚙️ Tech Stack python → ansible → digitalocean → lets-encrypt → jupyter → nginx → open-source → sqlite → ubuntu → 🛠 Libraries Used https://github.com/tornadoweb/tornado Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, Brett Kromkamp talks about building an open source knowledge management tool with Flask and Python. It’s running on a single $10 / month Linode server and has been up and running since February 2020. It already has 200+ users. Brett goes over how a graph database model fits perfectly for modeling knowledge, an open source Topic map library he wrote in Python, the power and simplicity of Flask, the value of keeping an eye on your external dependencies and a whole lot more. Topics Include 1:37 – Contextualise is a side project that scratches Brett’s own itch 3:01 – It uses a Topic map which is a type of graph model to model a lot of things 3:56 – A Topic map is a perfect fit for modeling any type of knowledge 6:04 – You can visualize a graph model by thinking about having circles connected by lines 6:50 – Graph databases can help you naturally model very complicated relationships 10:15 – Topic maps very much align with how you likely think about modeling data in real life 12:15 – Brett used a Topic map to build a CMS back in ~2006 13:14 – The current Topic map engine Brett built for this project came about in ~2016 14:43 – Both Topic DB and Contextualise are open source on GitHub 15:46 – Motivation for choosing Flask over Pyramid and Django 21:07 – Contextualise has been up and running since February 2020 22:10 – Google Analytics is used for very basic traffic metrics, not personalized tracking 24:39 – Handling file uploads using the code supplied in Flask’s documentation 28:43 – When files are written to disk, they go into a specific directory 29:14 – The application uses server side rendered Jinja templates with a bit of JavaScript 32:13 – Drawing out the graph map using the vis.js charting library 34:15 – There’s a few more visualization features coming to Contextualise soon 36:25 – Motivation for choosing Gunicorn and nginx over other app / web servers 38:54 – Brett doesn’t use Docker, but folks have contributed Docker support in his project 40:36 – It’s hosted on Linode using Ubuntu 18.04 with PostgreSQL and nginx w/ Let’s Encrypt 41:10 – Topic DB is an abstraction on top of PostgreSQL, not a dedicated graph database 42:46 – Topic DB uses the Psycopg2 library to connect to PostgreSQL 43:12 – The biggest problem with Psycopg2 was getting it running on MacOS 45:09 – On Windows you can use WSL, but ideally Docker solves this problem all together 45:47 – Let’s Encrypt’s Certbot tool was used to configure SSL certificates 46:53 – Everything is running on a single server with 2 GB of memory, 1 CPU core a 50 GB SSD 48:15 – Hearing stories about deploying everything to a single server is great 49:19 – About 200 users are signed up with 80+ active weekly users, the server is fine 50:27 – Linode makes it easy to vertically scale up if needed 51:21 – The server was set up by hand while following a bunch of DigitalOcean guides 52:38 – You can restart Gunicorn in about 500 milliseconds when not using Docker 53:18 – Brett breaks down a custom Python script he uses to deploy things 55:23 – Besides handling SSL certificates, nginx also serves static files 56:10 – Dealing with secrets using an INI file that’s not commit to version control 57:53 – An environment file would be pretty similar 59:20 – Flask-Security-Too is used because the original Flask-Security was abandoned 59:20 – His app sends email out but he uses his ISP’s SMTP servers to send it 1:04:16 – Logging is done with Flask’s logger, along with periodic manual lookups for errors 1:05:43 – Nick chimes in with a way to get notified by email if a Flask error occurs 1:06:57 – Topic DB is well tested but the Flask app has no tests at the moment 1:07:55 – Backups are handled with Linode’s automated server backups 1:09:22 – Brett really likes Linode 1:11:07 – Linode has alarms you can set up to get notified if your server gets too busy 1:11:45 – There’s currently nothing hooked up to do a healthcheck on the site itself 1:12:40 – Best tips? Use tools you can be productive with & be mindful of your dependencies 1:15:57 – There’s only a handful of external dependencies in Contextualise 1:17:17 – 21 Python files and about 4,200 lines of code make up the Flask app 1:18:00 – The Topic DB project has 27 Python files with 3,000+ lines of code 1:18:49 – Python allows you to write fewer lines of code compared to some other languages 1:21:07 – You can find Brett on GitHub, Twitter and he has his own site too Links 📄 References https://cerpus.com (the company Brett works for) https://en.wikipedia.org/wiki/Database_model#Graph_model https://en.wikipedia.org/wiki/Topic_map https://flask.palletsprojects.com/en/1.1.x/patterns/fileuploads/ https://timeline.knightlab.com/ https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-gunicorn-and-nginx-on-ubuntu-18-04 https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-18-04 https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-18-04 https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04 ⚙️ Tech Stack flask → python → lets-encrypt → linode → nginx → open-source → postgres → ubuntu → 🛠 Libraries Used https://github.com/un33k/python-slugify https://visjs.org/ https://gunicorn.org/ https://www.psycopg.org/ https://github.com/jwag956/flask-security https://github.com/sqlalchemy/sqlalchemy Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, Ty Cooper goes over building a cryptocurrency powered e-commerce store with Flask and Python. It’s hosted on Firebase along with PythonAnywhere and has been up and running since October 2019. Ty talks about the challenges of accepting cryptocurrency (specifically Ethereum), how he hosts the site using Firebase and PythonAnywhere, why he chose those services, the value of end to end tests using Selenium, the benefits of working with someone else instead of trying to do everything alone and more. Topics Include 1:02 – Switching gears and making a different site after running into a road block 2:36 – Motivation for using Flask over Django and other frameworks 4:27 – Flask makes it easy to get into without needing to open multiple cans of worms 4:54 – Would you use Flask again today if you had to re-write the site from scratch? 6:42 – Flask manipulates data but the database and storage is hosted on Firebase 8:00 – Flask-JWT-Extended helps deal with token based auth for his Flask API back-end 8:41 – The app is API based with a VueJS back-end because Ty likes JavaScript and VueJS 10:21 – Challenges for trying to sell products with cryptocurrency 11:25 – The web3.py library helps interact with the Ethereum blockchain 11:51 – Building an email newsletter sender with Flask-Mail 14:23 – PythonAnywhere is used to host the Flask app and comes with logging 15:07 – Twilio’s API is used to send Ty text messages if errors occur 15:46 – What Firebase is and how it helps run a few things in development and production 17:28 – Why use Firebase and PythonAnywhere vs using Heroku or hosting your own server? 18:21 – About $12 / month to host 3 different apps on PythonAnywhere 19:02 – You configure your CNAME DNS records to point to PythonAnywhere 19:29 – Firebase directly hosts all of the front-end files such as JavaScript and CSS 20:22 – PythonAnywhere gives you free SSL certificates 20:28 – Ty hosts 50+ sites through PythonAnywhere and gets 10k+ monthly visitors 22:15 – Learning to deploy your own apps is rewarding but there’s a lot to learn 23:38 – What it looks like to deploy code from development to production 25:14 – Automated end to end testing with Selenium 26:50 – Getting banned on Tinder for using Selenium to make too many automated requests 27:04 – Mozzarella cheese 28:16 – Dealing with secret keys through environment variables using PythonAnywhere 29:18 – Database backups are done once a month 30:59 – Product images are uploaded directly from the front-end to Firebase 32:48 – Placeholders are used to help show graceful errors if something goes down 34:54 – Firebase and PythonAnywhere will notify Ty automatically if the site goes down 35:40 – Best tips? Learn by doing and don’t worry about making mistakes 37:30 – One mistake Ty made was developing the project alone, a 2nd set of eyes is helpful 39:28 – Having someone else around really helps with keeping you on track 41:01 – Find Ty on Instagram along with Twitter, Reddit and Youtube Links 📄 References https://en.wikipedia.org/wiki/Cryptocurrency https://www.youtube.com/channel/UC-QDfvrRIDB6F0bIO4I4HkQ (Pretty Printed) https://ethereum.org/ https://www.twilio.com/ https://github.com/SeleniumHQ/selenium https://en.wikipedia.org/wiki/WebP ⚙️ Tech Stack flask → python → vue → firebase → python-anywhere → stripe → twilio → 🛠 Libraries Used https://flask-jwt-extended.readthedocs.io/en/stable/ https://github.com/axios/axios https://github.com/ethereum/web3.py https://github.com/mattupstate/flask-mail Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, Scott Johnson talks about building a Covid-19 case tracking / anonymous survey / information site using Ruby on Rails. It’s all hosted on DigitalOcean using the Hatchbox.io service (it’s load balanced across 2 servers). We talked a lot about using Jumpstart Pro, Hatchbox, working with team mates in the DOD and iterating on a project super quickly. The app’s initial release was created in less than 2 days and it’s currently serving 100,000+ visitors a day. Topics Include 1:23 – The app was initially released after 2 days of part time development 2:36 – The data is pulled in from the Johns Hopkins CSV data on GitHub 3:14 – Scott’s partner for this app works for the DOD (Department of Defense) 3:43 – The site is much more than just an infection counter for countries 4:45 – All of the crowd sourced data is anonymous and encrypted at rest 5:15 – They are using FDA protocols and everything is signed off by a doctor 6:00 – NY state has 15,000+ cases and it’s very hard to get tested 6:47 – Scott has been using Ruby on Rails since 2007, that’s why he used Rails 7:32 – Getting an MVP up and running quickly 8:30 – Jumpstart Pro was used to create the base application 9:06 – You can use Jumpstart Pro for more than a typical SAAS app 11:12 – Jumpstart Pro gives you a base application to build upon using Rails best practices 11:54 – Jumpstart Pro uses TailwindCSS and it looks nice out of the box 12:28 – Websockets are not being used, they are trying to keep things simple 13:19 – A rake task uses the Mechanize gem to grab the data from GitHub on a cron job 13:47 – How do the Coronavirus numbers compare to Worldometers? 14:44 – Doctor validated crowd sourced reports will help provide per county infection rates 15:39 – A few notable gems that were helpful for building this app 16:28 – Hatchbox is used to deploy the application ($99 / month for the cluster edition) 16:43 – Simple Analytics is used for basic analytics ($19 / month) 16:59 – Honey Badger is used for error reporting (~$20 / month) 17:08 – Google Suite is used for email and other business things 17:12 – Tuple.app is used for pair programming sessions on MacOS systems ($25 / month) 18:17 – All in all after server costs ($100 / month) , it’s about $300 / month for everything 18:35 – Not all side projects are close to free, these tools cost money 19:02 – Why not just throw it all on a single $20 / month DigitalOcean server? 19:50 – The Rails core team puts the burden of deployment on each developer 20:22 – Hatchbox helped remove some of the pain of managing servers yourself 21:40 – 650+ commits in about 10 days of development, so iteration speed is fast 21:52 – ~11,000 lines of code, most of which are separate from Jumpstart Pro 22:21 – Jumpstart is a base application to work off of, it’s not a gem you install 24:10 – Docker is not being used in development or production 25:17 – Something funky in happening with Rails 6 and code reloading in development 26:43 – Jumpstart uses Webpacker and Yarn 26:52 – PostgreSQL, Redis and Sidekiq are being used 27:03 – Sidekiq is used to move encrypted data from server 1 to server 2 29:06 – Having a partner in the DOD / FDA helped keep Scott on track with data security 29:24 – Hosting is being moved to Google Cloud for storing more sensitive information 29:56 – What exactly is clustered mode with Hatchbox? 30:58 – It took about 30 minutes to deploy everything to GCP thanks to Hatchbox 31:48 – Most side projects fall apart when it comes time to deploy things 33:20 – How Hatchbox manages project isolation since you can deploy multiple apps 34:22 – This Covid-19 site has 2 web servers that are load balanced to serve traffic 34:49 – Currently Hatchbox doesn’t appear to handle rolling restarts 35:41 – Hatchbox sets up nginx as the load balancer and also uses Let’s Encrypt (SSL certs) 36:06 – How code gets from development to production 37:33 – Creating a deploy script to help remove human errors from deploying 39:30 – The risk wizard feature was rolled out quickly to determine your risk factor 40:16 – The current situation in Italy is a tragedy 41:26 – Data is backed up daily but most of it can be recreated from source files 42:28 – Hatchbox and Jumpstart don’t help you out with backing up your data 43:07 – Deployment is hard with Ruby on Rails 44:45 – Basecamp, Shopify, GitHub and others all deploy things much differently 45:06 – Hatchbox helps, but it’s expensive and does lack some useful features 45:47 – There’s always Heroku, but that’s also expensive with a opaque pricing model 46:49 – Uptime Robot is being used for free notification if the site goes down 47:36 – It would be cool if Hatchbox hooked into each hosting provider’s monitoring APIs 48:58 – Dealing with secrets in development and production using environment variables 49:33 – Scott is the primary developer of this project 49:58 – A really brilliant way to handle internationalization by leveraging restaurants 52:23 – Best tips? Get your workflows right and then automate them with scripts 54:02 – Also, focus on keeping all of your URL shortcuts in 1 place for easy access 54:51 – Should these URLs be in your admin back-end? Maybe, maybe not 55:28 – Testing is important but it doesn’t always need to be classical tests 56:09 – Rubocop isn’t being used, but it’s a good tool once properly configured 58:29 – Scott’s code is very organic and not perfect but he ships working things quickly 59:08 – Scott is a prolific blogger and most posts are based on what he’s learning at the time 1:01:39 – Learn more about Scott by reading his blog and following him on Twitter / GitHub, there’s also his other projects at https://netlabeler.com/ and https://scottsfeedfinder.com/ Links 📄 References https://coronavirus.jhu.edu/map.html https://github.com/CSSEGISandData/COVID-19 https://en.wikipedia.org/wiki/United_States_Department_of_Defense https://jumpstartrails.com/ https://www.worldometers.info/coronavirus/ https://simpleanalytics.com/ https://gsuite.google.com https://tuple.app/ https://guides.rubyonrails.org/v4.0/command_line.html ⚙️ Tech Stack rails → ruby → digitalocean → hatchbox → honeybadger → lets-encrypt → nginx → postgres → redis → simple-analytics → uptime-robot → tailwindcss → 🛠 Libraries Used https://github.com/sparklemotion/mechanize https://github.com/zombocom/wicked https://github.com/monterail/zip-codes https://github.com/loureirorg/city-state https://github.com/awesome-print/awesome_print https://github.com/binarylogic/authlogic https://github.com/puma/puma https://github.com/rails/webpacker https://github.com/mperham/sidekiq https://github.com/phusion/passenger Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, Seth Black goes over building a webcomic platform with Flask and Python. It runs on a multi-node Docker Swarm cluster on DigitalOcean and has been up and running since 2010. It gets up to 20,000+ visitors a day during traffic spikes. Seth talks about the value of using nginx to cache Flask responses to keep the site from getting a hug of death from Reddit. He also talks about using nginx as a load balancer, running Docker Swarm, maintaining a web app that has been running for 10+ years, creating a flexible and robust deployment pipeline with GitHub Actions as a solo developer and a whole lot more. Topics Include 2:07 – Experiences with using Flask to build the site for the last 10+ years 2:43 – Traffic ranges from 20,000+ visitors a day to hundreds per month 4:54 – A simple 5 second cache can save your server from a huge traffic spike 5:34 – Motivation for using Flask after having used many different frameworks / languages 6:48 – It’s split into 2 services, one for the main app and one for analytics 7:51 – The main app is about 1,000 lines of Python code with 15 database tables 8:35 – Flask-Cache is used to cache the response of a few high traffic views 8:55 – Each service has its own git repo that is independently deployed 9:31 – Responses are sent out as server rendered templates using Jinja 2 10:30 – A custom admin dashboard was built up over the years and it was worth it 13:14 – Reducing page load speeds by lazy loading comic images as you scroll down 14:29 – Websockets are being used in the admin to auto-save content in a real-time editor 16:27 – MariaDB is being used as the main database and here’s why 18:15 – Search isn’t implemented yet but it’s the number 1 user feature request 19:25 – Having search allows you to get good insights on what folks are looking for 20:07 – Full page caching is done with the free and open source version of nginx 21:42 – gunicorn is being used for the Flask app server and it works fantastically 21:57 – Everything is inside of Docker, running in a Swarm cluster 22:47 – Docker Swarm was chosen over Kubernetes because Kubernetes was overkill 25:08 – 2 web app servers that are load balanced, 2 DB servers, Redis and a load balancer 25:34 – Celery isn’t being used but Redis is doing a few things 26:27 – Why Seth moved from AWS to DigitalOcean over the years 29:17 – The load balancer is on a 2 GB of memory / 50 GB SSD / 2 virtual CPUs server 29:36 – The app servers have 4 GB of memory / 4 virtual CPUs and a bit more SSD disk space 29:45 – The database server has 8 GB of memory / 4 virtual CPUs and a 128 GB SSD 29:53 – Everything is running smoothly and purs like a kitten with various amounts of traffic 30:18 – Seth uses the same database server for a few of his other sites 31:15 – A combo of Ubuntu 16.04 and Ubuntu 18.04 are running with intent to upgrade 32:04 – Seth’s upgrade process for when it comes time to upgrading Ubuntu to 20.04 33:50 – DigitalOcean’s load balancer wasn’t around when all of this was set up 35:06 – Everything was installed by hand on the servers but Seth likes Terraform and Puppet 36:32 – You don’t always need to upgrade to the latest tool that came out today 37:02 – Over time, the app was updated from Python 2.6 to 3.4+ which was painful 37:58 – How do you stay motivated working on a project that’s been going for 10+ years? 40:49 – Walking us through deploying code from development to production 43:14 – GitHub Actions are used for CI / CD and it’s all automated with a single git push 44:06 – Secrets are stored in DigitalOcean Spaces and then are built into images on demand 46:17 – Error reporting and logging are handled through flat file logs that get backed up 46:50 – Newsletters are sent through a semi-manual mail merge strategy with Google Sheets 48:43 – An RSS feed is also available and Pinterest provides a decent amount of traffic 49:45 – The database is backed up nightly and 30 days worth of backups are saved 50:24 – All of the comic assets are backed up on DropBox and the code is on GitHub 50:44 – If the servers were to get forcefully rebooted, some of them recover automatically 51:36 – Various DigitalOcean alarms are set up to do email notifications if things go bad 52:22 – Reddit is weird sometimes (the hug of death or down voted into oblivion) 53:11 – Using DigitalOcean and his comic reader community for determining site healthiness 54:55 – Best tips? Focus on doing something you’re passionate about, it really helps 55:39 – Picking the right tool for the job is an art form and lean on trusted tools 56:27 – One of Seth’s biggest mistakes was trying to build something fancy initially 57:34 – Another thing was chasing / building weird things that his users didn’t ask for 59:04 – Check out Seth’s webcomic - he’s also on GitHub and InstaGram Links 📄 References https://theoatmeal.com/ https://xkcd.com/ https://www.nginx.com/products/nginx/caching/ https://developers.google.com/gsuite/solutions/mail-merge https://rss.com/blog/how-do-rss-feeds-work/ ⚙️ Tech Stack flask → python → aws → digitalocean → digitalocean-spaces → docker → github-actions → lambda → lets-encrypt → mysql → nginx → redis → serverless → swarm → ubuntu → 🛠 Libraries Used https://github.com/thadeusb/flask-cache https://gunicorn.org/ Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, Francisco Barcena goes over building an internal food order processing system for the city of Los Angeles, CA with Flask and Python. It’s hosted on a single DigitalOcean server and has been up and running since 2016. Francisco talks about developing a freelance project for a client as a solo developer, scraping the USDA’s site, working with Flask-SocketIO to create a real-time ordering service and getting everything up and running on a single server without using Docker, CI or configuration management tools. Topics Include 3:15 – Replacing spreadsheets with a web app as the single dev on this project 4:12 – It was better off making a hand rolled solution instead of using an existing platform 5:08 – Motivation for using Flask instead of Django and other frameworks 5:56 – It’s a single monolithic app that was built up in phases / stages 6:20 – The Scrapy library is used to scrape CSV links from the USDA site 7:01 – The app uses server rendered Jinja 2 templates with a few ajax calls 8:35 – Websockets with Flask-SocketIO is used to provide a real time order placing page 11:28 – Caching some of the daily values with Flask-Cache 12:48 – SQLAlchemy with PostgreSQL is used for the database back-end 12:59 – Tracking down database session issues by disabling an accidental DEBUG flag 13:59 – gunicorn is being used as the Flask app server 14:21 – DigitalOcean is being used to host the app 15:15 – Docker isn’t being used at the moment since he didn’t know about it ~3 years ago 17:52 – The DigitalOcean server has 1 GB of memory and 2 CPU cores 19:05 – Ubuntu 16.04 LTS is the Linux distro being run on the server 19:24 – Configuration of the server was done by hand 20:14 – Process management is done with a “YOLO but call me” strategy 21:21 – nginx is set up with Let’s Encrypt and Certbot to secure the site over HTTPS 21:53 – nginx is also used to serve static files (images, CSS, etc.) directly 22:17 – Flask-Mail is used to send out emails such as password passwords 23:06 – The app itself doesn’t handle payments in real time, it takes food orders 23:46 – The deployment process is to push to BitBucket and pull it from the server 24:16 – Secret values aren’t in version control 24:53 – There’s no automated tests, but things are extensively tested by the few end users 26:55 – The server and data is backed up daily with DigitalOcean’s droplet backup feature 28:21 – Best tips? Have automated tests for your code base 30:40 – Find a bunch of info about Francisco on his personal site (also built in Flask) Links 📄 References https://scrapy.org/ https://www.usda.gov/ https://en.wikipedia.org/wiki/Linear_regression https://www.postgresql.org/ https://nginx.org/en/ https://github.com/certbot/certbot ⚙️ Tech Stack flask → python → digitalocean → lets-encrypt → nginx → postgres → ubuntu → websockets → 🛠 Libraries Used https://www.chartjs.org/ https://github.com/miguelgrinberg/Flask-SocketIO https://github.com/thadeusb/flask-cache https://gunicorn.org/ https://www.sqlalchemy.org/ https://github.com/mattupstate/flask-mail Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, Adam Conrad goes over building a Slack bot service with Phoenix and Elixir. It’s hosted with Gigalixir for about $50 / month and has been up and running since mid 2018. Adam talks about the wonderful Elixir community along with how Elixir helps him manage a bunch of concurrent tasks for his service. He also goes over a number of features that you get from using Gigalixir to host your Elixir projects. Topics Include 1:31 – Adam is the sole developer but he has a friend who helps him market the service 2:38 – The Slack bot asks you a number of questions, such as your music preference 6:01 – The public site uses Phoenix and the bot is written in Elixir 6:25 – What made Elixir a perfect fit for this application? 7:38 – Handling concurrency through tasks using Elixir’s standard library 9:16 – Several thousand questions are answered per day 10:23 – It’s all done through Slack webhooks using the Elixir-Slack library 11:28 – Slack bots are easy to start using but it gets complicated pretty quickly 12:26 – Unix timestamps vs Gregorian seconds 13:39 – The Elixir forums and Slack channel are great resources to get Elixir help 15:13 – Are you using the latest version of Phoenix? Not yet 17:50 – Phoenix is nice because it gently guides you into creating well structured apps 19:19 – The stripity_stripe library is used to handle Stripe payments 19:52 – The awesome-elixir repo on GitHub has a great list of libraries 21:25 – It was hard to find a good well maintained authentication library for Elixir 22:50 – The Elixir-Slack library was really helpful for Adam’s project 23:59 – Everything is stored in a PostgreSQL database 25:02 – Gigalixir is used for hosting and it’s well optimized for running Elixir in production 25:56 – Gigalixir has really good support 26:25 – Gigalixir’s team is mostly composed of ex-Google / ex-Stripe employees 27:50 – Gigalixir also has zero down time deploys for most deploys 28:42 – nginx isn’t being used and Gigalixir handles TLS too 30:20 – The push for HTTPS everywhere and Let’s Encrypt being easy but not that easy 31:15 – Accessing your logs on Gigalixir by tailing your logs on the command line 32:31 – Adam pays about $50 a month for everything on Gigalixir 33:59 – Metrics related to server health can be found in the Gigalixir dashboard 34:14 – Surviving the front page of HackerNews with an Elixir app on a hobby instance 35:22 – Depending on user feedback for error reporting 36:28 – It’s super nice to have a compiler warn you about flaws in your code 36:57 – Gigalixir lets you git push your code to get it from your dev box into production 38:04 – Code is also pushed to GitHub mainly for backup purposes 38:23 – Code is tested locally before being pushed but testing a Slack integration is hard 42:18 – Putting an app on the Slack app marketplace requires human review 44:30 – Managing secrets with Gigalixir is done through their CLI, similar to Heroku 45:03 – Database backups happen daily but the interval on Gigalixir can be configured 47:26 – Email alerts are sent out by Gigalixir if your app happens to go down 48:59 – Best tips? Leverage the Elixir community if you get stuck 49:24 – Enjoy a language that a bit of novelty to it since it’s still relatively young 50:37 – Getting help for basic things from the creator of Elixir on IRC 51:47 – Constantly looking at older code to see how it could be refactored 52:57 – Using the Credo library to help analyze your code base for best practices 55:07 – Check out their Slack app, follow Adam on Twitter @theadamconrad and GitHub Links 📄 References https://www.phoenixframework.org/blog/the-road-to-2-million-websocket-connections https://api.slack.com/messaging/webhooks https://adamconrad.dev/blog/verifying-request-signatures-in-elixir-phoenix/ https://elixir-lang.slack.com/ https://github.com/h4cc/awesome-elixir https://www.postgresql.org/ https://twitter.com/josevalim ⚙️ Tech Stack phoenix → elixir → gigalixir → postgres → slack → stripe → 🛠 Libraries Used https://github.com/BlakeWilliams/Elixir-Slack https://github.com/code-corps/stripity_stripe https://github.com/rrrene/credo Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, Nick Gregory walks us through what it’s like building Vortex Weather which uses Flask and Python to help understand the weather. It’s hosted on bare metal servers running in a colo on top of a self managed Kubernetes cluster. Nick goes into detail on having to manage 750+ GB of data that needs to be recycled every 2 days and having his own servers with a combined, 24 physical CPU cores, 128+ GB of RAM and 10+ TB of disk space. The app is open source and it’s his side project, so he also talked about tips on how to manage his time and enforce self imposed deadlines. Topics Include 1:59 – Weather data can come from a number of different sources 3:01 – 750+ GB of data is cycled every 2 days and it needs to be queryable in ~200ms 3:57 – This is Nick’s side project and he’s the sole developer which has its challenges 5:13 – The app is open source on GitHub 5:26 – Motivation for using Python and Flask 6:53 – Picking gunicorn over uwsgi, mainly for ease of deployment 9:32 – The web front-end is a monolith but the data processing happens in a different service 10:55 – Looking at Go for more performance on the math heavy calculations 12:15 – Ingesting data is done through Kubernetes cron jobs and it’s backed by PostgreSQL 15:00 – PostgreSQL is really amazing and works great as a queue back-end 16:34 – PostgreSQL was originally used for the 750 GB of data but then it was moved to S3 18:41 – The files on S3 are 10+ GB in size (each) 19:36 – The files are a huge series of floats in a binary format 21:02 – PostgreSQL ends up storing a little bit of meta-data that’s used to get data from S3 21:19 – It was all moved out of PostgreSQL because… 24:03 – Before the S3 change there were over 2.5 billion rows being saved to PostgreSQL 25:10 – Seriously, PostgreSQL is really good 25:49 – Visualizing weather in a radar view requires a massive amount of computing power 28:23 – Predicting the weather in the short term future based on clouds and wind 30:54 – Docker is being used in development and it’s making things a lot easier 32:01 – Ubuntu is being used as a base OS for the Docker images 33:28 – Traefik sits in front of gunicorn to do TLS termination and act as a reverse proxy 35:10 – Traefik is rock solid but it’s not a complete drop in replacement for nginx 38:27 – (2) servers are sitting in a colo for now, mainly for cost and performance wins 40:13 – The TL;DR on what a colo (colocation) is and what it’s like owning your own hardware 41:27 – Server 1 has 8 CPU cores, 96 GB of RAM, 6 TB of HDD space in RAID-5 and a 2 TB SSD 42:10 – Server 2 is a Threadripper with 16 CPU cores, 64 GB of RAM and (2) 2 TB NVME SSDs 42:53 – How much storage could you fit? 10+ SSDs plus the PCI slots on the motherboard 43:47 – Getting similar capacity and performance with AWS would be a lot more money 44:49 – Nick runs a Debian based distro called Proxmox 45:55 – A good GUI is one of the selling points for Proxmox, even though it’s on a server 46:41 – Kubernetes is self managed but it was configured with Rancher’s RKE 48:49 – Prior to finding RKE, he almost went with Swarm due to Kubernetes’ complexity 49:54 – Currently there’s no error reporting set up, but Sentry.io is on the horizon 51:24 – Deploying code from development to production 51:34 – The weather app is on the Docker Hub at kallsyms/wx_explore 53:22 – CI isn’t being used now but a self hosted Gitlab instance might happen in the future 54:09 – As long as the kubeconfig stays out of CI, then it’s all good 55:43 – How are database migrations dealt with? By avoiding them since they’re not needed 58:56 – What about backing up the database? It’s also not needed since it’s all recreatable 59:57 – The weather tool is made for both regular people and meteorologists 1:03:34 – StatusCake’s free tier is being used to do uptime monitoring 1:05:55 – Best tips? Learning to self manage your time and break down problems 1:07:01 – Learning best practices and tips on how to optimize Docker performance 1:08:05 – Keeping yourself accountable for hitting self imposed deadlines 1:11-20 – You can find Nick on Twitter @kallsyms and on GitHub too Links 📄 References https://www.epa.gov/ceam/meteorological-data https://www.ncdc.noaa.gov/ https://instagram-engineering.com/what-powers-instagram-hundreds-of-instances-dozens-of-technologies-adf2e22da2ad?gi=af837c58e3b https://en.wikipedia.org/wiki/List_of_common_coordinate_transformations https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range https://darksky.net/forecast https://en.wikipedia.org/wiki/Colocation_centre https://en.wikipedia.org/wiki/Standard_RAID_levels https://pve.proxmox.com/wiki/FAQ https://rancher.com/docs/rke/latest/en/ https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#merging-kubeconfig-files https://en.wikipedia.org/wiki/Convective_available_potential_energy ⚙️ Tech Stack flask → python → cloudflare → debian → docker → kubernetes → open-source → postgres → s3 → sentry → statuscake → traefik → 🛠 Libraries Used https://gunicorn.org/ https://pypi.org/project/pq/ https://github.com/python/cpython Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, Eric Meadows-Jönsson goes over building the Hex.pm site and surrounding services which was built using Phoenix and Elixir. It’s hosted on Google Cloud using their managed Kubernetes service (GKE). It’s been up and running since early 2014. Eric talks all about creating a highly available package manager for Elixir, the value in breaking out independent services, working with Google’s managed Kubernetes service, keeping administrative tasks simple, his struggles with Stripe’s SCA and more. Topics Include 1:02 – The hex.pm site is a community effort with a few core contributors 1:52 – 5 reqs / second for the main site and 140 reqs / second for the repo side of things 2:58 – A majority of the traffic is pulling down packages with mix 3:17 – Eric started working on the Hex client in late 2013 before Elixir 1.0 3:51 – Hex launched with both the CLI and web front-end at the same time 4:22 – Motivation for using Elixir to build an Elixir package manager 4:43 – For the site, Plug was used initially because Phoenix wasn’t out yet at the time 5:00 – They try to keep the site updated to use the latest stable release of Phoenix & Ecto 5:16 – Why Eric decided to open source the platform on GitHub 5:38 – The Hex site’s source code helped Nick learn a bit more about Elixir and Phoenix 6:01 – Trying to use the latest best practices so the site is a good working example 6:45 – The site is mostly a monolith but there are a few dedicated services 7:17 – https://diff.hex.pm/ is one of the separate services along with billing and the docs 7:52 – Why did they decide to make the billing service closed source? 9:45 – There’s currently 10,000+ packages on Hex with 60k+ different versions combined 10:08 – Less than 10 GB of data makes up everything that’s stored on S3 for all packages 11:30 – The benefits of splitting up a few services outweigh the pain points 12:35 – Johanna Larsson (@joladev) initially created the Hex diff service 13:16 – The diff service’s search feature is powered by Phoenix Live View 13:47 – The main Hex website is mostly server rendered templates with a touch of JavaScript 14:35 – Breaking out smaller services lets you experiment with new technologies 15:08 – There is no admin dashboard web UI, it’s all powered by version controlled scripts 17:01 – PostgreSQL is the primary database 17:41 – Docker is used in production with the managed Kubernetes service on GCP 18:06 – There’s no need to cache anything because Phoenix is fast out of the box 19:15 – They rolled their own rate limiter with very little Elixir code 20:22 – Docker isn’t being used in development since their stack is so simple 21:00 – What made them choose Google Cloud (GCP)? Mainly Google’s managed Kubernetes 22:28 – (3) servers power everything, each having 2 CPU cores and 4 GB of memory 22:54 – When you install public packages from the CLI, you don’t hit the web API 26:12 – For private packages you do hit the web API but that’s going to change soon 28:39 – Fastly is their CDN and it lets you upload Varnish config files 29:23 – Fastly gives you unlimited and instant cache purges, most other CDNs limit you 30:11 – Amazon SES is used for transactional emails 30:59 – Rollback is used for error reporting and Stripe is being used for billing 31:05 – For logging and metrics, that’s all handled on Google Cloud and it’s decent 33:05 – Elixir Telemetry isn’t really being used now, but they might use it later 33:57 – Stripe SCA support is being worked on but it wasn’t fun to re-write everything for it 38:10 – What it’s like to push code from development to production 40:25 – Dealing with database migrations isn’t easy 43:16 – Putting your site into a read-only or maintenance mode during a migration 45:03 – Creating an Elixir library to toggle your app into read-only mode, make it so! 46:10 – DB backups are done through GCP, and they also use Tarsnap for S3 backups 47:33 – Health checks are in place with Google to get notified if a service goes down 49:54 – Best tips? Don’t start out with a fancy infrastructure, just get your app up and running 51:45 – Mistakes? It’s hard to test the Hex client (the CLI) and they are looking to improve that 53:22 – For testing, they mock out S3 with the local file system 55:14 – Hex.pm is open source on GitHub and you can find Eric on Twitter @emjii Links 📄 References https://diff.hex.pm/ https://en.wikipedia.org/wiki/Erlang_distribution https://www.tarsnap.com/ https://cloud.google.com/stackdriver ⚙️ Tech Stack phoenix → elixir → aws → docker → fastly → gcp → gke → github-actions → kubernetes → open-source → postgres → rollbar → s3 → ses → stripe → terraform → varnish → Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, Henry Popp goes over building a platform to help explore your personal network which was built using Phoenix and Elixir. It’s hosted on Google Cloud using a self managed Kubernetes cluster. It’s been up and running since September 2019. Henry went into great detail about the value of using a service oriented architecture, DDD, event driven design and running a self managed Kubernetes cluster. There’s a lot of great insights in this episode around general code design and scaling that apply to any web framework. Topics Include 2:11 – 4 developers are actively working on the project 2:50 – It’s been running in production since September 2019 3:13 – Motivation for using Phoenix and Elixir 4:26 – Henry started using Elixir back in late 2014 5:48 – Ditching Umbrella apps for dedicated services 7:35 – 6DOS is built on 6 independent git repos with a service oriented architecture 8:20 – A break down of what those 6 repos are and what they do 10:37 – Each service has its own independent database (Postgres, Neo4j, Elasticsearch) 11:21 – Neo4j is a graph database which is a great fit for their main data model 12:55 – How is Elixir support for Neo4j? 13:46 – Each service talks to each other through RabbitMQ events / notifications 15:43 – Walking through the request / response cycle when a visitor hits the site 17:04 – How did you arrive at this service oriented architecture? 18:33 – It’s easy to get Domain-driven Design (DDD) wrong initially 19:42 – Are Phoenix contexts being used? Nope 20:20 – Monoliths vs micro-services vs something in between and industry trends 20:56 – “Instantaneous complexity” 21:39 – Using an app skeleton project to help spin up new services quickly 23:23 – Using VueJS on the front-end with Webpack, but not through Phoenix 24:43 – Currently 6DOS doesn’t need websockets but that could change later 26:47 – Quite a lot of work happens in the background 27:37 – RabbitMQ handles queueing up all of the jobs 29:10 – Docker is being used in production, but not in development (yet) 29:38 – The work flow for starting everything up locally in development 30:52 – Everything is hosted on a self managed Kubernetes cluster on GCP 31:19 – (3) 2 core master nodes, (3) 2 core worker nodes and extra servers for the databases 32:24 – The self managed Kubernetes cluster was terrifying to set up initially 34:00 – Kubernetes is not a magic button you press to scale your application 35:15 – Auto-recovering from a CrashLoopBackOff error with Kubernetes 37:45 – Those 2 CPU core servers have 8 GB of RAM but the app isn’t using all of that 38:47 – Handling an interesting auto-scaling problem with Kubernetes 40:20 – Performing rolling restarts so there’s no down time for each deploy 40:41 – Dealing with restarting containers while an important action is happening 43:23 – Walking through the deploy process from development to production 43:34 – It starts with a self hosted Gitlab instance with automated CI 44:15 – On the other side, Keel takes over to automate deploying any services 45:12 – Helm is being used for a few things, but not everything 46:17 – Humans needing to accept the release happens within Keel’s UI 47:51 – Secrets are stored directly in the self hosted config repo with strict access rights 49:09 – Balancing your time between low level infrastructure vs app features 49:58 – Handling SSL certificates on the cluster with cert-manager 51:06 – Everything is behind a Cloudflare proxy too 51:20 – Dealing with database migrations when you have automated deployments 52:40 – Migrations get run as part of the app boot up process 54:24 – Design your software like a space ship 55:16 – Diagnosing errors with custom tasks and 3rd party tools 56:23 – No one can agree on how to format API JSON errors 57:32 – Elixir best practices are still being discovered, the future is bright 58:19 – An example of one of Henry’s open source Elixir tools (Pigeon) taking off 59:14 – All of the databases get backed up hourly 1:00:26 – Kubernetes really needs to be configured 1:01:16 – Rate limiting is currently being added to all of the services 1:03:07 – What about alerts if something goes down? It’s a digital notification bomb 1:03:36 – Using UptimeRobot as a second sanity check to make sure things are up 1:04:12 – Hung over at 6am out in the middle of the woods and your server goes down 1:04:55 – Using an external tool like UptimeRobot is worth it 1:06:16 – Timber.io is being used for logging but that will change soon 1:07:00 – Kubernetes’ Stern package helps with tailing logs across pods 1:07:44 – Henry isn’t a fan of Kubernetes’ web UI tools to manage the cluster 1:07:54 – Weave Scope was interesting but it used too much resources to run it 1:08:41 – Best tips? Don’t be afraid to break your code up into multiple repos 1:09:45 – If you’re not at that point yet, at least look into using contexts and DDD 1:10:24 – Developers as a whole are getting better over time 1:11:56 – Codedge is Henry’s consulting company and they are on GitHub / Twitter too Links 📄 References https://en.wikipedia.org/wiki/Six_degrees_of_separation https://neo4j.com/developer/cypher-basics-i/ https://en.wikipedia.org/wiki/Directed_acyclic_graph https://kubernetes.io/docs/concepts/workloads/pods/pod/ https://github.com/keel-hq/keel https://github.com/jetstack/cert-manager https://github.com/bitwalker/alpine-elixir https://github.com/wercker/stern https://www.weave.works/oss/scope/ ⚙️ Tech Stack phoenix → elixir → vue → cloudflare → docker → elasticsearch → gcp → gitlab-ci → grafana → kubernetes → neo4j → postgres → prometheus → rabbitmq → sentry → stripe → timber → uptime-robot → webpack → 🛠 Libraries Used https://github.com/florinpatrascu/bolt_sips https://github.com/bitwalker/distillery https://github.com/codedge-llc/pigeon Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.