In this episode of Running in Production, Julien Blanchard goes over building Smart Music which uses a combination of Rails, Phoenix and .NET Core. It has roughly half a million users and it’s all hosted on AWS with EKS, ECS and Elastic Beanstalk. It’s been up and running since 2016.

There’s around 20 developers working on the project. We talked about managing git repos with a few apps, TDD, using GraphQL with Phoenix, contexts, multiple databases with Rails, InfluxDB, GitHub Actions and tons more.

Topics Include

2:41 – Roughly half a million users are on the platform (~1.5k requests a minute at times)
3:27 – What Rails, Phoenix and .NET Core are being used for
5:38 – End users of the site interact with the Rails app and .NET Core is for authentication
6:10 – It’s an API back-end driven app and React / EmberJS is used on the front-end
6:35 – Motivation for using Phoenix and Elixir for the data ingesting app
9:28 – About 20 developers work on all of the different parts of the site
9:55 – Organizing the git repos for each of the apps
10:34 – The back-end code has many tens of thousands of lines of code
11:04 – TDD is something their company practices and they like it a lot
12:24 – A JS front-end makes sense for this app since the UI is live and dynamic
13:17 – Trying to visualize a live sheet music application that helps you learn notes
14:02 – Maybe Phoenix LiveView could have been used, but they prefer what they chose
14:33 – The TL;DR on GraphQL and why in this case it works better than a RESTful API
17:55 – Docker isn’t being used in dev, but Kubernetes is being used in production
18:29 – PostgreSQL, InfluxDB and Redis are used to manage the data and for caching
19:32 – They knew from the start that InfluxDB would be needed to store the time data
20:33 – Redis is being used as a cache through AWS ElastiCache
21:49 – nginx is sitting in front of the Rails application with Elastic Beanstalk
22:44 – Motivation for picking AWS over Google Cloud and other providers
23:40 – AWS Aurora is being used to manage PostgreSQL
24:51 – They are using the Rails 6.x feature to select multiple databases
25:33 – Rails is very nice when it comes to getting community driven features merged in
26:08 – Julien also really likes Phoenix and here’s how they use contexts
28:50 – File uploads are sent directly to S3 using the ex_aws Elixir library
30:02 – For Kubernetes, they are using the managed EKS service from AWS
31:07 – (2) pretty beefy boxes with 8 GB of memory power the EKS cluster (overkill)
31:36 – They are still feeling out the resource usage of their services
32:18 – (20)’ish EC2 instances power the Elastic Beanstalk set up for the Rails app
32:54 – CloudFront is being used as a CDN for book covers but not much else
33:38 – Walking us through a code deploy from development to production
34:46 – Getting rid of Jenkins is the next step but GitHub Actions is a bit insecure currently
35:49 – GitHub Actions is a great tool and it’s being used for more than just CI
36:44 – You can use GitHub Actions to run tasks periodically (separate from git pushes)
37:27 – Dealing with big database migrations with scheduled down time
38:54 – New Relic and the ELK stack take care of metrics and logging
39:18 – Sentry.io (self hosted version) is being used to track exceptions
39:42 – The time series data doesn’t end up getting logged by these tools
40:20 – Braintree is used as a payment gateway to handle credit card and PayPal payments
41:44 – Transactional emails are being sent through AWS SES
42:24 – In app notifications are coming soon to SmartMusic (websockets, etc.)
44:05 – Another use case for websockets and events will be for collaboration features
44:49 – The databases are backed up daily and S3 is very redundant in itself for user files
45:31 – VictorOps handles alerting if a service happens to go down
45:58 – New Relic is being used in a few of the applications
46:55 – Handling bot related issues with nginx’s allow / deny IP address feature
48:46 – Best tips? Make a solid proof of concept in your new tech before switching to it fully
50:36 – Biggest mistake? Trying to use your old coding habits in a different language
51:27 – Dealing with N + 1 queries with GraphQL using DataLoader
52:58 – Ecto Multi is awesome for ensuring multiple things happen successfully
54:10 – Check out Julien’s blog, @julienXX on GitHub and he’s on Source Hut

Links

📄 References

⚙️ Tech Stack

🛠 Libraries Used

Support the Show

This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend.

Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included.

Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

6/14/23 - Fr. Dom Homily