Optimizing Erlang Server Performance

In this episode of Running in Production, Anthony Eden covers building a domain management service with Rails, Go and Erlang. It’s hosted on a combination of 70+ bare metal servers, AWS and Heroku. It’s been been up and running since 2010.

Anthony talks about handling millions of API queries per day, ~5 billion monthly DNS queries, spikes of up to 10,000 requests per second, sticking with a Rails monolith for the web dashboard, scaling PostgreSQL, building multiple data centers, feature flags and tons more.

Topics Include

• 3:58 – Millions of API queries per day and 2-5 billion DNS queries per month
• 6:40 – How Rails, Go and Erlang are being used along with why they were chosen
• 12:54 – How DNS lookups happen and the importance of DDoS protection
• 16:40 – The Erlang service has ~10k LOC and was written before Elixir existed
• 21:23 – Go is responsible for a lot of glue services
• 24:50 – A monolithic Rails app (server rendered templates) powers the web dashboard
• 28:08 – Sidekiq (Enterprise), Redis, PostgreSQL and all services run on Ubuntu LTS
• 29:41 – For cloud hosted services they end up on AWS or Heroku depending on what it is
• 31:45 – There’s 2 PostgreSQL instances and only the Rails app writes to it
• 34:55 – nginx is sitting in front of the Rails app
• 35:43 – Topping out at 5-10k requests per second through the Erlang service
• 42:44 – You can spin things up locally with or without Docker
• 47:15 – Datadog is used to help view metrics and logs to detect potential issues
• 50:10 – What exactly is Anycast?
• 52:07 – Picking out hardware for their data centers (roughly 70 physical servers)
• 59:29 – Chef is being used to configure all of the servers
• 1:02:23 – What the process is like to develop something and then deploy it to production
• 1:07:59 – Toggling feature flags, managing database migrations at scale and API versioning
• 1:16:15 – How developers add new features through pull requests and then deploy code
• 1:21:53 – Stripe handles all of the payments for each subscription tier
• 1:24:26 – Handling database backups with snapshots and streaming the data offsite
• 1:26:45 – Bugsnag is used for error handling and logs get written to Datadog as well
• 1:29:16 – Everyone’s been working remotely from day 1 and there is no centralized office
• 1:31:39 – Best tips? Have good processes in place as you grow in size
• 1:34:07 – Check out https://dnsimple.com/ and you can find Anthony on Twitter

Links

📄 References

• https://en.wikipedia.org/wiki/Anycast
• https://www.cloudflare.com/learning/ddos/dns-amplification-ddos-attack/
• https://miro.com/

⚙️ Tech Stack

• rails →
• ruby →
• erlang →
• golang →
• vue →
• aws →
• bugsnag →
• chef →
• cloudflare →
• datadog →
• docker →
• env-key →
• heroku →
• lets-encrypt →
• memcached →
• nginx →
• pingdom →
• postgres →
• redis →
• slack →
• statuspage →
• stripe →
• ubuntu →
• webpack →

🛠 Libraries Used

• https://github.com/hanami/hanami
• https://github.com/dnsimple/erldns
• https://github.com/dnsimple/dns_erlang
• https://github.com/wal-e/wal-e

Support the Show

This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend.

• Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included.

• Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.