
Elixir Wizards
Elixir Wizards is an interview-style podcast for anyone interested in functional programming and the Elixir Programming Language. Hosted by SmartLogic engineers and Elixirists Owen Bickford, Dan Ivovich, and Sundi Myint, this show features in-depth discussions with some of the brightest minds in the industry, discussing training and documentation in Phoenix LiveView, the evolution of programming languages, Erlang VM, and more.
In the current season, we're branching out from Elixir to compare notes with thought leaders and software engineers from programming languages like JavaScript, Ruby on Rails, Go, Scala, Java, and more. Each episode will take a deep dive into a topic from Machine Learning and AI, to ECS and game development, to education and community.
Learn more about how SmartLogic uses Phoenix and Elixir. (https://smartlogic.io/phoenix-and-elixir?utm_source=podcast)
Latest episodes

Apr 11, 2019 • 44min
Jeffrey Matthias from Community - Elixir in Production
We talk with Jeffrey Matthias from Community about their current and past Elixir projects and how they are deployed.
Jeffrey Matthias - Community
Find Jeffrey elsewhere online:
http://github.com/idlehands
https://twitter.com/idlehands
0:47 - Give us a quick overview of the Elixir projects you have in production.
3:29 - Why are you using Elixir in production?
6:04 - What are some of the high level advantages / disadvantages of Elixir, from your perspective?
10:14 - What do you use to host your Elixir app?
Linode, AWS, DO
Heroku
Enmesos
mesos
How do you deploy your application?
Ansible
Deploy scripts
Distillery
14:19 - Are you able to get zero downtime deploys?
If so, how?
17:06 - Do you cluster the application?
If so, how?
22:53 - How does your Elixir App perform compared to others in your environment?
Response time
Throughput
Jobs/hr
25:01 - How are you solving background task processing?
29:17 - What libraries are you using?
Phoenix
33:53 - Third party apps
37:28 - Do you have a story where Elixir saved the day in production?
40:42 - If you could give one tip to developers out there who are or may soon be running Elixir in production, what would it be?
Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Jeffrey Matthias.Links:EnbalaWeedmapsDocker: Up & RunningAsync APIRabbitMQAMQP Elixir ClienterlcassErlang in Anger

Apr 4, 2019 • 17min
Jay Ashe from Cava - Elixir in Production
We talk with Jay Ashe from Cava about their current and past Elixir projects and how they are deployed.
Jay Ashe - Cava
Find Jay elsewhere online:
https://twitter.com/jgashe
0:40 - Give us a quick overview of the Elixir projects you have in production.
CAVA is a fast-casual mediterranean restaurant chain with 75 stores across the US. Elixir and phoenix power CAVA’s online ordering platform (order.cava.com and the CAVA app). We’ve got a REST (and websockets) api sitting behind react and our mobile apps, and we use phoenix templates for some of our back of house systems.
1:11 - Why are you using Elixir in production?
We have from the start! The application was originally implemented by Chris Bell and his team at madebymany. Chris, by the way, has a fantastic talk from ElixirConf 2016 that goes into our architecture and how we use elixir and OTP constructs to model our business logic. Chris will occasionally talk about the CAVA project on his Elixir podcast, ElixirTalk.
Chris’ Talk - https://www.youtube.com/watch?v=fkDhU-2NWJ8
1:58 - What are some of the high level advantages / disadvantages of Elixir, from your perspective?
Advantages: Elixir and Phoenix gives you rails-esque productivity/developer experience that scales. I think phoenix channels are a great example of this. Build a channel with complex real-time functionality and let it scale effortlessly.
Disadvantages:
Hiring and onboarding, depending on your mindset, can be difficult. If you’re used to hiring for experience in your stack, its just going to be more difficult. Lately we’ve started doing one-hour weekly knowledge shares that cover elixir basics and are closely tied to our usage of them. So, here’s a test case, and here are all of the test helpers that we have set up that will help you write that test. We also just sent a new Elixir dev to lonestar elixir
3:59 - What do you use to host your Elixir app?
Heroku
How do you deploy your application?
Heroku-buildpack-elixir https://github.com/HashNuke/heroku-buildpack-elixir
4:44 - Are you able to get zero downtime deploys?
As close as possible! We get that out of the box with heroku. When we deploy, heroku won’t point traffic to the new dyno until the app is healthy. We make extensive use of Phoenix channels over websockets, and our clients will reconnect automatically and transparently.
5:10 - Do you cluster the application?
Nope.
5:52 - How does your Elixir App perform compared to others in your environment?
I can’t really talk about numbers here, but Elixir is not at all our bottleneck. We don’t have other production applications
6:25 - How are you solving background task processing?
Quantum for cron jobs, genservers for everything else. We’re running a single elixir application that handles all synchronous and async processing
7:07 - What libraries are you using?
Phoenix
Phoenix_swagger for API documentation that integrates with controller tests https://github.com/xerions/phoenix_swagger
Ex_rated for rate limiting calls to our integrations https://github.com/grempe/ex_rated
Timex and calendar for datetime support with timezones https://github.com/bitwalker/timex
A combination of httpotion and httpoison for HTTP clients, but im interested in trying Mint https://github.com/ericmj/minthttps://github.com/appcues/mojito
Bamboo for transactional emails, like order confirmations etc https://github.com/thoughtbot/bamboo
8:59 - 3rd Party Services (i.e. Email, Payment Processing, etc)
Sendgrid for email, Google for geocoding, slack for some internal alerting of application health, LevelUp for payments. https://www.thelevelup.com/
10:07 - Do you have a story where Elixir saved the day in production?
Yes and no. So I could tell this story by explaining the issue we saw and the underlying cause at the same time, but I think it would be more fun to tell it like our team experienced it.
One day at lunch our application started going down. Lots of 500 errors. Red lights flashing. Panic ensuing. Lunch is our busiest time of day, so 1) we thought it was load related and 2) we really needed to fix it
None of our traditional resources (database, cpu, memory) were constrained and our integrations that were synchronous were fine.
Our logs were littered with errors from an analytics integration that ran asynchronously on genservers, but it didn’t seem related because we could see the error logs at times when our application was otherwise healthy. The team that used the analytics didn’t have a pressing need for them, and we deprioritized fixing the issue because the bug we were working on was so much more important (that’s foreshadowing).
I spent a little time looking at websockets, but I was easily able to match the load of the websocket portion of our application on my local machine with no degradations in performance (thanks, phoenix), so that was out.
At this point the issue was going on every day at lunch and I was getting annoyed at seeing the logs from the analytics integration when debugging, so I spent like 15 minutes finding and fixing the issue (a bad API key, basically)
Voila, issue gone. Time to grab some lunch.
We spent a while coming up with an explanation for this. Eventually we learned about max_restarts on a supervisor. By default, if a process crashes 3 times in 5 seconds, the process won’t be restarted again. So if another process (like the one handling a web request) tries to call that process that wasn’t restarted, the caller would crash, and we’d start to get 500 errors, customers couldn’t log in, mass confusion.
So there are a few takeaways from this story: For a while, elixir saved the day in production.
- A supervision tree prevented failures from the analytics process from affecting customers, until the scale of our failures exceeded the max_restart level.
- Our supervision tree needed some love though, clearly.
- Monitor your resources. CPU is a resource, but calls to another API are also a resource and can get unhealthy too.
15:00 - Are you using any cool OTP features?
GenServers, definitely. There’s lots we can do asynchronously especially in terms of our integrations. One process per store is a cool model that scales well and keeps issues isolated to a single store.
15:50 - If you could give one tip to developers out there who are or may soon be running Elixir in production, what would it be?
If you’re on a small team, Heroku or a similar provider might give you a lot of value in terms of infrastructure you can set up and forget.
Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Jay Ashe.Links:ElixirConf 2016 - Selling Food With Elixir by Chris BellHeroku-buildpack-elixirPhoenix SwaggerEx_ratedTimexMintMojitoLevelUp

Mar 28, 2019 • 29min
Frank Hunleth - Elixir in Production
We talk with Frank Hunleth from the Nerves core team about their current and past Elixir projects and how they are deployed.
Frank Hunleth - Nerves
Find Frank elsewhere online:
https://twitter.com/fhunleth
https://github.com/fhunleth
0:53 Frank intro
2:02 Give us a quick overview of the Elixir projects you have in production.
4:25 Why are you using Elixir in production?
8:00 What are some of the high level advantages / disadvantages of Elixir, from your perspective?
9:25 What hardware do you deploy to?
12:05 How do you get code to hardware after deployment?
13:47 How do you secure the code?
18:12 Do you cluster?
If so, how?
How does your Elixir App perform compared to others in your environment?
22:45 How does Elixir compare to other languages?
26:15 More information about Nerves
Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Frank Hunleth.Links:GrispGrisp GitHub

Mar 21, 2019 • 37min
Mark Ericksen - Elixir in Production
We talk with Mark Ericksen from Elixir Mix about their current and past Elixir projects and how they are deployed.
Mark Ericksen - Elixir Mix
Find Mark elsewhere online:
https://twitter.com/brainlid
https://brainlid.org/
00:32 Intro
0:58 Mark intro
Developing a long time. C#, then Rails. Webforms were terrible. Rails is “Wow, this is how web development should be… I moved across the country to work with this technology”
The Rails Community is strong. Dave Thomas got Mark into Elixir
2:48 What Elixir projects do you have in production?
A Rails app and a number of Elixir Micro-liths
4:29 Why do you use Elixir
6:45 Trends in moving from Ruby and Rails to Elixir
Ruby Syntax
Pattern Matching
Concurrency primitives
Fault Tolerance and a functional paradigm
Erlang/OTP
6:48 Comparing Elixir community to Ruby community to C# community
8:27 Any disadvantages to using Elixir?
Building releases. Configuring releases.
mix.release
10:13 Where are you hosting these bad boys?
AWS
Kubernetes in Production. So Fresh.
Docker and Distillery 2.0 Releases
Yaml files and Bash Scripts
Makefiles
10:53 What else are you using besides docker
12:36 Helm and Ksonnet.
13:55 Deploys
14:39 Clustering
17:50 How do your Elixir apps compare to the Ruby apps?
Big Elixir Apps
20:00 How Mark handles background jobs
Easy to write yourself with BEAM primitives
21:27 Libraries - Quantum, Bamboo, ex_machina, prometheus_ex via Eric’s influence
23:29 Third party integrations. Major ones were easy. Banks were bad.
Literally had to FTP files. Had to use java to write xml spreadsheets. The horror.
25:26 Has Elixir ever saved the day for you in Production?
29:42: Cool OTP features
30:57 Tips to developers
35:36 Where to find Mark
Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Mark Ericksen.Links:libclusterExqabsinthedataloader

Mar 14, 2019 • 28min
Brooklyn Zelenka from SPADE Co. - Elixir in Production
We talk with Brooklyn Zelenka from SPADE Co. about their current and past Elixir projects and how they are deployed.
Brooklyn Zelenka - SPADE Co.
Find Brooklyn elsewhere online:
https://twitter.com/expede
https://github.com/expede
1:08 - Brooklyn’s Background
Brooklyn's background and experience with Elixir is deep. Huge open source contributor.
Got started in Elixir just after Phoenix got to 1.0.
1:53 - Worked on several Elixir projects in production.
2:43 - Why she got into Elixir.
Real Time
More performant than Rails.
Great documentation
Industrial-grade
4:38 - When you wouldn't use Elixir.
Easy to get stakeholder buy-in. Just point to WhatsApp.
Elixir is made for 2019
CLI tools
Repl-driven development
TDD tools built in by default
All the best practices we have today are built in.
7:14 - Where has Brooklyn hosted her apps?
Heroku for POC's. AWS for production. Dockerized because "kubernetes is the new hotness"
9:40 - Do you do any clustering?
Load balanced above. AWS load balancing is very standard. They're well understood and have a nice developer experience.
10:29 Are you able to get any zero downtime deploys?
Zero downtime deploys. Awesome but impractical. Rolling deploys are easier and usually more appropriate. Some requirements make it valuable.
Erlang error states. Exceptional. Allows you to build for the happy path. Don't worry about error handling all the time.
Witchcraft and dark magic. Monads.
Poke around the standard library.
12:50 - How does Elixir compare to Rails in terms of response times, and other aspects?
15:32 - What libraries do you use and what have you built?
22:41 - Any cool features of OTP you are using?
25:36 - One tip to developers new to Elixir
Build up a peer-to-peer cli chat from scratch in one GenServer.
Find her at @expede everywhere on the internet.
Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Brooklyn Zelenka.Links:Heroku Elixir BuildpackHeroku Phoenix Static BuildpackExceptionalWitchcraftQuickChat

Mar 7, 2019 • 26min
Todd Resudek from Weedmaps - Elixir in Production
We talk with Todd Resudek from Weedmaps about their current Elixir projects and how they are deployed.
Todd Resudek - Weedmaps
Find Todd elsewhere online:
https://twitter.com/sprsmpl
https://github.com/supersimple
00:00 - Intro
01:06 - Tom introduces himself
02:54 - What is Weedmaps?
04:33 - Overview of the Elixir projects you have in production.
06:25 - Why are you using Elixir in production?
07:21 - Advantages / disadvantages of Elixir
10:37 - What do you use to host your Elixir app?
10:50 - How do you deploy your application?
11:22 - Are you able to get zero downtime deploys?
12:00 - Do you cluster the application?
13:00 - How does your Elixir App perform compared to others in your environment?
14:15 - How are you solving background task processing?
16:40 - What libraries are you using?
21:09 - 3rd Party Services
23:56 - Do you have a story where Elixir saved the day in production?
24:22 - Are you using any cool OTP features?
24:39 - Tip to developers
25:12 - Where you can find him
25:32 - Outro
Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Todd Resudek.Links:A Deep Dive into Hex - ElixirConf talkCredoDialyxirSwoosh

Mar 6, 2019 • 49min
Lonestar ElixirConf 2019 Lunchisode
We sat down with numerous developers, including José Valim and Chris McCord, during the Saturday lunch at Lonestar ElixirConf 2019. Hear what they had to say about the state of Elixir!
Guests:
José Valim - Creator of Elixir
Chris McCord - Creator of Phoenix
Paul Schoenfelder - Creator of Distillery, Timex, Libcluster, and many others
Chris Keathley - Host of Elixir Outlaws, developer at Bleacher Report
Amos King - Host of Elixir Outlaws
Jim Freeze - Organizer of ElixirConf, ElixirConf EU, and others
Susumu Yamazaki - Creator of Hastega
Brian Cardarella - CEO of Dockyard
Osa Gaius - Engineer at Mailchimp
Spectating:
Bruce Tate and Ben Marx
00:00 - Intro
01:47 - LoneStar begins
02:03 - Panelists introduce themselves
09:15 - Where is Elixir going?
10:14 - Releases
19:04 - The issue with hype
26:30 - Osa Intro
29:00 - Define lists
40:00 - How can Elixir displace Java?
Learn more about how SmartLogic uses Phoenix and Elixir.

Feb 28, 2019 • 33min
Ryan Billingsley from ClusterTruck - Elixir in Production
We talk with Ryan Billingsley from ClusterTruck about their current Elixir projects and how they are deployed.
Ryan Billingsley - ClusterTruck
Find Ryan elsewhere online:
https://twitter.com/ryanbillingsley
https://horriblenight.com/
00:00 - Fade In
00:50 - Ryan introduces us to ClusterTruck.
02:00 - How did Ryan get into Elixir?
03:12 - Where does the name ClusterTruck come from?
04:17 - Tell us about the projects you have in production.
05:50 - Why are you using Elixir in these projects.
08:11 - Disadvantages of using Elixir
09:22 - Comparing Elixir with Ruby, Node, Go.
11:38 - Where is ClusterTruck hosting their applications?
15:03 - Kubernetes?
16:03 - Zero Downtime Deployments?
16:42 - Do you do any clustering?
18:06 - How does Elixir perform compared to other project environments you’ve worked in?
19:52 - How are you solving background task processing?
21:09 - Other libraries?
23:34 - Other third party integrations?
25:46 - Is there a time Elixir has saved the day in Production?
27:47 - Cool OTP Features!
29:45 - Tips to devs thinking about running Elixir in Production.
31:17 - Outro
Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Ryan Billingsley.Links:Programming ElixirSalt StackSwarmOpushandle_continue

Feb 28, 2019 • 28min
Dan Ivovich from SmartLogic - Elixir in Production
We talk with developers from the team here at SmartLogic about our current practices on deploying Elixir and Phoenix in production.
Dan Ivovich - Director of Development Operations @ SmartLogic
Learn more about how SmartLogic uses Phoenix and Elixir.
00:00 - Fade In
00:30 - Introductions to Eric, Dan and SmartLogic
Dan Ivovich - Director of Development Operations @ SmartLogic
Eric Oestrich - Developer, Elixir Lead @ SmartLogic
Justus Eapen - Full stack developer @ SmartLogic
Introduced to Elixir by an old colleague.
1:20 - What Elixir projects do you have in production?
Several client projects in production. Several Mobile Apps with APIs powered by Phoenix and Elixir.
Baltimore Water Taxi.
A digital marketplace.
And more!
1:57 - Advantages and disadvantages to using Elixir.
We made the switch when a colleague was stoked about Functional Programming and introduced us to Elixir. We were won over by the performance and rich feature sets, OTP, etc.
2:43 - Where are we hosting our Elixir Apps?
Heroku
AWS
Linode
Digital Ocean
6:20Deployment process, tools, scripting
Ansible - for underlying VPS’s, servers, and more recently deployment itself. (Similar to Capistrano).
Distillery
Mix.release
7:18 - Zero Downtime Deployments
Old school load balancers and rolling restarts
7:46 - What are the performance metrics like? Comparatively.
Ruby ends up with memory leaks. That doesn’t happen with Elixir.
Memory utilization is flat and low no matter what.
“Phenomenal response times”
8:54 - How does Eric think about clustered applications in Elixir?
Going Multi Node (https://www.youtube.com/watch?v=lCUKQnkjajo)
Pg2 - process groups
Mnesia distributed database (beware!)
“Just sending messages to pids because Erlang is great”
Swarm / Horde
12:40 - How do we handle background tasks?
Started with verk
Recently becoming more comfortable with spinning up GenServers
“The language itself is built to be concurrent.”
15:06 What libraries are we using in prod?
First thing: You don’t need a whole lot because the language is so well designed.
Phoenix - web framework
Ecto - sort of an ORM
Distillery - for releases
Bamboo - for sending emails
Quantum - for task scheduling
Timex - for dates and times, and timezones
Cachex - for caching
18:20- What third party integrations have we attempted
Stripe
Square
Twilio
Mindbody
Always building our own clients. Using HTTPoison
19:58Has Elixir ever saved the day in production?
It’s saved many days by PREVENTING ISSUES. Systems are architected for reliability and fault-tolerance.
21:48 - Where do supervision trees come from? What is OTP?
OTP is an Erlang standard lib
Includes supervision trees, genservers, ETS, and a lot of stuff we don’t even know about!
gen_tcp
Mnesia
dets
23:43- Tips for devs considering running elixir in production.
Jump in and read the docs
Understand how systems boot, distillery releases, config providers, etc.
“Good server monitoring hygiene”
“DIVE IN!”
19:54 Outro
Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Dan Ivovich.Links:Going Multi-NodePG2mnesiaSwarmHordeVerkBambooQuantumCachex

Feb 25, 2019 • 1min
Smart Software Season 1 Trailer
Welcome to the first season of Smart Software with SmartLogic. We'll be interviewing several companies about how they use Elixir in Production this season. In this preview episode, we introduce ourselves and some of the topics we’ll be covering.
Learn more about how SmartLogic uses Phoenix and Elixir.