Elixir Wizards

We talk with Jeffrey Matthias from Community about their current and past Elixir projects and how they are deployed. Jeffrey Matthias - Community Find Jeffrey elsewhere online: http://github.com/idlehands https://twitter.com/idlehands 0:47 - Give us a quick overview of the Elixir projects you have in production. 3:29 - Why are you using Elixir in production? 6:04 - What are some of the high level advantages / disadvantages of Elixir, from your perspective? 10:14 - What do you use to host your Elixir app? Linode, AWS, DO Heroku Enmesos mesos How do you deploy your application? Ansible Deploy scripts Distillery 14:19 - Are you able to get zero downtime deploys? If so, how? 17:06 - Do you cluster the application? If so, how? 22:53 - How does your Elixir App perform compared to others in your environment? Response time Throughput Jobs/hr 25:01 - How are you solving background task processing? 29:17 - What libraries are you using? Phoenix 33:53 - Third party apps 37:28 - Do you have a story where Elixir saved the day in production? 40:42 - If you could give one tip to developers out there who are or may soon be running Elixir in production, what would it be? Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Jeffrey Matthias.Links:EnbalaWeedmapsDocker: Up & RunningAsync APIRabbitMQAMQP Elixir ClienterlcassErlang in Anger

We talk with Jay Ashe from Cava about their current and past Elixir projects and how they are deployed. Jay Ashe - Cava Find Jay elsewhere online: https://twitter.com/jgashe 0:40 - Give us a quick overview of the Elixir projects you have in production. CAVA is a fast-casual mediterranean restaurant chain with 75 stores across the US. Elixir and phoenix power CAVA’s online ordering platform (order.cava.com and the CAVA app). We’ve got a REST (and websockets) api sitting behind react and our mobile apps, and we use phoenix templates for some of our back of house systems. 1:11 - Why are you using Elixir in production? We have from the start! The application was originally implemented by Chris Bell and his team at madebymany. Chris, by the way, has a fantastic talk from ElixirConf 2016 that goes into our architecture and how we use elixir and OTP constructs to model our business logic. Chris will occasionally talk about the CAVA project on his Elixir podcast, ElixirTalk. Chris’ Talk - https://www.youtube.com/watch?v=fkDhU-2NWJ8 1:58 - What are some of the high level advantages / disadvantages of Elixir, from your perspective? Advantages: Elixir and Phoenix gives you rails-esque productivity/developer experience that scales. I think phoenix channels are a great example of this. Build a channel with complex real-time functionality and let it scale effortlessly. Disadvantages: Hiring and onboarding, depending on your mindset, can be difficult. If you’re used to hiring for experience in  your stack, its just going to be more difficult. Lately we’ve started doing one-hour weekly knowledge shares that cover elixir basics and are closely tied to our usage of them. So, here’s a test case, and here are all of the test helpers that we have set up that will help you write that test. We also just sent a new Elixir dev to lonestar elixir 3:59 - What do you use to host your Elixir app? Heroku How do you deploy your application? Heroku-buildpack-elixir https://github.com/HashNuke/heroku-buildpack-elixir 4:44 - Are you able to get zero downtime deploys? As close as possible! We get that out of the box with heroku. When we deploy, heroku won’t point traffic to the new dyno until the app is healthy. We make extensive use of Phoenix channels over websockets, and our clients will reconnect automatically and transparently. 5:10 - Do you cluster the application? Nope. 5:52 - How does your Elixir App perform compared to others in your environment? I can’t really talk about numbers here, but Elixir is not at all our bottleneck. We don’t have other production applications 6:25 - How are you solving background task processing? Quantum for cron jobs, genservers for everything else. We’re running a single elixir application that handles all synchronous and async processing 7:07 - What libraries are you using? Phoenix Phoenix_swagger for API documentation that integrates with controller tests https://github.com/xerions/phoenix_swagger Ex_rated for rate limiting calls to our integrations https://github.com/grempe/ex_rated Timex and calendar for datetime support with timezones https://github.com/bitwalker/timex A combination of httpotion and httpoison for HTTP clients, but im interested in trying Mint https://github.com/ericmj/minthttps://github.com/appcues/mojito Bamboo for transactional emails, like order confirmations etc https://github.com/thoughtbot/bamboo 8:59 - 3rd Party Services (i.e. Email, Payment Processing, etc) Sendgrid for email, Google for geocoding, slack for some internal alerting of application health, LevelUp for payments. https://www.thelevelup.com/ 10:07 - Do you have a story where Elixir saved the day in production? Yes and no. So I could tell this story by explaining the issue we saw and the underlying cause at the same time, but I think it would be more fun to tell it like our team experienced it. One day at lunch our application started going down. Lots of 500 errors. Red lights flashing. Panic ensuing. Lunch is our busiest time of day, so 1) we thought it was load related and 2) we really needed to fix it None of our traditional resources (database, cpu, memory) were constrained and our integrations that were synchronous were fine. Our logs were littered with errors from an analytics integration that ran asynchronously on genservers, but it didn’t seem related because we could see the error logs at times when our application was otherwise healthy. The team that used the analytics didn’t have a pressing need for them, and we deprioritized fixing the issue because the bug we were working on was so much more important (that’s foreshadowing). I spent a little time looking at websockets, but I was easily able to match the load of the websocket portion of our application on my local machine with no degradations in performance (thanks, phoenix), so that was out. At this point the issue was going on every day at lunch and I was getting annoyed at seeing the logs from the analytics integration when debugging, so I spent like 15 minutes finding and fixing the issue (a bad API key, basically) Voila, issue gone. Time to grab some lunch. We spent a while coming up with an explanation for this. Eventually we learned about max_restarts on a supervisor. By default, if a process crashes 3 times in 5 seconds, the process won’t be restarted again. So if another process (like the one handling a web request) tries to call that process that wasn’t restarted, the caller would crash, and we’d start to get 500 errors, customers couldn’t log in, mass confusion. So there are a few takeaways from this story: For a while, elixir saved the day in production. - A supervision tree prevented failures from the analytics process from affecting customers, until the scale of our failures exceeded the max_restart level. - Our supervision tree needed some love though, clearly. - Monitor your resources. CPU is a resource, but calls to another API are also a resource and can get unhealthy too. 15:00 - Are you using any cool OTP features? GenServers, definitely. There’s lots we can do asynchronously especially in terms of our integrations. One process per store is a cool model that scales well and keeps issues isolated to a single store. 15:50 - If you could give one tip to developers out there who are or may soon be running Elixir in production, what would it be? If you’re on a small team, Heroku or a similar provider might give you a lot of value in terms of infrastructure you can set up and forget. Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Jay Ashe.Links:ElixirConf 2016 - Selling Food With Elixir by Chris BellHeroku-buildpack-elixirPhoenix SwaggerEx_ratedTimexMintMojitoLevelUp

We talk with Frank Hunleth from the Nerves core team about their current and past Elixir projects and how they are deployed. Frank Hunleth - Nerves Find Frank elsewhere online: https://twitter.com/fhunleth https://github.com/fhunleth 0:53 Frank intro 2:02 Give us a quick overview of the Elixir projects you have in production. 4:25 Why are you using Elixir in production? 8:00 What are some of the high level advantages / disadvantages of Elixir, from your perspective? 9:25 What hardware do you deploy to? 12:05 How do you get code to hardware after deployment? 13:47 How do you secure the code? 18:12 Do you cluster? If so, how? How does your Elixir App perform compared to others in your environment? 22:45 How does Elixir compare to other languages? 26:15 More information about Nerves Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Frank Hunleth.Links:GrispGrisp GitHub

We talk with Mark Ericksen from Elixir Mix about their current and past Elixir projects and how they are deployed. Mark Ericksen - Elixir Mix Find Mark elsewhere online: https://twitter.com/brainlid https://brainlid.org/ 00:32 Intro 0:58 Mark intro Developing a long time. C#, then Rails. Webforms were terrible. Rails is “Wow, this is how web development should be… I moved across the country to work with this technology” The Rails Community is strong. Dave Thomas got Mark into Elixir 2:48 What Elixir projects do you have in production? A Rails app and a number of Elixir Micro-liths 4:29 Why do you use Elixir 6:45 Trends in moving from Ruby and Rails to Elixir Ruby Syntax Pattern Matching Concurrency primitives Fault Tolerance and a functional paradigm Erlang/OTP 6:48 Comparing Elixir community to Ruby community to C# community 8:27 Any disadvantages to using Elixir? Building releases. Configuring releases. mix.release 10:13 Where are you hosting these bad boys? AWS Kubernetes in Production. So Fresh. Docker and Distillery 2.0 Releases Yaml files and Bash Scripts Makefiles 10:53 What else are you using besides docker 12:36 Helm and Ksonnet. 13:55 Deploys 14:39 Clustering 17:50 How do your Elixir apps compare to the Ruby apps? Big Elixir Apps 20:00 How Mark handles background jobs Easy to write yourself with BEAM primitives 21:27 Libraries - Quantum, Bamboo, ex_machina, prometheus_ex via Eric’s influence 23:29 Third party integrations. Major ones were easy. Banks were bad. Literally had to FTP files. Had to use java to write xml spreadsheets. The horror. 25:26 Has Elixir ever saved the day for you in Production? 29:42: Cool OTP features 30:57 Tips to developers 35:36 Where to find Mark Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Mark Ericksen.Links:libclusterExqabsinthedataloader

We talk with Brooklyn Zelenka from SPADE Co. about their current and past Elixir projects and how they are deployed. Brooklyn Zelenka - SPADE Co. Find Brooklyn elsewhere online: https://twitter.com/expede https://github.com/expede 1:08 - Brooklyn’s Background Brooklyn's background and experience with Elixir is deep. Huge open source contributor. Got started in Elixir just after Phoenix got to 1.0. 1:53 - Worked on several Elixir projects in production. 2:43 - Why she got into Elixir. Real Time More performant than Rails. Great documentation Industrial-grade 4:38 - When you wouldn't use Elixir. Easy to get stakeholder buy-in. Just point to WhatsApp. Elixir is made for 2019 CLI tools Repl-driven development TDD tools built in by default All the best practices we have today are built in. 7:14 - Where has Brooklyn hosted her apps? Heroku for POC's. AWS for production. Dockerized because "kubernetes is the new hotness" 9:40 - Do you do any clustering? Load balanced above. AWS load balancing is very standard. They're well understood and have a nice developer experience. 10:29 Are you able to get any zero downtime deploys? Zero downtime deploys. Awesome but impractical. Rolling deploys are easier and usually more appropriate. Some requirements make it valuable. Erlang error states. Exceptional. Allows you to build for the happy path. Don't worry about error handling all the time. Witchcraft and dark magic. Monads. Poke around the standard library. 12:50 - How does Elixir compare to Rails in terms of response times, and other aspects? 15:32 - What libraries do you use and what have you built? 22:41 - Any cool features of OTP you are using? 25:36 - One tip to developers new to Elixir Build up a peer-to-peer cli chat from scratch in one GenServer. Find her at @expede everywhere on the internet. Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Brooklyn Zelenka.Links:Heroku Elixir BuildpackHeroku Phoenix Static BuildpackExceptionalWitchcraftQuickChat

We talk with Todd Resudek from Weedmaps about their current Elixir projects and how they are deployed. Todd Resudek - Weedmaps Find Todd elsewhere online: https://twitter.com/sprsmpl https://github.com/supersimple 00:00 - Intro 01:06 - Tom introduces himself 02:54 - What is Weedmaps? 04:33 - Overview of the Elixir projects you have in production. 06:25 - Why are you using Elixir in production? 07:21 - Advantages / disadvantages of Elixir 10:37 - What do you use to host your Elixir app? 10:50 - How do you deploy your application? 11:22 - Are you able to get zero downtime deploys? 12:00 - Do you cluster the application? 13:00 - How does your Elixir App perform compared to others in your environment? 14:15 - How are you solving background task processing? 16:40 - What libraries are you using? 21:09 - 3rd Party Services 23:56 - Do you have a story where Elixir saved the day in production? 24:22 - Are you using any cool OTP features? 24:39 - Tip to developers 25:12 - Where you can find him 25:32 - Outro Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Todd Resudek.Links:A Deep Dive into Hex - ElixirConf talkCredoDialyxirSwoosh

We sat down with numerous developers, including José Valim and Chris McCord, during the Saturday lunch at Lonestar ElixirConf 2019. Hear what they had to say about the state of Elixir! Guests: José Valim - Creator of Elixir Chris McCord - Creator of Phoenix Paul Schoenfelder - Creator of Distillery, Timex, Libcluster, and many others Chris Keathley - Host of Elixir Outlaws, developer at Bleacher Report Amos King - Host of Elixir Outlaws Jim Freeze - Organizer of ElixirConf, ElixirConf EU, and others Susumu Yamazaki - Creator of Hastega Brian Cardarella - CEO of Dockyard Osa Gaius - Engineer at Mailchimp Spectating: Bruce Tate and Ben Marx 00:00 - Intro 01:47 - LoneStar begins 02:03 - Panelists introduce themselves 09:15 - Where is Elixir going? 10:14 - Releases 19:04 - The issue with hype 26:30 - Osa Intro 29:00 - Define lists 40:00 - How can Elixir displace Java? Learn more about how SmartLogic uses Phoenix and Elixir.

We talk with Ryan Billingsley from ClusterTruck about their current Elixir projects and how they are deployed. Ryan Billingsley - ClusterTruck Find Ryan elsewhere online: https://twitter.com/ryanbillingsley https://horriblenight.com/ 00:00 - Fade In 00:50 - Ryan introduces us to ClusterTruck. 02:00 - How did Ryan get into Elixir? 03:12 - Where does the name ClusterTruck come from? 04:17 - Tell us about the projects you have in production. 05:50 - Why are you using Elixir in these projects. 08:11 - Disadvantages of using Elixir 09:22 - Comparing Elixir with Ruby, Node, Go. 11:38 - Where is ClusterTruck hosting their applications? 15:03 - Kubernetes? 16:03 - Zero Downtime Deployments? 16:42 - Do you do any clustering? 18:06 - How does Elixir perform compared to other project environments you’ve worked in? 19:52 - How are you solving background task processing? 21:09 - Other libraries? 23:34 - Other third party integrations? 25:46 - Is there a time Elixir has saved the day in Production? 27:47 - Cool OTP Features! 29:45 - Tips to devs thinking about running Elixir in Production. 31:17 - Outro Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Ryan Billingsley.Links:Programming ElixirSalt StackSwarmOpushandle_continue

We talk with developers from the team here at SmartLogic about our current practices on deploying Elixir and Phoenix in production. Dan Ivovich - Director of Development Operations @ SmartLogic Learn more about how SmartLogic uses Phoenix and Elixir. 00:00 - Fade In 00:30 - Introductions to Eric, Dan and SmartLogic Dan Ivovich - Director of Development Operations @ SmartLogic Eric Oestrich - Developer, Elixir Lead @ SmartLogic Justus Eapen - Full stack developer @ SmartLogic Introduced to Elixir by an old colleague. 1:20 - What Elixir projects do you have in production? Several client projects in production. Several Mobile Apps with APIs powered by Phoenix and Elixir. Baltimore Water Taxi. A digital marketplace. And more! 1:57 - Advantages and disadvantages to using Elixir. We made the switch when a colleague was stoked about Functional Programming and introduced us to Elixir. We were won over by the performance and rich feature sets, OTP, etc. 2:43 - Where are we hosting our Elixir Apps? Heroku AWS Linode Digital Ocean 6:20Deployment process, tools, scripting Ansible - for underlying VPS’s, servers, and more recently deployment itself. (Similar to Capistrano). Distillery Mix.release 7:18 - Zero Downtime Deployments Old school load balancers and rolling restarts 7:46 - What are the performance metrics like? Comparatively. Ruby ends up with memory leaks. That doesn’t happen with Elixir. Memory utilization is flat and low no matter what. “Phenomenal response times” 8:54 - How does Eric think about clustered applications in Elixir? Going Multi Node (https://www.youtube.com/watch?v=lCUKQnkjajo) Pg2 - process groups Mnesia distributed database (beware!) “Just sending messages to pids because Erlang is great” Swarm / Horde 12:40 - How do we handle background tasks? Started with verk Recently becoming more comfortable with spinning up GenServers “The language itself is built to be concurrent.” 15:06 What libraries are we using in prod? First thing: You don’t need a whole lot because the language is so well designed. Phoenix - web framework Ecto - sort of an ORM Distillery - for releases Bamboo - for sending emails Quantum - for task scheduling Timex - for dates and times, and timezones Cachex - for caching 18:20- What third party integrations have we attempted Stripe Square Twilio Mindbody Always building our own clients.  Using HTTPoison 19:58Has Elixir ever saved the day in production? It’s saved many days by PREVENTING ISSUES. Systems are architected for reliability and fault-tolerance. 21:48 - Where do supervision trees come from? What is OTP? OTP is an Erlang standard lib Includes supervision trees, genservers, ETS, and a lot of stuff we don’t even know about! gen_tcp Mnesia dets 23:43- Tips for devs considering running elixir in production. Jump in and read the docs Understand how systems boot, distillery releases, config providers, etc. “Good server monitoring hygiene” “DIVE IN!” 19:54 Outro Learn more about how SmartLogic uses Phoenix and Elixir.Special Guest: Dan Ivovich.Links:Going Multi-NodePG2mnesiaSwarmHordeVerkBambooQuantumCachex

Welcome to the first season of Smart Software with SmartLogic. We'll be interviewing several companies about how they use Elixir in Production this season. In this preview episode, we introduce ourselves and some of the topics we’ll be covering. Learn more about how SmartLogic uses Phoenix and Elixir.