AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
How Do You Refactor Your Models?
We originally released our models in R. So it's just R in a Docker container as an API. And they ran pretty OK, but this kind of is where we switched to voice. We had to have 20 containers of our our model that it was serving 2 million responses a day. It almost just depends on how chatty people are. That's when we said, OK, we need to do something smarter with how we're serving these models. They are now deployed as Java Spring Boot services that run Python as a sidecar and Kafka consumer producers.