So really learning from getting hands on like the need, solve those problems, just figuring it out in real time, figuring out what you need to learn to solve whatever that problem was, correct?

Yeah, exactly, because it was very hard to bring anything to life, right? When you had machine learning model, you had some output, that output had to go to some system, it had to go via emails, but how do we deliver it in the first place? They need to go and talk to different departments in the organization. How do you do that? And learn from them, learn from other developers, engineers, as well. Because in a large organization, they already have these tools. They already have version control. They already have something for CI3D, for orchestration. So we had Airflow. We had Jenkins back then. We had our code on Bitbucket. And we also had Kubernetes as a service. So that will happen to be the place for us to deploy things. We had Elks stack, which we were using for monitoring Grafana. So, you know, just by figuring out what's out there, we just got to that place.

a big list. We just talked about it. In fact in the recent KubeCon in Paris, like they had an entire day's keynote dedicated to just everything Gen AI, machine learning, like the whole stack. And there was a, I specifically remember a diagram that I haven't seen actually anywhere else. Like it was just in the keynote. I don't if Narmel remembers this, but it had a lot of dots in a circle. It was maybe 100 or so, and I think they were all products or tools related to AI stacks and ML stacks. And of course, Kubernetes was one of those dots. And then it showed all these connections between how they're all related. And I thought, who can know all those things? That's all I could think of is like,

there's too much there. No, you don't need to know all of that. So there is, I don't know whether you have seen, there is a mad landscape you can search with. It's called mad landscape. It's machine learning and it's extator. I'm not sure, something like that. That's crazy. The amount of tools that is there, like growing every year and they release this thing every year, and it's like unbelievable. And yeah, you can't learn all of that, obviously. And the tools itself change all the time. So you're using one tool, and in two years, it's completely different tool you have to migrate anyways. So it just never stops. It just never stops. But I think what's important to mention, I talked about the tools, it's about the principles. And that's what I feel is the most important piece. It's just like DevOps is also about the principles that you have. You need to make sure that there is traceability and reproducibility of whatever you do. So when you have a machine learning model deployment or run, you need to know like where a certain prediction came from. What data did it come from? What model artifact it came from? What code did it come from? Also, whether there were any changes in the data, like data drift, model drifts in the past, so that you could track things back. It's basically about connecting the dots and monitoring all of that. Just to make it happen, it's a lot of tools. If I just look at the, like, buckets of the tools, so you have version control, obviously, you need that, just as in DevOps. You have CI3D tools, you have orchestration tools, you have tools for compute, like where you train your models, you have tools to package your model, you have model registers, experiment tracking systems, You have like serving infrastructure, where they actually serve your models. And it can be a community. It can be many different places. Can you proprietary tools as well? Also feature stores became popular recently. You also have now with AI hype going on, you have vector search really and gaining popularity, vector databases, all kinds of human in the loop system, foundation model and APIs. It's a lot. It's getting even more crazy than it used to be. That

is a lot.