If

I could summarize what you said, the key point about a virtual cluster to understand what it is, is it is a Kubernetes cluster that runs inside of a host Kubernetes cluster. And it does have some of its own services, and then it shares the nodes with the host. Was there anything about that you'd like to correct? No,

that's an accurate summary. That's exactly the idea. Some things are shared, certain things are completely isolated. And that's the beauty of the virtual cluster. You can mix and match ultimately. Does

each of each cluster have its own isolated control plane? That

is correct. Yes. The virtual cluster ultimately is a container. So in this container, you have a fully-fledged Kubernetes control plane. That means you have an API server, you have a controller manager, you have state, right? That state can live in a SQLite database or in a full-blown etcd cluster, right? Like a real Kubernetes cluster would be using. The only thing that it doesn't have, which a regular Kubernetes cluster has in its control plane, is a scheduler. Because the underlying cluster has a scheduler and the scheduler distributes the odds to the different nodes. The virtual cluster typically doesn't have any nodes or may just have some optional nodes attached to it, but usually it uses the underlying cluster scheduler to actually get containers launched on the specific VMs of the host cluster. I

heard you correctly. You said the vCluster control plane is a container. Did you mean a single container, or does each piece of the control plane have its own container? Yeah,

it's actually one pod and a couple of containers in the pod. That's correct.

In a control plane, if you're on a large enough traditional cluster, you might scale out different parts of the control plane differently like you might have three or five instances of etcd for high availability and you might scale your api server horizontally if you are putting your entire control plane in one pod, do you have to decide upfront what size of resource you give to each piece and that's pretty much fixed for the duration of the vCluster?

Yeah, so we actually do something really interesting. So when you look at the multiple pods of the control plane, so there's one pod that contains pretty much all the core components, right? So controller manager and API server, we actually bake them in a single container. But then you have things, for example, for the DNS, like core DNS, we have two options to have it baked in or to run it separately as a separate container, even as a separate part, right, to be launched. And that way you have that flexibility of what do you want to run baked in and what do you want to run separately? Typically, to run things separately makes a little bit heavier weight, right? And running them embedded, which is our default typically, makes it much more lightweight. The same counts for the data store. So one thing we do, for example, is when you spin up a vCluster, you just go to vcluster.com right now and run the quick start, right? You download the CLI. We have this command called vcluster create in the CLI that helps you spin up a vcluster. It ultimately just sets a couple of config options and then runs a Helm install. It's nothing more than that, but it spins up a virtual cluster in the most lightweight form because we know people who just want to get started, right? Want to see the wow effect, right? And there's a lot of things that can go wrong if you want to spin up a fully fledged etcd cluster, right? So what we do instead is we even bake in the data store. So for example, the data store is just a SQLite in a persistent volume, right? And you could even disable the persistent volume and that would be entirely ephemeral, right? You restart the container, the cluster is completely reset, right? And so the vCluster is pretty dynamic. It can be as ephemeral and as lightweight as you want it to be also on the other extreme to be as heavyweight as you want it, right? And you can horizontally scale pretty much each component of the vCluster pretty easily. So it's really depending on your use case and how much resilience and separate scalability for each component you actually need for that particular scenario that you're running the vCluster in. I

want to highlight one aspect of what you just said. If you want the vCluster to be durable, you need to make some arrangement for it to access a persistent volume for the persistence piece. Anything you'd like to expand on that? No,

absolutely. If your Kubernetes cluster has persistent volume claim provisioning enabled, like dynamic provisioning of persistent volumes, that is what we're using by default. So we look into the cluster when you run vcluster create and actually see, hey, is that possible? And then we provision the PV that way, which is obviously super straightforward. Most Kubernetes clusters, even like Docker Desktop and Minikube, have that either by default enabled or let you enable that with just a single CLI command or click in the Docker desktop UI. But obviously, when you are in cloud environments, sometimes regulated industries don't want dynamic provisioning. Every PV needs to manually get provisioned. Hopefully, you don't have to be in that strict of an environment. But if you are, then you can also specify these things via the values YAML. We call it vcluster YAML, which is essentially the central file where you have all the config options available to you. And then you can apply that with your vcluster create command or with the Helm install command. You

have up to this point mentioned most of the pieces of the control plane, but not the networking. Does the vcluster share in the host cluster's networking, or does it run its own network? So

with regards to the networking, the virtual cluster uses the underlying cluster's networking. That's because the underlying cluster's nodes are being used, and that's obviously where most of the networking is happening, other than DNS, right. DNS is actually something that we run, as I mentioned earlier, as part of the control plane of the virtual cluster. So there's a separate DNS for each V cluster, which makes sense because things in DNS are based on the names of your namespaces. And if you have three virtual clusters and they all have a namespace called database, you would have conflicts if you were to use. You would try to map that to the underlying DNS. That's why every cluster gets its own DNS. or from a container to the internet, right? Or within your VPC, all of that runs on the nodes and on the network that your actual Kubernetes cluster, your host cluster is a part of. Are

there any best practices about how you launch these in the namespaces of the host cluster? Does each V cluster get its own namespace or how do you do that?

Yeah, so you can have multiple vclusters in the same namespace, but we typically encourage people to have one vcluster per namespace. That's definitely a best practice we encourage. And mostly the reason for this is every part that gets launched inside the vcluster, let's say the vcluster has 20 namespaces, right? typically happens is we got to translate the names of these pods down to the host cluster because the pods get actually launched by the host cluster. And the way that works is we copy them into the namespace where the virtual cluster runs. So if I have 100 pods in these 20 namespaces and I look into the host cluster, I see one namespace where the V cluster is running with the V cluster part for the control plane. Plus, I see these 100 parts, right, that come from the V cluster all in that very same namespace. So if you have multiple V clusters in there, it's much more difficult to get an overview of what belongs to which V cluster. We obviously, you know, we have some prefixes and suffixes, etc. to make it more clear and understandable. We set labels as well in this process, right, to make it, you know, filterability, etc. But it's just so much easier if you split up by namespace. And the added benefit is also if you're introducing things like network policies, or you're using things like Caverno or Open Policy Agent, right? know you had a session with Jim the other day about Kaverno and policies and Kubernetes. It's very, very important to set these policies on a namespace level because a lot of these constructs in Kubernetes are designed to be used at the namespace level. You can use it with labels as well, but the chance that you're going to make a mistake is going to be much, much higher. So we typically recommend do all of this on a namespace level and one week list of your namespace. Okay.

Now you did mention that this is a great solution for multi-tenancy. I can think of at least two other ways you might handle that. One would be to put each tenant in its own namespace. Another would be to give each tenant their own Kubernetes cluster. Could you give us some pros and cons of these different approaches?

Yeah, absolutely. Let's start with the namespaces, right? If you give every tenant a namespace, I mean, in a way, with a vCluster, as I just said, every vCluster should run in a separate namespace. So in a way, you're kind of doing that already with your tenants. The benefit of having the vCluster layer on top, rather than giving tenants directly the namespace, lies in giving the tenants the autonomy that they actually need. If you are restricted to a single namespace in Kubernetes, it's kind of like if I gave you just a single folder on a shared Linux host, and you had minimal permissions to do anything, right? So if you don't have root access to that machine, and you can't install anything in this Debian system or something like that, right, then you're going to be very, very limited. And every tenant will now need to agree on certain things. We actually had a lot of this going on in the late 90s and the first internet wave where people had these folders, web space sharing, where hosters were essentially giving you these limited capabilities to host your little website. But everybody had to agree on what PHP version or something is running on these servers, right? With virtualization, everybody can roll their own and they have a lot more freedom. You feel like you've got a full-blown server. And that autonomy is really important for engineers to do their work and to move fast, right? When you give somebody a namespace, you're going to be super limited, similarly limited. One of the most pressing limitations is cluster-wide objects. In Kubernetes, for example, CRDs, custom resource definitions, are things that allow you to extend Kubernetes. And when you look at most Helm charts out there and most tools designed for Kubernetes, take some of the popular ones like Istio, Argo CD, all of these tools introduce their custom CRDs. And many companies are building their own custom CRDs. They may have a CRD for backend and frontend. And when they're building these CRDs, CRDs are operated at a cluster-wide level. So there's no way for two tenants to work on the same CRD in the same cluster and not get in each other's way. It's not possible to constrain them to just a namespace. Another option may be, let's say you want another scenario where you're really limited, maybe if you're architecting your application to run across three namespaces. If you just get a single one, or I give you three isolated ones, and they can't communicate each other because you set up network policies as you should, as a cluster admin who wants to keep tenants apart, well, now you as an engineer can't architect your application the way you wish to architect it. It's very, very limiting. So that's the benefit of the vCluster. You really are still in a single namespace, but it doesn't feel like it for you. You actually have multiple namespace. You have cluster-wide objects. You could even choose a different Kubernetes version. Each of these vClusters can have a separate version. They don't need to be all the same. They don't need to be the same as the host cluster. And that gives tenants a lot of freedom. And then compare that to having separate clusters, which obviously gives you the ultimate freedom, but it comes at a very hefty price. So if you were to provision a thousand clusters for a thousand engineers in your organization, that's a pretty hefty cloud provider bill. And a lot of our customers, like commercial customers that work with us, the large enterprise, the Fortune 500s of the world, they have hundreds or even thousands of clusters. And that's a really big burden on these operational teams. It's not just the cost for compute, it's also the cost for upgrading things, keeping things in sync. You need fleet management suddenly for that large fleet of clusters. It's a really complicated operation to maintain 500 or 1,000 plus Kubernetes clusters. And with virtual clusters, it is much, much cheaper and becomes much easier. Because when you think of it, you can now run an Istio in the host cluster, and you can share it across 500 virtual clusters. So instead of maintaining 500 Istios, you have to maintain one. And you may not even need automation for that one. I would still recommend automating it, right? And using things like GitOps and infrastructures, code, etc. But the burden becomes much, much lower in the amount of code and plumbing you need to write around things. The inconsistencies you have between systems becomes much, much smaller when you have newer host clusters and have virtual clusters instead. And then, as I said, the cost is much, much lower as well compared to running 500 separate clusters. Just think about 500 clusters with three nodes each. We have 1,500 nodes running. Most of them are going to be idle, especially if you're thinking pre-production clusters, right? Most of them are going to run idle most of the time. In a virtual cluster, you may get away with having 500 nodes for everybody because they're much higher utilized, right? And much more dynamically allocated across your tenants.