Making Testbeds for Carbon Aware Computing

Host Chris Adams is joined by special guest Philipp Wiesner, a research associate and PhD student at TU Berlin, to discuss how computing systems can better align energy consumption with clean energy availability. Contributing to Project Vessim, Philipp explains how researchers are now able to model different energy consumption scenarios, from solar and wind power integration to the complexities of modern grids despite the scarcity of available testing environments. They discuss federated learning and its role in carbon-aware designs, along with challenges in tracking real energy savings. Tune in to learn about the future of carbon-aware computing and the tools being developed to help software become more sustainable.

Connect with us on Twitter, Github and LinkedIn!

TRANSCRIPT BELOW:

Philipp Wiesner: Optimizing energy usage for when the sun is shining and the wind is blowing, of course it makes sense. But the big problem is that these green energy certificates, these guarantees of origin certificates, are traded independently of the physical flow of energy.

Chris Adams: Hello, and welcome to Environment Variables, brought to you by the Green Software Foundation. In each episode, we discuss the latest news and events surrounding green software. On our show, you can expect candid conversations with top experts in their field who have a passion for how to reduce the greenhouse gas emissions of software.

I'm your host, Chris Adams. Hello, and welcome to Environment Variables, where we bring you the latest news and insights from the world of sustainable software development. I'm your host, Chris Adams. Today, we're diving into the world of carbon-aware computing, how computing systems can align the energy consumption with the availability of clean energy.

Natural cycles govern all kinds of things we do. We mostly go to bed at night, and the food we eat is often influenced by what season we're in, and so on. This also shows up in how energy is available to us. You can see how day follows night in the output from a solar panel. And broadly speaking, it tends to be windier around winter than it is in the summer.

Even when we burn fossil fuels, we're basically digging up energy stored millions of years ago. As we move away from fossil fuels though, where we get to use the sunlight that was conveniently stored for us, we move towards a system based around harvesting energy, like we harvest food. The way we consume energy needs to adapt as well.

And that's the idea behind carbon-aware computing. But how do we know if our carbon-aware designs are actually better for the environment? With us today is Philipp Wiesner, a research associate and fourth year PhD student at TU Berlin, Technical University Berlin, whose work focuses on carbon-aware computing.

Philipp is deeply involved in research that connects the dots between computing power and renewable energy, and today will unpack the work he's been doing, including his work on Project Vessim, a testbed for carbon-aware computing. So let's dive in. Philipp, thanks so much for coming back, and for those who haven't heard the last episode when you joined us in 2022, can I give you a bit of space to introduce yourself?

Philipp Wiesner: Yeah, so hey Chris, and thanks a lot for having me back. I'm really excited. Yeah, so I'm Philipp, I'm a PhD student, as you just said, and a research associate at TU Berlin. I'm in the research group for distributed and operating systems. And I've been doing research on carbon-aware or in general, sustainable computing since the start of my PhD or my master's thesis in 2020.

And yeah, at the start of my PhD, I was mainly looking at the basics. So that's the first podcast we had two years ago. The field was still very young and we were looking at the potential of delaying the execution of workloads to make use of cleaner energy. Later, we also looked at migrating workloads from this to different data centers and so on.

And it was all based on carbon intensity. And then from then on later, we moved on from only optimizing for carbon intensity to really better understanding how electric grids work and rather optimizing for curtailed energy, excess energy. So, for green energy that would otherwise go to waste.

Chris Adams: Cool. Thank you. Okay. So you mentioned we spoke in 2022, two years ago, and the field has moved. And while we'll share a link to that podcast, where we go into a lot of detail, can I just give you a bit of time to just provide a brief summary of the paper that you spoke about then? I think it was called Let's Wait Awhile.

That was one of the projects that you're working on, and that was one thing related to the cloud. If we can talk about that'll be a nice scene setter. Then we can talk about the new work that we, honestly, I'm really excited about in all kind of nerdy glee. So yeah, Wait Awhile. What's the deal with that one?

Philipp Wiesner: Yeah, sure. So the idea in Let's Wait a While was just that in different regions of the world and at different times we have different carbon intensities. So carbon intensity is a metric that describes how dirty or how clean is the energy that you draw from the public grid in terms of emissions per kilowatt hour.

And at the time we were looking just, if you have a single data center and you might have some flexible workloads. So workloads where you do not really care if they happen now or tonight or tomorrow morning, how much potential do we have in shifting them, delaying them, computing them a bit earlier, maybe dissecting them in different parts and computing these times in different parts and so on?

And it was like a bit of a, yeah, it was a basic work at the time.

Chris Adams: And there were some, I think, when I first read about it, there were some quite, the savings there were around the low tens of percent for, compared to what the other ones would be. It was something like that, I think, from there. So that was one thing that caught my attention while we ended up speaking about it last time.

This new work that you presented at Hot Carbon, Vessim, as I understand it, this is something which is more general and applicable to all kinds of grids rather than being focused on one or two. Can you tell me a little bit about why you worked on this and what the general idea for creating a testbed for hardware computing was in the first place?

Philipp Wiesner: Yeah, sure. So the thing is that in carbon-aware computing, it's really common to simulate. The thing is that we're often talking about highly distributed systems, often globally distributed systems. And also these approaches often only take effect if you like simulate them or like run them for weeks or months because we need different seasons.

We need different, yeah. So the thing is that in Let's Wait a While and also in basically pretty much all the papers I wrote so far, we have to simulate systems. And in pretty much all the literature so far, people simulate systems because it's just extremely expensive to actually build and operate such systems.

Only big players can do it. And also results are not reproducible. So no one can really check if the stuff that you're reporting is correct.

Chris Adams: So let me just check. So basically, there is no way to... essentially there's no counterfactual, there's no way to see if my, like, I might say "yeah I've made the savings" but without showing your workings it's quite difficult to stand by them or even challenge them actually, that's one of the things that seems to be a bit complicated with that, right?

Philipp Wiesner: Exactly, a general problem in research, to make stuff reproducible. But here we have the big problem that everyone is implementing their own simulation. So everyone is doing it by themselves and just iterating over a big time series for carbon intensity is maybe not that hard. But the moment you have more complex systems and you're working with forecasts and so on, there's a lot of things you can do wrong.

And it's quite a heavy development effort often to do things right. So, just over the years we collected a lot of experience and eventually came up with the simulator. So, Vessim is basically a big co simulation suite where you can plug and play your own simulators for renewable power generation, carbon intensity of the grid, energy storages, and so on.

And you can connect them, you can connect existing simulators, and then test your real system, real applications, as if they were interacting with this simulated energy system.

Chris Adams: Ah, I see, so let me just check if I'm So one of the things you mentioned before is that we tend to look at the grid as like, just one thing that might be changing. And even our interactions are influencing the grid as well, so there's, that's one thing you need to take into account. But you also mentioned just then, that while most of us think of electricity as just power coming out of the wall, there's actually, it's more helpful to think about it in terms of, well, there's maybe on site clean energy, which is maybe very, clean, but isn't always available, and there's maybe on site batteries where the carbon intensities might be influenced by what has gone into it.

Like if it's clean energy that's gone in, it's going to be very clean. If it's dirty grid energy, it's going to be quite dirty. And then there's the grid. So there's multiple different, kind of, ways energy might go into the system that you use for your computing, for example, and you need to account for some of those things.

And then in addition to that, different parts of the world might have totally different grids. So Poland, for example, right now, loads and loads of coal, but comparing that to Iceland, which is, or France, which is like full of nukes, it's going to be very different carbon intensity. So you can't make the same assumptions, and you'll need to essentially do something to recreate the conditions that you might be in so that you can make somewhat more robust and intelligible statements about the savings you might achieve by making any kind of carbon-aware designs.

Philipp Wiesner: Yes, exactly. So yeah, as you said, like most of the existing research is looking at carbon intensity is looking at the global grid. And the problem with this metric is in general, that it's oversimplifying how complex grids actually are. So energy markets are really complex, grids are really complex. So if we only optimize for carbon intensity, which is a relevant metric in the future, we will run into several problems.

Because it's just hiding a lot of underlying complexities. And, as you said, we can maybe avoid these complexities if we're looking more local at microgrids. So if we assume, I don't know, a data center provider is putting solar panels on their roof or like having a direct power purchase agreement with some close by wind park, then we have a lot more direct connection of who's producing and who's consuming, and Vessim is exactly designed for this, to simulate such systems so that eventually you can optimize really for the energy that you're producing or storing.

Chris Adams: I see, okay, so you're still looking at this and we're looking at more than just a grid, basically. We're saying, we're focusing on the carbon intensity because the power can be coming from different places, like the one I'm immediately thinking of, actually. So in the news, Elon Musk's talking about this new data center where he's, where they bought 100 000 Nvidia H100 GPUs using 100 megawatts of power.

And that's so much power that they can't get that from the grid. So what they've done is, it's not clean energy they've used. They've used, say, like, a bunch of mobile gas generators, plugged into the data center that's come in. So we spoke about on site clean energy, but there's actually all kinds of energy that might be going in, and we're now seeing something like this in this kind of world of centralized AI, where you do see organizations looking to things other than just the grid to power things.

So we might need to take into account a whole basically the system seems to be getting more dynamic and being able to simulate this seems to be one of the ideas that Vessim is intended to kind of support so that we can essentially show our working and understand what kind of changes we might be making, or under these conditions, this is what a carbon-aware design might achieve, for example.

Philipp Wiesner: Yeah, exactly. Yeah. As you're saying, it's unfortunately becoming more and more popular to again, build gas turbines or even small nuclear power plants right into data centers to like come up with this insane amounts of energy that they need by now. We of course, like rather focusing on renewable sources because they're also a lot more difficult to schedule.

But yeah, this is one of the use cases of Wessim. So you can like explore how does it affect my data center if I cover the roof with solar panels, build a wind park nearby. How does it affect my data center if I invest into a lot of energy storage? How can I optimize my data center to make best use of these local resources?

And then there's also a lot of like systems questions, like how do I optimize? How do I make applications aware of locally produced energy? How do I make them aware of the state of charge of local batteries? What are they allowed to know? Like there's also security questions involved. And all of this we want to make explorable, so to say,

by providing this very general purpose, extendable simulation framework.

Chris Adams: Okay, cool. That sounds, okay, this sounds lots and lots of fun, and I remember getting quite excited when I read the paper from Hot Carbon. So you've shared this, but as I understand it, some of this has actually been used for some production work, or this isn't just like a wacky paper, there's actually people, there are examples of this being used.

I know that your paper spoke about the use of "Okay, here's how we've been running this on a kind of test system in a Raspberry Pi. Here's how it works with AWS cloud instances." So there are various examples of it being used. But there is a new project that you told me about beforehand called FedZero.

Can you tell me a little bit more about that? Because that seems to be applying some of these ideas in the real world. So yeah, maybe I'll just give you the chance to plug that one, because yeah, it was a fun read.

Philipp Wiesner: No, sure. Thanks a lot. Yeah, exactly. So we're trying to, deploy Vessim in a lot of different projects now. One of the main ideas behind Vessim is also to be have like hardware and software in-the-loop integration. So you can also plug in real devices, real energy generators, and so on.

In FedZero, we didn't do that. FedZero is also fully simulated. But it's a new, it's a scheduler basically we proposed for federated learning systems. This is,

federated learning is some type of distributed machine learning training. So you have a bunch of different clients and they locally train a machine learning model.

So they locally consume energy and the scheduler basically decides when, who is training. So it's like distributed batch processing in the end. And the entire evaluation of the system was done with Vessim. We have, for example, a lot of data sets that are coming with Vessim. They've been provided, for example, by Solcast.

It's a big solar data provider. And the entire idea behind the system is to only use excess energy. So only use curtailed energy and spare computational resources to perform a distributed machine learning training to make it effectively (an exclamation mark) zero carbon! Because you don't use any energy that would have been used by anyone else, basically.

Chris Adams: Oh, I see. Okay, so there's two things that I'd like to unpack there. So could you maybe just talk a little bit about the relationship to, I think I was confusing this with Flower, perhaps. So Flower is one of the wider projects which has something to do with federated machine learning. you maybe just talk on that because I think, for people who have listened to this before, they might not be aware of Flower or the even idea what federated machine learning might be. Because I just spoke about a centralized massive data center, but that's not the only way you can do machine learning and AI training by the sounds of things.

Philipp Wiesner: No, sure. We can also go a bit more into detail. So in federated learning, we assume that our data is distributed. So it's a different clients, a very common example are hospitals that, for example, want to train a common model for skin cancer detection whatsoever, but they cannot share their private patient data.

So we need to somehow train on their data without them releasing this data. And what's happening and Flower is one of the frameworks that lets you do this, that lets you perform federated learning as we distribute the model to the different hospitals, they locally train on the data and send it back.

And we try to schedule this local training in a way that it's only using green energy. Yeah, that's the main idea.

Chris Adams: Ah, I see. So, these would be devices which might already exist, for example, you don't have to have the same kind of emphasis on having a bunch of new stuff that you're using all the time to kind of sweat the asset and make the most of it, so you could be using maybe older technology or a more kind of mixed set of computing, rather than 100 000 H100s, for example, in one straight place, for example.

Philipp Wiesner: Yes, that's true. That's true in one sense. On the other hand, we always have to see what we're optimizing for. And if you're optimizing for something like curtailed energy, like for example, in the north of Germany, we have a lot of wind parks. But the German grid is not always strong enough to get all this energy to the south where like the big consumers are, the big industries.

So often times we have an excess of energy in the north. Now if you have hundreds of kilowatt hours of excess energy in the north, it doesn't get you anywhere to turn on a bunch of iPhones or old computers to make use of this energy. So usually with our approaches we target big consumers. We target the A100s and H100s to, because they actually make a difference.

Like turning them on or off actually makes a difference to the grid. A few thousand iPhones, not so much.

Chris Adams: Okay, so you still would be using some of this kind of relatively powerful hardware, if it represents a significant power draw on the grid, for example. And, is there anything special about where these might be placed? Because I've, seen examples and I've seen people talking about the idea of like literally putting this stuff directly into, say, the towers of wind turbines.

I know this is one thing that Windnode in Germany were doing, for example, because that's basically, where the power's not being used, it's one of the most fastest ways to actually

Philipp Wiesner: yeah, clearly, yeah, that's a super important topic, like placement of data centers. Just what I said, of course, you like place these data centers again, and if we talk about Germany in like Frankfurt region or in the south of Germany, it doesn't get you anywhere because you still need to transfer the energy there.

So you would actually need to build data centers very close to where energy is produced. However, like the big problem that we're facing right now with all this LLM and GPU stuff. is that there's actually not a lot of flexibility. So, FedZero for example assumes that we have some spare resources and some excess energy, and maybe the excess energy is fair, we do have that but we do not have spare resources.

Like, whenever I keep telling people since the last six months about this project they're like, "no, there are no spare resources." Like, GPUs are used all the time, we have a GPU shortage so, carbon-aware computing on GPUs is not really working at the moment.

Chris Adams: Yeah, I'm glad you mentioned this because this was one thing I've always been struggling to make sense of because when you've got a graphics card which is like 40k and you've got only a certain amount of years to use it, for example, the cost of having it off is actually quite high. So therefore, being able to simulate what the savings might be from some carbon air computing might, would actually be meaningful.

Or, if you're not going to switch off the power, then you might need to find another way to, like, not draw power from the grid. So that kind of implies either an increased amount of storage that you might need, so that rather than switching things off, you just reduce the power you're drawing from the grid by pulling it from somewhere else, for example.

So that's another kind of way you might flex your system to... still reduce the emissions, because it may be the case that you're able to load up on clean energy when there is an excess in other places, whilst keeping that thing still running, for example.

Philipp Wiesner: Yeah, exactly. No, you're completely right. And that was different, a few years ago when you, like, were mostly talking about cloud computing on CPUs, when we were talking about carbon-aware computing, because there you usually have a big buffer, because cloud providers want to have this image of infinite scalability, so they usually have spare resources at hand.

So, yeah, like this entire load shifting stuff was somewhat reasonable. Now with GPUs, I mean, you're right, you would need to plug them off, but no one's doing that. Like, if you're buying a GPU, you're using it.

Chris Adams: Using as much as you possibly can to make it, to make the per hour cost as low as possible, because you're, I see, okay, so, this is one thing I'm really glad you allowed us to dive into this because one of the things that I would initially think about, like there's some trade offs you might be making, and I might, before I spoke to you, one of my assumptions was that by using a federated model, you might have a lower embodied carbon because you're able to use a wider set of computing, which might already exist, for example, but it sounds like, because some of these chips are, they use more power, but per instruction, they're much, much more efficient per instruction. So, you're probably not going to save that much from the total embodied carbon of your fleet, for example, by reusing existing stuff in some cases. Is that what it would be like?

Philipp Wiesner: Yeah, there's different studies on whether highly distributed training or computation in general is actually more or less efficient. It often really depends, like usually the centralized, like you won't get anything more efficient than an H100. Energy per instructions. It's super, super efficient, but then of course, if you put them all in one place, you need to cool more and so on.

So often it can be more efficient to distribute it. there's no clear answer to that. I would say it really depends on the system, whether federated distributed system is more energy efficient or a local one, but of course in distributed system, you might have more flexibility for scheduling and waiting and yeah.

Chris Adams: Ah, I see. And I'm just going to move on to another question in a second, but while we talk about that, I just wanted to speak about maybe this idea of more distributed federated learning. A lot of the time you see people talking about saying, "I need a cluster of machines in one place because they need to talk to each other."

And the closer they are, the lower the latency, the faster the responses might be. And. When you're using a more distributed version, a distributed model, if you're not doing, like, even if you're not thinking about machine learning, you still need to think about latency, how long your response comes back.

How, like, what are the trade offs that you have there when you're looking at a kind of federated model, like with Flower, or providing some of these kinds of more distributed, carbon-aware style approaches, for example? Like, how do you address that?

Philipp Wiesner: So in general, in federated learning, we differentiate between cross device and cross silo federated learning. And cross device is what you were mostly referring to as like you have a lot of iPhones, a very distributed small system. They're often connected via mobile network. So latencies are usually quite high.

And there you really need to consider this latency trade off. In cross silo federated learning, it's a bit less important because you assume that each of these clients is, as I said, like a small data center or a big data center, they are connected via fiber. So, latencies are usually very low and not super relevant compared to how long a training takes.

So like sending models back and forth, model weights is not really relevant compared to how much time you need to compute actually. 

Chris Adams: Ah, thanks for explaining that, because there are various companies who are talking about having a distributed model, or saying, you need to do, you need to have, maybe, infrastructure that is complementary to the grid. And I was never able how to, how you get around the issue of speed of light, right?

Because they're going to be very, different if, even if it's just opposite ends of the country, that's going to, depending on the size of your country, that's going to be somewhat meaningful there. Okay. All right then, so we spoke a little bit about the use of Vessim, we spoke about why it exists, and like kind of this idea that you can simulate various kind of configurations of grids or configurations of power coming into the system.

I imagine there'll be some people who are listening thinking, "this sounds cool, how do I actually use this on a project when I'm designing something in a kind of carbon-aware way, or how do I actually, what am I looking for?" Could you maybe talk me through? The actual process of using Vessim specifically.

We can start with something simple like a machine learning job first and then we might talk about some other configurations like perhaps a CMS because that's what lots of people might be working on for example.

Philipp Wiesner: I mean, the entire idea of Vessim is that you do not really notice that it's a simulator. It should behave as like a proper energy system. So you can, on the one hand, you have your normal applications. It doesn't matter if it's a machine learning job or if it's a CMS or if it's anything else. You just have your applications that are running.

And then all you need to do is you somehow need to get their power usage into the simulation. You can either do that by modeling. You can, I don't know, look at the CPU usage of your application and somehow estimate how much power it would use. Or you actually let it run on a machine, plug a proper power meter, and then periodically feed this information into the simulation.

And the simulation itself you configure as you wish. So we have a lot of READMEs. You can plug and play custom simulators. We have some implemented, we're right now implementing the ones from the National Renewable Energy Laboratory. They have lots of simulators on everything from solar to wind to geothermal and whatsoever.

So you can plug your microgrid together. Also provided with energy storage. We are releasing Vessim 0.9 this week, I think, which then comes with a few more energy storage solutions. So you can really plug this together as you wish. And then you can write your own API basically to access this data center. The new applications, as in normal carbon-aware computing, can just query a REST API or something and really ask "what's the state of charge of my battery, how much energy is produced right now, what's the forecast for carbon intensity," all of this.

Chris Adams: Okay so I'm going to break that down a bit because there's a couple of quite interesting concepts that you spoke about. So first of all I feel a bit embarrassed by not mentioning this, we just assumed it, of course, was open source, but yes, this is entirely open source, anyone can access it, if they search for Vessim, V E S S I M, they'll likely find the project, because there aren't that many other things called Vessim right now, is that the case?

Philipp Wiesner: I hope so, because we were searching for a searchable name. 

Chris Adams: We'll also share links to the actual project on GitHub as well. It's written primarily in Python as I understand it, so if you know Python or know someone who works in Python, and given that Python's basically the most popular programming language on Earth, or by various metrics, is one of them, then the chances are fairly high that you can start doing this.

Are there, like, presets for different grids? So in the same way that you might have a preset for, I'm viewing a website on a mobile phone, or I'm viewing a website on a desktop, you said a little bit about NREL, the National Renewable Energy Laboratory. They've said, "well, this is what the, this is how a geothermal plant might work, or this is how a wind turbine might work and this is solar panel might work."

And that gives you an idea of saying, "well, this is somewhat similar to my grid." And then there's kind of, are there some presets or things so people can come up with some approximation of what it might be in their part of the world?

Philipp Wiesner: Not yet publicly. We're working on this and we will publish it soon. But right now there's no presets, but Vessim comes with some data sets that were provided. One was provided by Watttime on carbon intensity. One was provided by Solcast on solar data. And you're right, Vessim itself is in Python, but we're like building on a library called Mosaik.

Been developed in Oldenburg, also in Germany, and this is basically the entire co-simulation system behind Vessim. And they allow you to basically plug and play any type of simulator that can be written in Java, or it can also be a real system. You can also just take a real battery and measure that and just feed this information.

So it's really very flexible in that sense.

Chris Adams: Ah, cool, I'm glad you mentioned about the power use thing, because that's the thing I was going to speak to you about, because you said that you can draw power from a physical system, so you might time series of power usage, but one thing that we've spoken about previously with other guests on this podcast is that if you know what kind of compute usage, how much of your compute of a server is, you can make some I'm a bit worried about using the word inference when we've just spoken about AI.

But like, you can, there'll be some idea of what the likely power might be, just because machines tend to use more power at 100 percent utilization versus maybe 50 percent utilization. There's various curves and things that give you some numbers. And I believe there's an organization, Green Coding Solutions, who share some of this.

But also, I think this is under the hood what, other projects like Kepler also have some reference to. So, there are different ways that exist right now to give you some numbers. Even if you don't have something plugged into a system, for example. Maybe, you don't have access to, a power meter like you're deploying in a cloud, for example.

Philipp Wiesner: Exactly. Yeah. And this is cool that you mentioned that because this is one of the, it's a master thesis that we are running right now with one of the students who has also been developing Vessim for the last two years. And we are also using Kepler and trying to really calibrate Vessim for real systems.

And of course, these calibrations can be published. So we can really see, because there's this big difference between software-based power metering and like actual hardware power metering. In software-based power metering that's like included in Intel processors and also NVIDIA graphics cards, they just estimate how much power do I draw right now.

But it comes with a certain error. And what we're doing is like we're benchmarking these machines, using the integrated software-based power metering that actually everyone has access to but also like plug a real power meter and see how big is the difference under different conditions and so on and try to, yeah, make Vessim a bit easier to configure for bigger systems. Because this is the entire idea, maybe you have access to one GPU that you can benchmark but then you want to explore, "okay, how does my system behave if I have 200 GPUs?" And you can scale up in the simulation.

Chris Adams: Okay, so when we, so when someone is starting with this, like, we're going to talk about actually running the software, but we're just setting the scene to figure out, okay, what does the setup look like? Because I think most developers might be familiar with the idea of running a test to see if it works as intended.

There's this whole new world outside of your computer system that we need to talk about that I think you might need to kind of, that people need to understand before they start thinking about what, how the system is supposed to be designed. So we spoke a little bit about the fact that Vessim is using a existing tool called Mosaik, which designed for simulating grids.

So someone's done all that work already to figure out how to simulate a grid and you're using, you're relying on some of that. So there's a bunch of work that's gone into that already that you can draw upon. And we spoke a little bit about, there are ways that you can get readings on power, so even if you don't have something plugged in, you can still get some maybe low precision but still useful numbers to help inform some of your, like, design choices.

And then, now you've got that part there. You said a little bit about maybe things being in the, software being in-the-loop here, and I think while I've spoken excitedly about the notion of ecovisors and things on previous podcasts, I think that's what you're referring to here, like if the software itself knows how much, how full a battery is, or how green the power is, it might change in certain ways or it might respond to certain things.

Could you talk a little bit about that part? Because I think the software in-the-loop part is the thing that most people who've been taking an existing project and maybe trying to extend it to be carbon-aware. That's where they might be spending their most times either thinking about how to test something or figuring out how to make and adapt their existing system so they can start coming up with some metrics.

Philipp Wiesner: Yeah, It's interesting that you mentioned Ecovisor because it was a big inspiration for this project. Because I think they spent tens of thousands of euros on this one solar array simulator and a testbed. And we wanted to, like, explore similar ideas, but we didn't have the funding at the time to just buy such a machine.

So I had to write a simulator, more or less. So this is how it all started. Sorry, what was the question?

Chris Adams: in-the-loop, software in-the-loop,

Philipp Wiesner: Ah, software-in-the-loop, sure. Yeah, 

Chris Adams: We've simulated all this stuff, we know roughly how much power we're using, so the system, the software now has the data going into it. So what happens next? That's the thing.

Philipp Wiesner: Yes. So the idea is we need to run the simulation in real time, right? In wall clock time. So one second of simulated time needs to be the same as one second of real time. And if we manage this, then we can simply from the simulation, host any kind of API. We can just spin up a REST API that gives us access to the system.

And in Vessim, users can define these APIs themselves. So in Ecovisor, for example, they already came up with an API. I don't know, "/battery" and then you can query the battery state of charge, but something like this. Vessim lets you explore this. How do we abstract the complicated energy system and how do we like expose relevant information for the computing system?

And then you can just run a real normal application running on your computer, running in a docker container whatsoever. And it can send requests to this energy system API. And best case, if we did everything correctly, it behaves exactly as if you were doing a request to an actual energy system. So your application doesn't know that it's working with the simulation.

Chris Adams: Okay, so I'll try and make this a bit concrete. So when I'm working on a project, in an existing kind of carbon-aware system, I might make an API call to like, I might ask electricity maps, "Hey, what's the local grid intensity where I am right now?" For example, and like, it's a little bit like that kind of request I'd be making, but I might be making it to Vessim to say, "what's the grid intensity?

What's the state of my charge or stuff like that?"

Philipp Wiesner: Exactly. You just write a normal application that is running on a system, just that the system is power metered, so Vessim can simulate how it would be behaving if it was plugged to batteries, for example, and then it can talk to this simulated energy system.

Chris Adams: Gotcha. Okay. And we're using Vessim as just the kind of, simulator to share this stuff and it may be that in the real world, like I might be, speaking to the electricity maps API or something like that.

Philipp Wiesner: Absolutely, would be, yeah, absolutely.

Chris Adams: Okay.

Philipp Wiesner: A plug and play thingy, yeah.

Chris Adams: All right, that part, I think I understand. So then at that point, I'm then able to basically say, well, let's imagine electricity maps has given me this figure for carbon intensity, or let's imagine my, the state of my battery is high rather than low, I then might make some changes and say, well, "When the energy is low, don't do this stuff, or just delay it for maybe half an hour to check it again."

So that's the kind of stuff. So maybe you could talk a little bit about, now that we're at that point, how does that manifest into me seeing if that has actually reduced carbon emissions over a period of time, for example. Maybe we could talk about that part, because I think that's the final thing, now that we've set up our system, we've connected to, we've figured out how to get data coming into my system to tell me what the carbon intensity might be, or what batteries might be, or what the power from a wind farm might be. So yeah, maybe you can talk about that. So over a given time, has this actually helped, for example?

Philipp Wiesner: Yeah, And this is actually what makes simulations so powerful, because you can just execute a bunch of different scenarios. You can execute your scenario one time without any carbon-aware strategies. You can execute it a second time with the carbon-aware strategies. You can simulate as if you were executing it in summer or winter, or you can simulate power outages and whatsoever.

Because it's all really fast and really cheap to do. And then of course Vessim is recording all these metrics. Like for every second it's recording how much power has been drawn, produced, how much emissions were there. So afterwards you can analyze these results and compare the different runs.

Chris Adams: Ah, okay. I think I'm understanding this now and I

kind of figure out how I might build some of this now if I was to be building maybe a batch job, for example. I can kind of maybe imagine how this might work for maybe a content management system or like a CMS or WordPress or something like that. So let's say I would do, would I be looking at like, this would be my traffic the last month, run it through and then compare it to what the carbon intensity might have been.

Is that how I might do this?

for 

Philipp Wiesner: Yeah, absolutely. Absolutely. And you can also use it in testing pipelines, for example, in continuous integration pipelines. You can just simulate a bunch of different scenarios for your carbon-aware application. Some of them might be power outages or something like really edge cases that you do not want to happen, but you can really test cheaply that your behavior, that your system or your application would behave as expected

Chris Adams: Ah, 

Philipp Wiesner: In a reproducible manner.

Chris Adams: So you've raised something quite interesting for me here because you, when you spoke about the idea of like things, power outages or stuff like that, this kind of implies that if I've got a system here, I could see to what degree I might make a change to the software, or I might just say, well, given I have control over my infrastructure, I'm running a data center, what if I was to just put a battery inside that data center?

How would that impact the carbon footprint of using this system every month, for example? Would that, could that be something that I could simulate with that?

Philipp Wiesner: Yes, absolutely. Absolutely. And if you have a model that is able to, like a battery model, a simulator that is able to simulate battery aging, for example, you can also simulate what's happening over the course of the next three years with my batteries and so on.

Chris Adams: Ah, okay, so it's actually two way So therefore, I can look at, I can, look at, say, making interventions at my code level, and I can interventions at the kind of deployment scenario level as well, actually. Okay, that is something I hadn't actually realised before then. Okay, so we've dived into this quite a bit and I hope there are at least some people following us now.

Can we just take a step back to say because we have just been assuming that carbon emissions or carbon intensity is absolutely the figure we should be looking for and that is our North Star and you mentioned before that maybe that isn't or there's a bit more to it than that because we can't just look at this number, see if it's going down, and say, "Oh yeah, we're making progress."

Maybe you could outline on some of that, because it's a lot of nuance here that people often miss, I think. 

Philipp Wiesner: Ah Yeah, sure sure sure. So carbon intensity is a metric that defines how clean or dirty the energy in the public grid is. That means if it's low, you can use it if it's high, you should maybe rather not. And the problem here is that it's hiding a lot of the underlying complexities of how grids work and how energy markets work.

So for example, one big problem that we have in energy markets is, like in carbon accounting in general, is this double spending of guarantees of origin certificates. So for example, you can easily buy certificates produced in Iceland or something. They produce almost all the energy through renewable sources, hydropower, geothermal power.

You can buy their energy and then tell everyone you're 100 percent renewable energy. But at the same time, of course, any data center that is located in Iceland will tell you that they're 100 percent renewable, which is also true. and in fact, there's not even a power line between Iceland and mainland Europe, and still you can buy their certificates.

so we have a lot of double accounting and of course, everyone is coming up with new methodologies of what's good and what's the right thing to do. And of course, everyone's also just doing what's in their favor. And this is the big problem with carbon intensity. So of course you can say now "carbon intensity in Germany is really low.

I should consume energy." But unless you have a green energy contract, you're for sure only getting the dirty fraction, because all the green energy is already sold.

Chris Adams: So let me just check if I can stop you there for a second. So what you spoke about just then was there are kind of two approaches or like there's one one approach of looking at how kind of green power is might be the existing kind of system that you see in most developed energy markets right now where there's like a market-based system where you essentially can purchase renewable energy certificates and apply that to kind of brown energy to then say it's green.

And there may be reasons why we have ended up with this scenario. As you said, there are two problems there when we talk about, say, the idea of buying green power certificates from Iceland and then applying them to dirty power in, say, Poland or Germany to say it's green. There's like a, there's one issue which is that there's literally no way to connect, there's no connection, like we definitely are not using geothermal that's generated in Germany to get there. There's no physical connection so it's not deliverable. And the other issue is that if Iceland was to do that, they would then have to stop saying their power is green because they've already sold on their greenness. And in many cases they're not doing that. They're saying, "oh, we're still green even though we've solved the greenness."

So, there is a kind of problem about the, yes, let's assume that the market, the concept was conceptually sound. There is a whole issue about people kind of cheating a bit and saying no, it's still green when they've sold the greenness on it already, basically.

Yes, 

Philipp Wiesner: and exactly in this sense, optimizing for carbon intensity is cheating, because you're doing exactly this. You're like saying, "I'm using low carbon energy," but all the green, as you said, has already been sold. At the same time, it makes sense. The metric also makes sense. Optimizing energy usage for when the sun is shining and the wind is blowing, of course it makes sense.

But the big problem is that these green energy certificates, these guarantees of origin certificates, are traded independently of the physical flow of energy.

Chris Adams: Ah, okay. Okay, so there is, there's a whole discussion about this, like, where we work at the Green Web Foundation, we're kind of borrowing some of the ideas we've seen in the hydrogen sector, where they talk about if you want to say energy is green, it needs to be, I believe it is, timely, deliverable and additional.

So you need to be new power, like the, you mentioned this example of nuclear being used in other data centers, where essentially you're reallocating power that would have been going into the grid. So everyone has slightly greener power, but you say, no, that all has to go over the data center. Then that has the impact of essentially making the power that everyone else is using be seen as slightly less clean because of this accounting system, and that's one of the unfortunate artifacts of that. So that's one thing you mentioned about that. And I think the system that you might be talking about was the idea of like a location-based system, which is much more about like the physical environmental footprint of the power you're using rather than, say, the market-based approach, right?

Philipp Wiesner: Exactly. So we need a system that is somehow time based and location-based, a certificate system, then this all would make a lot more sense. But even then, grids are very complex and it's too easy to boil it all down to a single number and say at Germany, for an EPM we have this number of carbon intensity.

It would maybe make more sense if we, like, break down everything into smaller pieces, again, go more local. I have, for sure, Northern Germany separated from the South and stuff like this. But still, it makes sense, and this is what, for example, Vessim is good for, to really think about and understand how is energy actually flowing, how is the energy system actually working, because it's a bit more complex.

Chris Adams: Ah, I see. Okay, so for people who want to go down this weird accounting rabbit hole that we just kind of skirted around, I will share a link to a journalist called Hanno Böck who's dived into the world of Iceland, and in particular about this whole idea of selling green energy twice like this, because if you are looking into this, it's, as we end up with a much more decarbonised grid, this is going to come up again and again.

And right now, we do see the systems being used, like the GHG protocol, the people who actually define these standards. They're going through a process of reform and basically saying, "well, these are the changes we need to make and this is why this is possibly not fit for the purposes that it was 10 years ago," for example.

So there is a chance to make interventions and talk about this, if you want to look into this as well as looking into the other things we spoke about with Vessim. Okay, so we've spoken about this idea of setting up a kind of testbed that we've established that this is kind of open source, that anyone can pick up and start playing with right now, like the code is open source on GitHub, and as I can see, documented well with like, videos and stuff like that.

We're just coming to time, so I just want to give a bit of space for some of the future events that are coming up, because I understand that, well, part of your job, as you told me, is like, I'm the publicity chair for this all about low carbon computing, so I should probably give you a bit of space to talk about that as well.

Philipp Wiesner: Yeah, that's super nice. Thank you to just take two minutes. So we're organizing this workshop. It's the first workshop for low carbon computing, which will take place on the 3rd of December in Glasgow, and It's mainly an initiative, a local initiative from some folks at the University of Glasgow, but it's an international conference, it's an international workshop I mean.

And we expect submissions from all over the world, but mainly probably the European community, I guess. And yeah, welcome everyone to submit your work. So we have a very broad scope on really anything related to sustainable computing, from energy efficiency, carbon efficiency, embodied carbon and so on, but also topics that are often a bit underrepresented in computer science like circular economy, sufficiency, demand reduction, and all of this. exactly. So this is all very welcome. And for regular talks, all you need to submit is like a two page extended abstract by the 24th of September. Although we're like currently discussing maybe a tiny deadline extension and all accepted authors, all accepted papers will afterwards be invited to submit like a full workshop paper of up to eight pages if you're interested, which will be published in our post proceeding.

So yeah, feel free to have a look. I think we will also put a link in the

Chris Adams: a link in the show notes, for sure, absolutely. So if I understand this correctly, is this a little bit like, possibly, a kind of like a sister conference to Hot Carbon, but where it's a little bit colder, and you're more likely to kind of find a burrito served at Legal Jack's.

Is the name of a restaurant in Glasgow that eaten at before.

Philipp Wiesner: Yes, I guess like it's the first version. Let's see where it's going. But maybe we can, if it's going well, we can maybe establish it as like a European alternative to Hot Carbon or something, which will, I guess, take place in the US in the upcoming years. 

Chris Adams: I see. And there's one thing I wanted to just check because when I looked at the kind of call for papers, the set of topics, it's a little bit more wide than just like let's focus on efficiency. There seems to be a kind of systemic angle that I hadn't seen quite in the same way, for example. So there are discussions about like you said,

okay, looking at the demand side as well as just the supply side, for example. And like, I think I've even seen the word frugal computing used inside here. Like, okay, this is almost like somewhat possibly like the limits conference or maybe some of those ideas somewhat comparable to inside this. 

Philipp Wiesner: Yeah,

guess, yeah, I guess it's somewhere in between that, in between Hot Carbon and limits, maybe.

Chris Adams: cool. All right. Well, I am now thinking it might be nice to figure out a way to get a series of trains to get up to Glasgow, because, I really like that city.

Philipp Wiesner: Yeah, 18 the last times.

Chris Adams: Okay, cool. So that's one thing that's taking place. So the deadline was the 24th of September. Easy to apply to, there is, and that'll be taking place in December.

Philipp Wiesner: Just one last thing, it's a hybrid event. It's a hybrid event, so anyone is welcome to participate, and also authors are of course welcome to present remotely.

Chris Adams: Okay, so I don't need to get on that if I really don't want to, because, well, I might want to see people, but it takes quite a long time

Philipp Wiesner: only a one day event, so you need to consider if it's worth two days of travel.

Chris Adams: Alright, thanks for that Philipp, I appreciate that and we'll share a link to that so people know when the CFP is open how, well actually when it closes more specifically because I think it is open right now. Okay, cool. So I'll just touch on some other events that we have inside here and then I think we'll wrap up if that's okay with you.

So these are some of the other events that our producer Chris Skipper has shared. So there's an event called CodeGreen, the intersection of software engineering and sustainability. This is on September the 12th at 11am AEST. This has Sara Bergman from Microsoft Norway, and one of the, one of the authors of the book, Building Green Software, and also a regular on our podcast.

She's presenting, speaking with Michelle Sandford. And there's also an event in Vienna, in Austria. This has Asim Hussain, our Executive Director of the Green Software Foundation. He's talking about the Impact Framework, this radically transparent tool for sharing the environmental impact of your tooling, and that's on September the 16th, and I think that is in person, but then maybe we'll share a link for people to see about if they can get involved remotely.

There's also another event in the UK, Engineering a Greener Future for IT, that's on September the 18th at 5pm British Standard Time in London. And there is also another event in Brighton as well, so the UK is quite heavily represented right now, Collaborating on Digital Sustainability, that's September the 18th.

I'm going to use this platform while I still can. I'll share a link to Green IO London as well, which is the conference that I'll be keynoting at with a number of luminaries. We'll share a link to that because I, it's very front of my mind right now and I'll be there for the day. So if you listen to this and you fancy coming along, we'll share a link where you can get a free ticket to join at the conference.

And yeah, that's us. I think we've come to the time. We'll just actually make sure that all the things we've discussed, like I mentioned before, we always try to get as many of these show notes full of links as possible, because we cover a lot of ground on these free wheeling conversations, so that will be visible.

The thing you might need to do though, if you are listening via Spotify or the Apple platform, you might need to look up podcast.greensoftware.foundation, then follow the link to this podcast to see all the links, because I don't think they're made available if you just are listening to it in Spotify, for example.

And we do put quite a lot of time into doing that. And Philipp, I just want to say thank you so much for helping unpack what can be an extremely complicated topic and putting the work in to make a open source project that makes it easy for people to figure out what to do with carbon-aware computing and hopefully guide themselves to slightly more sustainable software.

Philipp Wiesner: Yeah, no, thanks a lot for having me. Yeah, had a lot of fun. It was a pleasure. Thank you.

Chris Adams: Okay, final question I should ask, if people do want to find out about you, where should people look? Are you on LinkedIn? Are you still using X slash Twitter slash what's the best way to follow your work these days?

Philipp Wiesner: Probably just Google me, I guess there's maybe one or 2, 3, 4 Philipp Wiesners that are more important than me, but if you just look at TU Berlin or Philipp Wiesner carbon-aware where I should pop up and then you can see.

Chris Adams: Cool, well your name will be in the show notes and the show title, so if anyone is curious about how to spell Philipp Wiesner and not sure how many P's they should be using, then that'll be inside there. All right, Philipp, really lovely chatting to you, mate, and thank you so much for coming along, and yeah, have a lovely week.

Philipp Wiesner: Thank you. You too. Thanks a lot.

Chris Adams: Cheers, bye!

Philipp Wiesner: Bye.

Chris Adams: Hey everyone, thanks for listening! Just a reminder to follow Environment Variables on Apple Podcasts, Spotify, Google Podcasts, or wherever you get your podcasts. And please, do leave a rating and review if you like what we're doing. It helps other people discover the show, and of course, we'd love to have more listeners.

To find out more about the Green Software Foundation, please visit greensoftware.foundation. That's greensoftware.foundation in any browser. Thanks again, and see you in the next episode!