Real Time Cloud with Adrian Cockcroft

Chris Adams is joined by Adrian Cockcroft, former VP of Cloud Architecture Strategy at AWS, a pioneer of microservices at Netflix, and contributor to the Green Software Foundation’s Real Time Cloud project. They explore the evolution of cloud sustainability—from monoliths to microservices to serverless—and what it really takes to track carbon emissions in real time. Adrian explains why GPUs offer rare transparency in energy data, how the Real Time Cloud dataset works, and what’s holding cloud providers back from full carbon disclosure. Plus, he shares his latest obsession: building a generative AI-powered house automation system using agent swarms.

TRANSCRIPT BELOW:

Adrian Cockcroft: We figured out it wasn't really possible to get real time energy statistics out of cloud providers because the numbers just didn't exist.

It turns out the only place you can get real time numbers is on things that are not virtualized.

Chris Adams: Hello, and welcome to Environment Variables, brought to you by the Green Software Foundation. In each episode, we discuss the latest news and events surrounding green software. On our show, you can expect candid conversations with top experts in their field who have a passion for how to reduce the greenhouse gas emissions of software.

I'm your host, Chris Adams.

Welcome to Environment Variables, where we bring you the latest news and updates from the world of sustainable software development. I'm your host, Chris Adams. If you have worked in cloud computing for any length of time, then even if you do not know the name yet yourself, it's very likely that the way you design systems will have been influenced by my guest today, Adrian Cockcroft.

When at Netflix, Adrian led the move to the cloud there helping, popularize many of the patterns we use when deploying applications ourselves to the cloud. And his name then became synonymous with serverless throughout the 2010s when he joined AWS first leading on open source engagement, and then as a VP focused on what we might refer to now as cloud sustainability.

After leaving AWS, Adrian's kept his fingers in many pies, one of which is the Green Software Foundation's real time cloud project, an initiative to bring transparency and consistency to cloud emissions reporting. With the first dataset release from that project out the door, it seemed a good idea to invite him onto the show to see what's up.

Adrian, thank you so much for joining us today. Can I give you a bit of time to tell us about yourself and what you are, what's what you're keeping? What's keeping you busy these days? I.

Adrian Cockcroft: Yeah, it's great to see you and thanks also for your contributions to the project. We've had a lot of discussions over the last few years as we've worked on that together. well, I'm sort of semi-retired. I stopped my big corporate job at Amazon in 2022. and yeah, I spend my time worrying about my family.

I've got old parents that live in the uk, so I spend a lot of time with them. And, fixing stuff around the house and generally goofing around and doing things I feel like doing rather than stuff that's driven by some corporate agenda. So I'm enjoying that freedom. And, let's see the, yeah, I spend time on the, Green Software Foundation project.

I go to a few conferences and give a few talks and I try to keep up with, you know, what's happening in technology by playing around with whatever the latest tools are and things like that. And that's been my career over the years. I've generally been an early adopter through my entire career. as you mentioned, we were early adopters in cloud.

Back when people said This isn't gonna work and you'll be back in the data center soon. People forgot that was the initial reaction to what we said. it's a little bit like that now with people saying, all this AI stuff doesn't work and we're gonna be giving up and whatever. And it's like, well, I'm making bits of it work well enough to be interesting.

We can talk a bit about that later. and then I know you probably see behind me various musical instruments and things like that, so that's kind of, I, collect musical instruments that I don't have time to really learn how to play and mess around and make bad noises that make me happy. But luckily no one else has to listen to them particularly.

So that's kind of my, that and messing around with cars and things, that's sort of the entertainment for me.

Chris Adams: That sounds like quite a fun, state of stem semi-retirement, I have to say actually. So before we dive into the details of cloud, I have to ask, where are you calling from today Because you have an English accent and like, I have an English accent, but I'm calling from Berlin and I'm guessing you're not in England, so maybe you could do that.

'cause I follow you on social media and I see all these kind of cryptic and interesting posts about cars and stuff and it's usually sunnier than where I am as well. So there's gotta be a story there. What's going on there, Adrian?

Adrian Cockcroft: Well, I lived in England long enough to decide I didn't want to be rained on all the time. which is why I never moved to Seattle when, you know, I didn't move to California to move to America to go live in somewhere with the same weather as England. So that was one reason I never moved to Seattle when I was working for Amazon.

So used to live in the Bay Area in Los Gatos, near Netflix. about five years ago we moved down near Monterey, about an hour or two south of the Bay Area. I. Depending on traffic. we are within earshot of a race track called Laguna Seka that most people know. I can kind of see it outta my window.

I can see a few dots on the horizon on the, you know, moving and that's, there's a few cars you can just about hear them on if they're loud cars. and this is where they have in every August, this thing called Monterey Car Week with the Pebble Beach concourse and historic races. And we used to go to that every year and we like the kind of messing around with cars and going to the track occasionally culture.

So we moved down here and that's been, it's been fun. It's, you know, I don't have to commute anywhere. We have a nice place. The house prices are a lot cheaper down here than they are in the Bay Area itself. So we live in, technically we live in Salinas. lots of good vegetables around here. That's where a lot of the growers are.

and it's, we live actually out in the countryside, sort of. Just in the hills near, near there. So we have a nice place, have plenty of room for messing around and a big house, which requires lots of messing around with. And we can talk a bit about one of the projects I have later on to try and automate some of that.

Chris Adams: Yeah, that's quite a hint. Alright, well that does explain all the kind of cars and coffee stuff when I, like say 30 verse and Okay. If you're near a racetrack, that would explain some of the cars as well. Alright. Thank you

Adrian Cockcroft: Well, actually there's cars and coffee events just about everywhere in the world. If you, like looking at old cars and hanging out with car people, there's one probably every Saturday morning somewhere within 10 miles away. Pretty much anyone. Anyway, the other things, on that front that's sort of more related to Green Software Foundation is we've had a whole bunch of electric cars over the years.

I have one of the original Tesla Roadster cars that was made in 2010. I've had it since 2013. it actually has a sticker on the back saying, I bought this before Elon went nuts. so I'm keeping that. we used to have a Tesla model three and we replaced it recently with a Polestar three, which is quite a nice car with very bad software initially.

But they did a software update recently that basically fixed just about every bug and we, it's actually fun driving a car where you don't worry if it's about to do something strange and need a software reset, which was the state it was in when we first got it in April. But the difference, a bug fix can make whether they actually went and just fixed everything that was currently going wrong with it and went, transformed the car into something That's just actually a fun thing to drive now.

Chris Adams: So it was a bit like turning it off and turning it off and on again. And then you've got like a working car,

Adrian Cockcroft: Yeah. Well, yeah, we got really used to pushing the reset button. You hold the volume control down for 30 seconds and that resets the software and we would be doing that most days that we drove it

Chris Adams: Oh my God. I didn't realize that was a real thing that people did. Wow.

Adrian Cockcroft: Yeah. It's one of these things where a product can be transformed from something buggy and annoying to, oh, we just fixed all the software now.

It actually works properly. And, you know, it's, interesting to see. So, so it went from bad, really bad to actually pretty good with one software release. Yeah.

Chris Adams: guess that's the, wonders of software I suppose. Wow. Alright then, and I guess that gives us a nice segue to talk about, I guess some back to some of the cloud and serverless stuff then. So. Before you were helping out in some of the Green Software Foundation projects. I remember reading a post from you called the evolution from Monoliths to microservices to functions.

And I think for a lot of people it actually really joined the dots between how we think about sustainability and how things like scale to zero designs, might kind of what role they play when we design cloud services. And in that post, you laid out a few things, which I found quite interesting. You spoke about the idea that like, okay, most of the time when we build services, they may be being used maybe 40 hours a week and there's 168 hours a week.

So like 75% of the time it's doing nothing. And just like waiting there. Yet we've still spent all this time and money building all this stuff and, post. I remember you writing a little bit about saying, this actually aligns incentives in a way that we haven't seen before. And I think this idea of actually like changing the programming model that actually incentivizes the correct behavior.

I think that's really, that, that was really profound for me. And I figure like, now that I've got a chance to have you on the call on this podcast, I wanted to ask you what drove you to write that in the first place? And for folks who haven't actually read it, maybe, you could just talk a little bit about the argument that you were making and then why you wanted to actually write that as well.

Adrian Cockcroft: Yeah, that's actually one of the highest traffic blog posts that I ever wrote. There was a lot of, reads of that. The context then, so it was soon after I joined AWS, so it was probably 25. Early 2017, something like that. I joined AWS in 2016. I'd spent a few years basically involved in kind of, helping promote microservices as an architecture.

And, I was also interested in serverless and AWS Lambda as, an architecture. And I wanted to connect the dots. And it's a kind of, when I write things, some of the things I write, the approach I take is along the lines of his, this is how to think about a thing, right? These are the, it, I have a systems thinking approach generally, and so what I do is I try to expose the systems that I'm thinking about and the incentives and feedback loops and reasons why things are the way they are, rather than being prescriptive and saying, just do this, and this.

I. And the world will be great, or whatever the, you know, the more typical instructive things. So I tend to try and explain why things are the way they are and, sort of work in that. So that's, it's, an example of that type of writing for me. And we were, at the time, people were talking a lot about the monolith and microservices transition and what it meant and how to do it and things like that.

And I was trying to explain what we'd done at Netflix. And then I was thinking that there was a, the next generation of that transition was to serverless. And the, post was basically to just try and connect those dots, that was the overall goal of it. And then it is quite a long post. It's one of these things when you work with somebody, you know, PR people or whatever, and they say, you, you should write short blog posts and you should, you know, da Well this, and they shouldn't be so technical. So this is one of the longest and most technical posts I wrote, and it actually has the highest traffic. So, you know, ignore the PR people. It turns out if you put real content in something, it will get traffic. and, that's, the value you can, provide by trying to explain an idea.

So I think that's generally what that was about. This idea that. it was, I mean, the microservices idea was, is a tactic for implementing a for solving a problem. It isn't an end in itself. Right. And that's one of the distinctions I was trying to make. It's like if you have a large team working on a code base, they'll keep getting in each other's way.

And if you're trying to ship code and the code has a hundred people's contributions in it, one person has a bug, then that stops the shipment of the other 99 people. So there's this blocking effect of, of bugs in, in, in the whole thing. And then it also, you've got it destabilizes the entire thing.

You're shipping completely new code when you ship a new monolith was when you have say a hundred microservices with one person working on each. They can ship independently. And yeah, you have some interaction things you have to debug, but 99 of those services didn't change when you pushed your code. So it's easy to isolate where the problem is and roll it back.

So there's a bunch of things that make it easier. And then we thought, well, you've got the microservice, which does a thing. But it contains a bunch of functions. If you blow that up into individual functions, then you don't actually need all those functions all the time. And some code paths are very busy through the code.

They may be do it a hundred times, you know, every request goes through this part of the code, but may one times in a hundred or a thousand it does something else. So what you can do is break those into separate functions and different lambda functions. And you've got, so the code parts that don't get executed very often just aren't running.

The code gets called and then it stops and it's doesn't get called again, for a long time. Whereas the busy ones tend to stay in memory and get called a lot. Right. So that way you're actually, the memory footprint is more tuned to, and the execution footprint is tuned to what's actually going on.

So that was, the second thing. And then the third thing was that a lot of applications, particularly corporate in access, you mentioned they're only used during work hours. And those are the perfect ones to build serverless. They're internal. They are, they only exist for as long as anybody is actually trying to use them.

And they aren't just sitting their idle most of the time just because you need to have a wiki or something, or you need to have a thing that people check in with in the morning. Like anything that salespeople at the end of the quarter or the end of the month, those sorts of things make things super busy and it's idle the rest of the time, so you need very high concurrency for short periods of time.

Anything like that is, is sort of the area where I think serverless is particularly good. And later on I did another, series of talks where I basically said serverless first, not serverless only, but start trying to build something with serverless because you'll build it super quickly. And, one of the books I should reference is by, David Anderson.

is it called the Value Flywheel Effect or something like that will give a link in the show notes. And I helped. Talked, I, talked to him, helped him get, find the publisher for that book. And I wrote, did I write, I think I wrote a foreword for it, or at least put some nice words on the cover.

and that book talks about people developing app, entire applications in a few days. And then you get to tune it and optimize it. And maybe you take some part of it where you say, really, I need a container here. Something like that. but, you can rapidly build it with the tag I used to say was in the time it takes to, have meetings about how you're going to configure Kubernetes, you could have finished or building your entire application serverless, right?

And, you just get these internal discussions about exactly what version of Kubernetes to use and how to set it up and all this stuff. And it's like, I could have finished building the whole thing with the amount of effort you just put into trying to figure out how to configure something. So that's the sort of, a slightly flippant view I have on that.

And, anyway, the, other thing is just, and effectively the carbon footprint of a serverless application is minimal. But you do have to think about the additional systems that are running there all the time when you are not running. And a little bit of a, sort of a future segue, but AWS just changed them, their own accounting model to include those support services so that, when you look at the carbon footprint of a Lambda app that isn't running, you actually have a carbon footprint because the Lambda service needs to be there ready.

So you actually get a share of the shared service attributed to each customer that's using the, using it, right? So it's a little, it's a little bit deeper and it's kind of an interesting change in the model to be explicit that's what they're doing.

Chris Adams: Ah, I see. Okay. So on one level, some of this post was about like the, I guess the unit of code or the unit of change can become smaller by using this, but there's also a kind of corresponding thing on the hardware level. Like, you know, typically you might be, I remember when I was reading this, there was like, okay, I'm shipping a monolithic piece of code and I've got a physical server to begin with.

It's like the kind of. That was like how we were starting at maybe, I dunno, 10, 20 years ago. And then over time it's becoming smaller and smaller and that has made it a bit easier to build things kind of quickly. And, but one of the, flip side that we have seen is that, if you just look at say the Lambda function, then that's not a really accurate representation of all the stuff that's actually there.

You can't pretend that there is an infrastructure that has to be there. And it sounds like the accounting has now starting to reflect that the fact that yeah, you, someone needs to pay for the capacity in the same way that someone has to pay for the electricity grid, even though you're not, even when you're not using the grid for example, there is still a cost to make that capacity available for you to use.

Basically that's, what it seems to be a reference to.

Adrian Cockcroft: Yeah. And just going back to the car analogy.

People own cars. People lease cars. People rent cars, right? And you can, if you rent a car for a day, you can say, well, my carbon footprint of renting the car is one day's worth of car ownership, right? Except that in order for you to rent a car for the day, there has to be a fleet of cars sitting around idle That's ready for you to rent one. So really you want to take into account the overhead of your car rental company managing a fleet, and it's maybe got whatever, 70% utilization of the fleet. So 30% of the cars are sitting around waiting for somebody. So you basically have to uplevel your, I just need a car for a day to add an extra overhead of running that service, right?

So it's, it kind of follows that same thing, you know? And if you basically rent a car for every single day and you have a car every day of the year, but it's a rental car, that's an expensive way to own a car, right? I mean, even at a monthly rate, it's still more expensive than buying a car or leasing a car because you're paying for some overhead.

But it's kind of those sorts of models. So it's a bit like owning a car, maybe leasing a car, and, doing a rental car with sort of the monolith microservices. Serverless sort of analogy, if you like. cost model's a little different because, you're giving stuff back when you don't want it anymore.

is sort of the cloud analogy, right? The regular cloud service. I can just deep, I can scale things down.

Chris Adams: mm going back to something else you mentioned, I was talking to a CIO once and he was very annoyed 'cause he said that he'd only just found out that he could turn off all his test infrastructure at the weekends and overnight. and it was like they, he'd been running this stuff for two years and this, he finally realized and, he'd just, like, three quarters of his cost had just gone away from his test environment. And, he, was happy that had happened, but he was annoyed that it, took him two years for him to somebody to mention to him that this was possible and for him to tell them to do it.

Adrian Cockcroft: Right. So there's. Yeah. Any, tests, anything that's driven off people should absolutely be, you know, shut down. There are ways to just freeze a bunch of a, bunch of cloud instances can just be shut down and frozen and come back again later.

Chris Adams: so this is something I might come back to actually, because one of the things that in somewhat on, in some ways, if you look at, say maybe cloud computing, each individual server is probably quite a bit more efficient than maybe a corresponding, server you might buy from Dell or something like that from a few years ago because it's in a very optimized state.

But because it's so easy to turn on, this is one of the cha challenges that we can consistently have. So it's almost like a, and also in many ways. It's kind of in the interest of the people running very effect, very efficient servers to run, but have to basically have people paying for this capacity, which they're not using.

'cause it makes it easier to then like resell that. Like this is, I guess maybe this is one of the things that the shifts to serverless is supposed to address, or in theory, you know, it does align things somewhat, better and more. More in terms of like reducing usage when you're not actually using it, for example, rather than leaving things running like you're saying actually.

Adrian Cockcroft: Yeah, you don't have to remember to turn it off With serverless, it's off by default and it comes on and it's sort of a hundred percent utilized while you're running and then it turns off again. So in that sense, it is much more like you have a rental car that returns itself after 15 minutes or whatever.

Whatever your timeout

is or when you're done with it. It's more, maybe it's more like a taxi, right? That kind of going, one level beyond rental car, you have taxi, right? Which is you just use it to get there and you're done. So serverless is maybe more like a taxi service, right? And then, right. And then a daily rental is more like a.

Like an EC2 instance or something like that. And there's all these different things. So there we're used to dealing with these things and you wouldn't, you know, you wouldn't have a taxi sitting outside your house 24 hours a day just waiting for you to want to go somewhere, right? People say, well, serverless is expensive.

if you used it in that very stupid way, right?

Chris Adams: wouldn't, you'd, either lease a car or you'd buy a car if

Adrian Cockcroft: Yeah. If you, if it's being used continuously, if you've got traffic, enough traffic that the thing is a hundred percent active, sure you should put it in a container and just have a thing there rather than, waking it up every time, you know, having it woken up all the time

Chris Adams: Ah. I never really thought to make the comparison to cars, to be honest. 'cause I, I wrote a, piece a while back called A demand curve for compute, which compares these two, like, I just like energy for example. Like if you do something all the time, then you have something running all the time, it's a bit like maybe a nuclear power station, like it's expensive to buy, but per unit it makes a load of sense.

And then you work your way up from there basically. So, at the other end, like serverless, there are things like peak plants, which are only on for a little bit of time and they're really expensive for that short period of time. But because they're only on, 'cause they, can charge so much, you'll need to have them running maybe five to 15% of the year.

And that's how they, and that's how people design grids. And like, this idea of demand curves seems like, it's quite applicable to how we think about computing and how we might use different kinds of computing to solve different kinds of problems. For example.

Adrian Cockcroft: Yeah. Well that brings up another current topic. What's actually happening now is the peaker plants are running flat out running AI data centers capacity load, and the peaking is moving to battery, which is now getting to the point where batteries are sufficiently cheap and high capacity, that the peaker capacity is being driven by batteries which respond much more quickly to load.

And, some of the instabilities we've seen in the grids can be fixed by having enough battery capacity to handle, You know, a cloudy day or whatever, you know, the sort of the effects that you get from sudden surges in power demand or supply, right? And once you get enough battery capacity, that problem is soluble that the problem historically as the batteries have been too expensive, but they're getting cheaper very quickly.

So there've been a few, there's a few cost curves that I've seen recently showing that it's actually the cheapest thing to do for power now is, solar and batteries just put that in. And the batteries that they're now getting, originally they were saying you can get a few hours worth of battery cost effectively.

I think they're now up to like six to eight hours is cost effective. And we're getting close to the sort of 12 to 18 hours, which is means that you can go through the night in the winter on batteries. and it's cost effective to deploy batteries to do that. It's something about the economics that means that you have.

A certain amount of capacity, you still need some base load. geothermal isn't particularly interesting for that. I think as one of the cleaner technologies, a company called Vos building a station that, Google are using for some of their energy, I've spent some time looking at alternative energy.

But yeah, those peak of plants, they were sitting there mostly idle, and then all this extra demand suddenly appeared that wasn't in the plan for these big AI data centers and they're hoovering up all that capacity. So people are desperately trying to figure out how to add additional capacity, to take that on.

Chris Adams: We will come to that a little bit later in a bit more detail actually. So, but thank you. So maybe we can talk a little bit about, actually some of this stuff about. Essentially observability and being able to track some of this stuff because one thing that I've seen you present before is this idea of like carbon being just another metric.

And I think, what we'll do is we'll share a link in the show notes to a YouTube video. I called Monitoring Carbon. I think you presented this at Monitorama I Portland in 2022. And the argument that I understood it covers various other, it, it does talk a little bit about like the state of the art in 2022, but one of the key things you were kind of saying was basically as developers, we're gonna have to learn to track carbon because it's just gonna be another thing we have to track.

Just like, space left on a disc requests and things like that. So maybe you could talk a little bit about that and some of the re and just tell me if you think that's still the direction that we're going in. Basically, I.

Adrian Cockcroft: Yeah, so that was the first talk I gave after I left AWS I'd already given, agreed to present there. and then I left AWS I think just a few weeks before that event. so it was kind of an interesting thing. Hey, I, by the way, I quit my job and sort of retired now and, but this is the thing I was working on.

So I was, the last job I had a WSI was a VP in the sustainability organization, which is an Amazon wide organization, but I was focused on the AWS part of the, that problem in particular, the how to get. all of AWS sort of on the same page every, there was lots of individual things popping up. so we and lots of people writing their little presentations about what they thought AWS was doing.

And so we basically created a master PR approved, you know, press, press relations approved, deck that everyone agreed was like what we could say and should say, and it was high quality deck and got everyone to use the same, get on the same, be saying the same thing externally. Now, part of the problem there was that the various constraints we had at Amazon, we couldn't really talk about a lot of the things we were doing for all kinds of reasons.

So the story of Amazon, I think is better than most people think, but the, way it's told is really poor and it's very difficult to get, get things out of Amazon to actually, I. cover what they've been up to. So, so that was what I was working on. And along the way I thought, you know, we need to monitor.

ARM is a monitoring, observability conference I've been to many times and I have a long history in monitoring tools in particular. I thought, yeah, we should, I, should be trying to get everybody to add carbon as some kind of metric. And the problem is, then where do you get that metric from? And that wasn't very obvious at the time.

And I think there's sort of two things that have happened since 2022. One is that we actually haven't made much progress in terms of getting carbon as a metric in, most areas. There's a co with a couple of exceptions that we'll get to, but we haven't made as much progress as I hoped we would. And then the other one is that the sort of standards bodies and.

government regulations that were on the horizon then have mostly been stalled or slowed down, or delayed, whatever. so the requirement to do it from the business has generally come back, has reduced. Right. So, which is disappointing. 'cause now we're seeing even more climate change impacts and, you know, the globe doesn't care whether you're,

what your, corporate profitability or what you're trying to do or you know, what the reasons why you aren't doing it.

But, so we're just gonna get more and more cost from dealing with various types of climate disasters and we're seeing those happen all around us all the time. So, I think in some sense it's got to get much worse before people pay attention. And we're, you know, there's a big sort of battle going on to try and just make it, keep it focused and certainly Europe is doing a much better job of.

Right now. but even, the European regulations are a little watered down. And that's, I mean, I know that you are all over that's really your specialist area, you know, far more than I do about what's going on in, in that area.

Chris Adams: But yeah.

Adrian Cockcroft: It's a big topic, but I think in 2022, I thought that we would be having more regulations sooner, and that would be pushing more activity.

And then I wanted to basically, by talking about this, at that event, I wanted to get some of the tools, vendors to basically I would, for me to talk to them about how to do this. I ended up doing a little bit of advisory work for a few people, as a result, but not really that substantial. So that's kind of where I was then.

And then over the next year or so, I did some more talks, saying it's basically I just tried to figure out what was available from the different cloud providers. Did a talk about that, and then, wrote a. A-P-R-F-A-Q or a, proposal for a project for DSF saying, well, we should fix this. And it would be really nice if we did actually have a, this is what people would like to see, and then went and tried to see what we could get done.

Chris Adams: Okay, so that's, that, that's useful sort of kind bring, us up to this point here. And like, one thing I've appre appreciated about being on the Real Time Cloud project is that it's very easy, to basically call for transparency bec and there are absolutely reasons why you, why a company might not want to share their stuff, which are kind of considered like, I don't know, wrong reasons I suppose, or kind of like greedy reasons.

So, I used to work at a company called A that stood for avoid mass Extinction engine. And one thing we did was I. we were, we raised something in the region of 20 million US, dollars to find out all the ways you can't sell or carbon API in the early 2010s. And, you know, pivoting like a turntable, it's kind of a bit embarrassing at times.

Right? And one of the things that we, one of the potential routes that people went down was basically, we are gonna do this stuff and we are gonna work with large buyers to basically get people in their supply chain to share. Their emissions information, with the idea being that this would then be able to kind of highlight what they refer to as, supply chain engagement.

So that sounds great. Like we'll lend you some money so you can buy cheaper, you can buy more efficient fridges and do stuff like that. But there was another flip side to this, where when you're working with large enough companies or large enough buyers, one of the things they would basically say is they could use this information to then say, well, who are the people who are the least efficient?

And like, who am I gonna hit with my cost cutting stick first? Basically like who is, and this is one of, and for this reason, I can totally understand why organizations might not want to expose some of their cost structure. But at the same time, there is actually a kind of imperative coming from, well, like you said, the planet and from the science and everything like that.

And like, this is one thing that I feel like this is one of the drive, this is one of the thing that's been a real blocker right now. Because companies are basically saying we can't share this information 'cause we are going to end up revealing in how many times we maybe sell the same server, for example, like the, and these are kind of, you can see why people might, might not want to release that or, disclose that information.

'cause it can be sited, considered commercially sensitive. But there is also the imperative elsewhere. And like I wanted to ask you like. Faced with that, how do we navigate that? Or are there any things that you think we can be pushing for this for? Because I think this disclosure conundrum is a really difficult one to actually,, to get around basically.

And I, figured like you are on the call, you've been on both sides. Maybe you have some perspectives or some viewpoints on this that might be better. Shed some light here rather than it just being this, you are transparent. No, we're not gonna destroy our business kind of thing, because there's gotta be something, there's gotta be a third way or a more, useful way to talk about this.

Adrian Cockcroft: Yeah. And I think, I mean, there are three primary cloud providers that we've been working with or attempting to work with. And they're all different, right? And just Google generally have been the most transparent. they produce data that's easy to find, that's basically in a useful format. And they came out with their, their annual sustainability report recently, and there's a table of data in it, which is pretty much what we've been adopting as this is useful data.

Right? So that's one. but still they don't disclose some things because they don't have the right to disclose it. For example, if you want to know the power usage effectiveness, the PUE, they don't have it for all of their data centers. When you dig into that, you find that some of their regions are hosted in data centers they don't own,

right?

So somewhere in the world there's a big colo facility owned by Equinix or somebody, right? And they are, they needed to drop a small region in that area. So they leased some capacity in another data center. Now, the PUE for that data center is not the they, because they're not the only tenant. It's actually hard to calculate, but also the owner doesn't necessarily want to disclose the PUE, right?

So there's a one, the number isn't really obtainable. You could come up with a number, but they have to, you know, as a third party that they'd have to get to approve it. So that's a valid reason for not supplying a number. It's very annoying because you have p OE for some data centers and not others, and that applies to all the cloud providers.

so that's a valid, yeah, it's annoying, but valid reason for not providing a number. Right. So that's one level. And Google are pretty good at providing all the numbers, and they've been engaged with the project. They've had a few people turn up at the, on the meetings. they've fixed a few things where something wasn't quite right.

there was some missing data or something that didn't make sense and they just went fixed it. And there was also a mapping we needed from there. They're the Google data centers, which support things like Gmail and whatever, Google search to the Google Cloud data centers, which is a subset of it. But that we, they actually went and figured out their mapping for us and gave us a little table so we could look up the PUE for the data center and basically say, okay, this cloud region is in that data center.

They've worked well with it. So that's kind of what I'd like to see from the other cloud providers. It show, it's like, I like to see existence proofs. Well, they did it. Why can't you do that? Right. So that's what I'd expect to see from everybody. Microsoft were involved in setting up the GSF and were very enthusiastic for a while.

Particularly when Asif was there and driving it and, since he's moved on and, is now working directly for the GSF, I think the leadership at Microsoft is off worrying about the next shiny object, which is ai, whatever. Right? There's less su less support for sustainability and, we've found it hard to get, engagement from the Microsoft, Ah,

to get data out of them.

they have a report, they issued their new report for the year and they had total numbers for carbon, but they didn't release their individual regions updates, you know, so they released overall carbon data for 2024, but we haven't got any updated, nothing that I can find anyway on the individual regions, which is what we've been producing as our data set.

Chris Adams: Ah, okay. So basically as the moon and the moonshot has got further away, as they say, it's also got harder to see. Basically we still have this issuer then that this, it's less clear and we have less transparency from that. That's a bit depressed. That's a bit depressing. When early on they were basically very, they were real one.

They were. I was really glad to have them inside that because that they, they shared this stuff before Google shared it, so we actually had, okay, great. We've got two of the big three starting to disclose this stuff. Maybe we might be able to use this to kind of find against concessions from the largest provider to share this.

Because if you are a consumer of cloud, then you have some legal obligations that you still need to kind of, kind of meet, and this is not making it easy. And for the most part, it feels like if you don't have this, then you end up having to reach for a third party, for example, where you, like, you might use something like green pixie, for example, and like, that's totally okay to use something like that, but you happen to go via a third party where you know, you're, that, that's secondary data at best.

Basically it feels like there's something that you should be able to have with your, supplier, for example.

Adrian Cockcroft: Yeah. Just to clarify, I think there's several different types of, Sustainability data or sustainability related data that you get from a cloud provider. One of them is, well, I'm a customer and I have my account and I pay so much money to it, and how much carbon is associated with the, the things I've used, right?

And that is they all provide something along those lines to greater or lesser degree.

Chris Adams: Mm.

Adrian Cockcroft: but you can get, an estimate for the carbon footprint of an account, right? typically delayed by several months, two to three months, and it's a fairly, and it's pretty high level. So, and it gets, there's more detail available on, Google and Microsoft, and there's fairly high level data from AWS, but that's, one source.

The other source that we're interested in is, let's say I. I'm trying to decide where should I put a workload? And it could be I have flexibility, I can put it pretty much anywhere in the world or I can choose between different cloud providers in a particular country. what's the, and I want to know what the carbon footprint of that would be.

Right? So to do that, you need to be able to compare regions, and that's the data set that we've produced and standardized so that it lists every cloud region for the three main cloud providers. And for each of them we've got whatever information we can get about that region. And back in 2022, we have a fairly complete data set and 2023, it's missing.

Microsoft provide less data than in 2022. And in 2024 data, currently we have Google data, we have Microsoft have released their report, but haven't given us any new data. And AWS are probably releasing their data in the next, Few days, last year, it was on July the ninth, and I just checked this morning and it hasn't been released yet, so it's probably coming next week.

It's sometime in July. Right. So, we're hoping to see, well, we'll see what information we get from AWS and I'll, I, every year I write a blog post where I, they said, okay, the three reports are out. This is what happened. This is the trend year on year, and I'm working on an update of that blog post.

So probably by the time this, this podcast airs, I'm hoping that pod, that blog post will

Chris Adams: out there.

Adrian Cockcroft: I should have got it. I, you know, I've written as much as I can right now, but I'm waiting for the AWS ones, so. So we've sort of discussed Google have been pretty good, I guess, corporate citizens, disclosing whatever they can and engaging with the project.

Microsoft's sort of early enthusiasm. In their latest report, they actually mentioned the GSF and they mentioned they founded it and they mentioned that they support the real time cloud project, but they're not actually providing us any data and we're still trying to find the right people at Microsoft to escalate this to, to figure out, well, so gimme the data.

Right? and then AWS then they have, some different issues going on. they, the way that they run their systems, one of the things they found is that if they disclose something about how they work, people will start leveraging it. Right. You get this sort of gamifying thing. If there's an interface or, a disclosed piece of information, people will, optimize around it and start building on it.

You see, there's a lot in eBay. One of the reasons eBay's interface hasn't changed much over the years is that there are sellers that optimize around some weird feature of eBay and build a business around it. And every time eBay plans to change that, they're like, some sellers gonna lose their business, right?

So, if you over expose the details of how you work, there's sort of an arbitrage opportunity where somebody will build something on that and if you change it data, they get upset. So that's a one of the reasons that AWS doesn't like saying how it works,

right? Because it would cause people to optimize,

Chris Adams: yeah. Private

Adrian Cockcroft: optimize for the wrong things.

And, one example is that there's an Archive capability, tape Archive capability. That AWS has, and you can, and if you're thinking about I have lots of data sitting on desk, I should move it to tape. 'cause that is a much lower carbon footprint. And it is, except if you're in a tiny region that AWS has just set up, they haven't actually really got tapes there, the same services there, they're actually just storing it to disc until they have enough volume there, for them to put in a tape unit and transfer that to tape.

Like they want the same interface, but the implementation is different. Now, if they exposed which regions the, this is actually going to dis, it would say, well, this is a high carbon region, so I shouldn't store my data in there. Which means it would not get enough volume to actually install the tape.

Right? So you get the sort of negative feedback loop that's actually counterproductive. Right. So, so, so there's this, there's that sort of a, an example of. It's one of the reasons that they don't want to tell you how much carbon every different service is because it could cause you to optimize for things that are gonna cause you to do the opposite of what's the right thing to do Ultimately.

Chris Adams: okay. So that's one of the argument we see used for not disclosing how an organ, like. Per, like, per like service level and per region level things. 'cause one thing that when you use, say Amazon's carbon calculator, you'll get a display which broadly incentivizes to do, incentivizes you, you use to change basically nothing.

Right? like that's one thing we actually see. But, and that's different to say Google and Microsoft. We do provide service level stuff and region level stuff. So one of the reasons they're trying to hide some of that information is basically it's making it harder for us to kind of basically provide that service, for example, or there's all these second order effects that they're trying to basically avoid.

That's one of the arguments people are using,

Adrian Cockcroft: That's the argument that they have, and it's something that's pervasive. It's not just related to carbon. This is something that they've seen across lots of services is that people will, people will depend on an implementation. And they changed the implementation frequently. Like we're on, I dunno what the eighth or the ninth version of S3 total rewrite from scratch.

I dunno. When I was there, I think they were up to the seventh or eighth version and I knew somebody that was working on the team that was building the next version. Right. And this is tens of exabytes of storage that is migrated to a completely new underlying architecture every few years. If you depend upon the way it used to work, then you end up being suboptimal.

So there's some truth in that, however, and this is the example we were pointing at when I was at AWS, is that Microsoft and Google are releasing this data and we haven't, there's no evidence of bad things

Chris Adams: Yeah. The sky hasn't fallen when they

Adrian Cockcroft: Yeah. So, so I think that it, would be just fine too. And they are gradually increasing the resolution.

So what they had when. When they first released the, the account level information when I was there, and we'd managed to get this thing out in 2022, I guess 20 21, 20 22 was the, you had regions being continents, right? You just said Europe, Asia, and Americas.

And you had S3, E, c two, and other,

Chris Adams: yeah.

Adrian Cockcroft: and you had it to the nearest a hundred tons or something, or nearest a hundred kilograms.

Yeah, a hundred 10th of a ton. So most, so a bunch of people in Europe just got zero for everything and went, well, this is stupid. But actually, yeah, because of the way they, the, model works, they were generate, generating lots of energy to offset the carbon. It probably is zero for at least scope two.

scope, scope two, for the market based model.

Chris Adams: where you, count the, green energy you've used to kind of offset the, actual kind of, yeah. Figure. Alright.

Adrian Cockcroft: Yeah. So what they've done in the last couple of years, they finally got a team working on it. There's a manager called Alexis Bateman that I used to work with in the sustainability team that's now managing this, and she's cranking stuff out and they finally started releasing stuff. So the very latest release from AWS now has per region down to per region.

It has location based, just got added to the market based. So we actually have that finally.

Chris Adams: okay. Yeah.

Adrian Cockcroft: So this happened a few weeks ago. and the, and they've added, I think they have cloud. CloudFront because it's a global, CDN, it doesn't really live in a region. So they've separated CloudFront out and they also changed this model, as I mentioned earlier, so that the carbon model now includes supporting services that are required for you to use the thing.

So your, Lambda functions, even if they're not running, you've still got a carbon footprint because you need to have the lambda control planes there, ready to run you. So you pay for a share of that. And then the question is, how do you calculate these shares? And it's probably, you know, dollar based or something like that.

Some kind of usage based thing,

Chris Adams: Okay. Alright. So that's, yeah, I think I've, I read the, I hadn't realized about the location based, information being out there as well.

Actually,

Adrian Cockcroft: the location and the model with a new thing and they've now got this sort of, every few months they're getting a new thing out. They have def, they've clearly said they're going to do scope three. I know they're trying to do scope three where they real scope three thing rather than a financial allocation scope three.

So we could talk about that if you want, how much you wanna get into the weeds, of this stuff. But anyway,

So what we ended up with in the real Time cloud project was we figured out it wasn't really possible to get real time energy statistics out of cloud providers because the numbers just didn't exist.

It turns out the only place you can get real time numbers is on things that are not virtualized. And the thing that people don't generally virtualize is the GPUs. Yeah. So if you're using an Nvidia GPU, you can get a number out of it, which is the energy consumption of that GPU. So if anyone working on AI based workloads, you can get the dominant energy usage cap calculation is available to you, sources available.

But the CPUs, because the way virtualization works, you can't provide the information unless you're using, what they call a bare metal instance in the cloud, which you get access to the whole thing. So that's we gave up a bit on having like real time energy data and also the CNCF came up with a project called Kepler, which does good estimates and it does a workload analysis for people running on Kubernetes.

So it just, we just did a big, like point over at that. Just use, Kepler. If you want workload level energy estimates, use Kepler. and then. If we want to, and we focused instead on trying to gather and normalize the data, the metadata available on a region so that you could make region level decisions about where you want to deploy things and understand why certain regions were probably more efficient than others in terms of PUE and water usage and, carbon and the carbon free energy percentage that the carbon that the cloud provider had, meaning how much local generation did they have in that region.

So that was the table of data that we've produced and standardized, and we've put a 1.0 standard on it. And the current activity there is to rewrite the doc to be, basically, standards compliant so that we can create an ISO standard or propose an ISO standard around it. And the other thing we're doing is talking to the finops Foundation who come at this from the point of view of.

standardizing the way billing is done across cloud providers and they have all the cloud providers as members and all working on billing and they're trying to extend that billing to include the carbon aspects

of what's produced. working. so, we've done an interview with someone from Focus already who is basically talking about, they are almost. You, like you mentioned before, the idea that, okay, Microsoft and Google have shared this kind of per service level information and the sky hasn't fallen.

Chris Adams: They've created something a bit like that to kind of almost list these diff different kind of services. What, if I understand it, the GSF, you know, the, real time cloud thing might be like a carbon extension for some of that, because that doesn't necessarily, the, right now the focus stuff doesn't have that much detail about what carbon is or what, the kind of subtleties might be related to the kind of other, the kind of non-cash non, yeah, the, non-cash things you might wanna associate with, the way you, purchase cloud for example.

Adrian Cockcroft: Yeah, so focus is the name of the standard they've produced. Really all the cloud providers have signed up to it. If you go to an AWS's billing page, it talks about focus and has a focus, a conformant, schema. So the idea was all the cloud providers would have the same schema for their billing. Great obvious thing to do, but all the cloud providers have joined up to do that, which is fine.

Now Focus does, has some proposals for sustainability data, but they are just proposals for maybe the next version. They had a working group that looked at it and the problem they run into. One of the things is we've deeply looked into that in our group. We know why you can't do that. So what you'd really like is a billing record that says you just used, you know, 10 hours of this instance type.

And this is the carbon footprint of it. And the problem is you, that number cannot be calculated. and that's what you'd like to have. And intuitively you'd like to just no matter how much carbon it is, the problem is the carbon is not known at that time. You can generate the bill 'cause you know, you've used 10 hours of the thing, but you can't know the energy consumption and the carbon number, the carbon intensity, those two numbers are not known for a while.

So you typically get the data a month or two later. Whereas like, yeah, but you have to go back to your billing data. So you could put a guess in there. And things like the cloud carbon footprint tool and other tools that are out there will just generate guesses for you. but they are guesses. And then when you go and get the real data from your car cloud provider, the numbers will definitely be different, sometimes radically different.

so the question is, do you want to have an early guess or do you want to have a real number and what are you doing with that number? And if what you're doing is rolling it up into an audit report for your CFO to go and buy some carbon credits at the end of the year, that's what the monthly, reports are for.

Right? If you're a developer trying to tune a workload that is useless information to you, you need real, that's what the real Time cloud group was really trying to do is like if you're a developer trying to make a decision about what you should be doing. You know, calculating an SCI number or, understanding which cloud provider and which region has what impact.

That's the information you need to make a decision in real time about something. So the real time aspect is not about like in my milliseconds, I need to know the carbon or whatever. It's like I need to know now. I need to make a decision now.

Chris Adams: to make a forward looking decision

Adrian Cockcroft: Yeah. It's like I need to make a decision now, so what information do I have now?

Which is why we take the historical, metadata they have for the regions and we project it into the current year with, so just trending and filling in the gaps to say, this is our best guess for where you'd be if you needed to make a decision this year, on it. And we've got some little code that automatically generates the Nafus, estimate.

Chris Adams: so that's, at least useful. So people have an idea about what you might be using these two different kinds of data for. I guess maybe the thing, if we could just unpack one last thing before we move on to one of the questions is that one of the reasons you have this delay is basically because, is it, 'cause companies aren't, don't get the billing data themselves and they need to go then go out and buy credits.

Like this is for the market based calculations. So this, what you've said here is basically about carbon based on a market based figure. But if we had something like, maybe if we were to separate that out and looking, look at something like location based figures for electricity, which is like representing the kind of what's happening physically on the grid.

You plausibly could look at some of this stuff. Is that the, I mean, is that the way you see it? Really? Because I feel that we are now at this point where there's a figure for the grid, but that's not necessarily gonna be the, only figure you look at these days, for example, because as, because it's, we increasingly seeing people having different kinds of generation in the facility.

If you've got batteries, you might be, you might have charged batteries up when the energy's green, for example, or clean and then using it at a certain time. that's there's another layer that we need to, that you might need to take into account. Right.

Adrian Cockcroft: Yeah, so there's a couple of different reasons why the data is delayed. you know, you're in Germany, I'm sure with Germanic efficiency, you know exactly when you are going to get the information from your energy provider,

Chris Adams: They fax it to us. Yep. Yeah.

Adrian Cockcroft: and it will be nice and precise and there'll be high quality on it. now if you're operating a region in a developing nation.

not so much, right? There's bits of paper moving around. Probably. There's, random things happening. You dunno quite know when. So if you are trying to produce a service that is a global summary across all regions, you have to, you are limited to the slowest region that you operate in, right? you take this sort of distribution of how quickly you find out about the carbon intensity and the power usage of what's going on in your country and in the energy supply for your region.

And, you know, it's, whoever is slowest will de determine it, right? And AWS operates in regions in India, and Indonesia and places like that where, I don't know, maybe, there are efficient, maybe they aren't. But there, they, there are more global regions in more different countries on AWS. than in particularly in Asia than Azure and Google have, but fundamentally, it's gonna take you a few months to gather your billing and carbon data accurate to the point where it's not gonna change.

So then on top of that, you can then say, I'm gonna buy some credits to offset that. And there's two different ways of doing credits. You can buy green energy, procure your energy from a supplier that says, okay, I'm this energy that we already generated, you can buy the credits for it later. And so you can basically pre post allocate it, and you can do that within the rules for up to a year afterwards.

So at the end of the year, it comes to December, end of December, okay, how much energy we did we use, how much wasn't offset. I can buy energy credits from my energy suppliers to offset that. And the first thing you do is try and do it in region so that the energy is happening in the same grid. That you, your consumption was, and then you get to Singapore and go, okay, we all give up on Singapore.

There isn't enough local energy that's green, so we're going to buy energy somewhere, anywhere we can, green energy somewhere else and do a global offset on it. Google's been doing that since 2017, I think, or whenever they, said they were a hundred percent green back in the day, long time ago.

AWS since 2023, a hundred percent offset. but what, that's the mechanism they use and it's documented in their disclosure that they do it on a region by region basis and then they use global offsetting just for the, to mop up whatever's left over at the end. Right. So that's, and, then. A s does less of this, but is starting to do more, which is, carbon offsetting where you go and, you know, pay for a forest to not be cut down or you pay for built, grow some trees or you sequester some carbon.

And that is a little bit on the end that people are investing in to try and develop those markets. but most of it is, buying green energy. Like for the house here, I have an option to just subscribe to a different cloud, a different energy provider. It's called Central Coast Community Energy. And, Yeah, I pay them at slightly higher, you know, an extra cent or so per kilowatt hour. And I have a hundred percent green energy. And by market method, my, I'm completely green here, right? So that's fine. But it's the same thing going on. So, because what I'm paying for is the green energy. I'm not paying for carbon.

I'm probably is emitting carbon at night, certainly, but I'm generating more during the day 'cause I've got some solar panels here. Right. So that it, it's that mechanism that's being developed basically.

Chris Adams: Okay. Thank you for that. Alright, Adrian, I realize we we're coming up to time, so I, did have a bunch of questions about, what's making it harder to track, this stuff, like, because we are, we're now moving to work to the world of grid responsive data centers, for example, like various data.

We've been doing some stuff like that, and we're seeing cases of like, I don't know, in Memphis, the Colossus data center running primarily on gas turbines, right now, which is playing, which, massively complicates some of this. But we did actually say that there was some stuff going on in the house, and I do wanna kind of come back to that if we could, because that feels like it's, we won't have time to explore to do those, the, those subjects.

Justice basically. So we were talking, at the beginning of this podcast that in addition to doing this stuff here with the Green Software Foundation, you've been exploring and playing around with some of the, tools and some of the technology and like finding out if there's a, there, for example, and, When I looked up this project, when before doing some research for this podcast, I heard, I, I read about this thing called the House Consciousness System. So maybe you could talk a little bit about that because, you've been working as a technologist for, you know, at least 40 years now, and I see you've messing around with things like generative AI and AI for this and doing things that I am not expecting people to do.

So maybe you could talk a little bit about this, that HCS or the, or whatever, this project is, because I, found it quite interesting that you were to see your take on this, basically.

Adrian Cockcroft: Yeah. So the history of this going back a while is that a while ago, I mean, every now and again, I, do some programming and I wanted to do some programming a year or two ago in the R language, which is a statistical language, which I use occasionally. I keep forgetting the syntax of. So I thought, well, maybe I can ask the AI to remind me of the syntax.

And the AI just started writing code that worked. So I went, oh, this is cool. I can just tell it what I want and it'll write the code. And this is very early days when people. Most people weren't doing that kind of thing, so that was fine. and then, more recently I wanted to write some code in Python.

I'm not ever, never really wrote in code in Python. I can sort of read it, but I can't write it. So I started telling it what I wanted it to do and it wrote the code for me and it could get it working pretty well and it worked. So I have code, I was using it to generate code there. and this is mostly just 'cause I'm not, I don't have patience and time to be like a full on developer, but it, these are the things that it's good at.

So one of the things that I tend to focus on, if as a new tool around, I figure out what can it do and what can't it do. Think of cloud, the early days of cloud. We built Netflix on an extremely rudimentary set of cloud services. And it was like just about possible to make it work given the services we had.

And most people today would look at that and say, we wouldn't even start trying to build anything on that, right? But we made it work, and we made it work reliably. And, that became like a template and it caused other people to try and figure out how to do it. Now there's a lot more capability there now, so we're sort of in that early stages day, thing where a bunch of people say, well, this'll never work.

there are people figuring out how to make it work. What happened? where shall I go next? Let's talk about the idea. So years ago, I mean, I have lots of iot devices around my house. I like buying random, automated, and then none of them talk to, or some of them talk to each other, but I have too much random stuff and I have a, like an iPad with lots of icons on it, and I have to know which one does what.

Right? and it's annoying. And if I'm not home and my wife's trying to do something, she can't figure out which one. And she knows some of them, but like, she doesn't, know how, this stuff works. other visitors to the house don't know how to do things. And that's just, and a lot of people are in that environment.

And I was thinking about this a few years ago when I was at Amazon and I was talking to the Alexa team because they have, house automation kind of stuff. So why don't you build in something that is just like a more general thing that knows what's going on in the house. That would, and it's sort of like a, central consciousness because the re thing about consciousness, it's an observability system.

I regard consciousness as human observability. And part of the definition of consciousness for me is that you have to understand what unconscious means, right? If your definition of consciousness doesn't include unconscious, then you're not, you haven't picked the right thing. So it's the thing that goes away when you're asleep,

right?

'cause you're unconscious when you're asleep. So anything that goes away when you're asleep is consciousness to me. Right. And there are, and this isn't the standard definition. People have big arguments about it, but that's a, good working definition for me. 'cause what it means, it's the thing you can talk to, to discover its state, right.

So if somebody is conscious, you can ask them questions and find out what's going on with them. So in that sense, what I want is want my house to have a memory of all the things that have happened to it. I want it to look at the weather and remind me there's a storm coming and have you dealt, you know, ti it up outside so things don't blow away and all the stuff, right?

but right now the iot devices live in the moment. Let see, your temperature is 73 degrees and they sort of have a schedule for changing stuff, but they don't really have a memory and they aren't talking to each other. And so I had this idea that, Hey, why doesn't Alexa team build something like this?

And they, I sent, I found somebody in that team and they never built it and weren't interested in it, right? So I had this idea of. Kicking around. And then a few weeks ago I saw that Rueven Cohen, R-U-E-V-E-N, Cohen Coen, C-O-H-E-N, on LinkedIn, is just posting and posting about his agent's swarm work.

And he's just, like, he's building amazing stuff and said this, does this really work. So I wanted to play around with it and I decided I needed a new idea that I needed to try and build. That was a fairly aggressive thing. So I wrote up a rough idea of what this house consciousness would be.

And then I got together with Reuven. He showed me how to start an agent swarm, to just get with the CLO and Philanthropics Claude Code service to just go build it. And it wrote about 150,000 lines of code in a day or so.

Chris Adams: So these, so when you talk about a swarm of age, a swarm of agents, it's basically kind of like a model in a loop that's writing some code and there's multi, there's lots of them working together. That's what a swarm of agents in this case. Yeah.

Adrian Cockcroft: Yeah, but they aren't all writing code. So what you've got is sort of a, the latest version it, they call, he calls it a hive. So they're sort of a queen bee that who is just managing the hive. And it's basically like there's an AI acting as a light, as a development manager,

a dev manager. And the dev manager picks these specializations they want, so they start a selection of agents and they've got a QA agent, a DevOps agent for a deployment, a spec reading agent, a researching agent.

And they basically specialize. And what happens is if you're doing AI coding. The context. If you use one agent to do, if you just use one AI and you tell it all the things you want to do, it sort of gets confused. 'cause you've asked it to do lots of things at once. What you do here is by giving them each one track, mind specializations and a, and an ability to communicate, you get dramatically better results out of it.

So that's the aha moment if you like. But what it means is that to manage this swarm of agents, you need basically product manager and line manager skills, not developer

skills, right? You need a bit of developer skill to read the code and see if it works. But I don't, I'm currently writing, I switched from writing in Python.

The first thing I tried building, which was more just like a, can it build anything at all? And it built a thing and it ran, but didn't really work. 'cause I didn't, really specify what I wanted well enough. I'm now building an i an iPhone app in Swift, which I absolutely cannot write a single line of Swift.

I have no idea. And it's writing the code. I'm telling it to do code reviews and, Run and tests and things. So it's actually coding and testing and building itself and build a UI design and a plan. And so I'm doing that anyway. So I'm basically, I've now got a little obsessed by building myself this thing.

And you basically need a Max plan to do this, which is sort of about a hundred dollars a month AI plan. And once I finish building this, I'll sort of wind that back down to the usual $20 a month kind of level. and yeah, I mean, from my point of view, you can use AI to do a bunch of bad things, you know, generate fake news and stuff and adverts and things, but I'm actually using it to develop something that I always wanted to have, that isn't, there's no real business model for this thing other than I want it to exist.

I don't need a business model. I can spend a hun few hundred dollars, which is sort of go out to dinner. You spend a hundred dollars, right? That on building something, getting it to build something, which I'm sharing on GitHub. you can go and have a look at the repo, the, original Python repo is, it's sort of, it's there, it runs, but it's not, doesn't really do what I think.

It doesn't really work, right? 'cause I hadn't thought it through. I'm now doing front back, front end backwards. So I'm doing as much functionality as I can in this iOS app, and then I'll build the service to go behind it. I'll revisit that when I get to it. So that's kind of what I'm doing. I'm just happily use using this new tool to do something that will make me happy and potentially be useful for other people if they feel like it, but I don't care whether anyone else uses it.

So that's sort of my approach to figuring out new tools and finding where they work and where they don't work.

Chris Adams: Okay, so let me, if I can, I just wanna paraphrase some of that. So you've, so when people, like, I am not using a all that, that, that much at the moment, but I am dabbling and I'm using, I've been messing around with Claude and stuff like that to ask questions or, okay. I'm in Europe, so we use Misra 'cause it's the, French equivalent for example.

But one thing you said that was significant was that a, rather than me using one thing in like serial, it's happening concurrently. So there's lots of different things all burning

Adrian Cockcroft: They typically run like five to eight of these agents in parallel, and they're coordinating and communicating. They make to-do lists and they, do different specializations and they, it's basically like managing a team of developers.

Chris Adams: Ah, and that might be why you have people talking about, say, like, what the hell, you know, vibe, coders, dunno what they're doing here. 'cause in many cases. it's a new set of skills. It's not necessarily just, can you read, I mean, yeah. It helps to be able to read Python in the same way that if you are reading the output from a, chatbot, you want to, you know, you'll probably tweak it to make it sound like a human rather than, a, an ai model.

But there's also a bunch of other skills that you need to do, like spec writing and all these other things that you might, that typically might live in a product manager rather than a developer, for example, or someone who in different roles.

Adrian Cockcroft: it's much more product management than anything else. You have to have a clear idea of what you want. And figure out what the user flows through it are, and you know what functionality you want, how you want it arranged. And then it will build whatever you tell it to build and it will add on things you didn't ask for.

So you do it. So the other thing is to do it very incrementally and check every time to see what it did build. And you ask for this and it does like two or three times as much, and you go like, I want to keep these things. That was a good idea. No, I don't want that. And if you're working with a team of engineers, you say, I want them to, build a thing, they'll come back with extra stuff that they thought they, you might want.

Right? So there's actually, it's normal. I mean, this is how you manage a team of engineers to go build a new thing, right? So in that sense, it feels anybody that's managed a development team, this actually feels very familiar. If you're a developer on one of those teams, it doesn't feel very familiar. So I think that there's this sort of weird.

Thing where we sort of brought it up a level into management, but you still kind of need specialist experts. Like I'm stuck in a whole bunch of Apple stuff to do with iOS. That is nothing to do with the ai, it's just stuff that Apple makes difficult

Chris Adams: That would explain what you've been posting about what? How? Okay. That, that, that. Adds a context. 'cause I've seen you posting like, how do I know what to ask from Apple now? And I was like, why is he asking that? But okay. This is putting two and two together for me now. S Adrian. Thank you. So there was one last thing I was just gonna ask before we kind of wrap up.

Right. you mentioned that you're doing this all locally on your own computer? Or like, is it, are you running in like an, in an environment in the cloud, like a code space with GitHub? Or maybe, yeah. Where is that happening?

Adrian Cockcroft: Yeah. So when you have an agent swarm, it can do anything. You basically let it free run, so you need to put it in a box so that it doesn't delete your computer or do random things or whatever. Right. So. You go to GitHub and they have a free, it's free for up to some amount, which is plenty for me.

So far, having been doing this for a few weeks, you create a code space, which is basically an instance I guess running on Azure, which is like a little container. It shuts down if you don't use it for a few minutes or a few, you know, when it's idle. But basically it opens it and the only thing it can really do is run against the repo.

You opened it on, so the AI can sit there and it can do anything it wants in a copy of that repo and then it can push to the back, to the repo. I tell it usually when it's, when you finish this work, just push it back to the main repo and GitHub because when I'm working on iOS, I have to pull it back out into X code on my machine to build it.

So it's a safe box. It's a safe box. And I wouldn't run an agent swarm today on my own machine. Right. You could, but you're, it's sort of dangerous if it gets a bit carried

Chris Adams: That's what I was wondering about. That did seem like the idea of having one rogue machine work on my machine is on my own personal laptop feels a bit weird because I don't dunno where to post it in the, but if I've got a whole bunch of them, that times terrifying. Adrian.

Adrian Cockcroft: the option on Claude is minus, minus dangerous dangerously enable all permissions or something like that.

Chris Adams: Okay. All right.

Adrian Cockcroft: so that is the, mode we're running Claude in, and on, I think on our Google Gemini it's called YOLO mode.

Chris Adams: Yeah.

Adrian Cockcroft: you just, right, right. So, but when you're running it in that mode and a swarm other, and you can't just sit there and say, yes, it's okay to do this.

Yes, it's okay to delete that file, whatever, because it's tidying up, it's moving things around. It's writing code fragments and running them and deleting them again and stuff like that. It's doing all these things, but you don't want it to just do like an R minus rf.

Chris Adams: to just like hose your

Adrian Cockcroft: I once had somebody call, what does con stat CWD mean?

Oh, it means you're in a directory that doesn't exist. Now I'm at my home directory. What did you just type? Well, I was cleaning up some dot files, so I did RR minus R star,

Chris Adams: Oh dear.

Adrian Cockcroft: dot. He recursively deleted his parent directory.

Chris Adams: Oh geez. He just like wiped his entire machine.

Adrian Cockcroft: Yeah. The end of the day tidying up before he went home and said, well, you just lost your day's work.

Luckily we have a backup. I was the guy that ran the backups. This is before I, when, I was,

Chris Adams: Okay. I'm glad you mentioned this, final story here, because it sounds like you're using a code space almost. Whereas typically you might use it for convenience. You're almost using it for safety. Like I want to minimize the blast radius of these agents running am mock inside my system. And also I guess like conveniently, because it's, I mean, surely this should be something what you can work out the environmental footprint of this because if it's a billable tool and like if GitHub knows to bill me for all those minutes, they should be able to tell me the carbon footprint of this as well, right?

Like, I mean, yeah, there's could be some complexity with like using Azure, but like I should have something indicative at

Adrian Cockcroft: There's a little, console that tells you how many hours you've used, and if you use more than a certain amount, they start billing you for it. But you get some number of hours per month of CPU hours per month, which is enough for me. So far I haven't hit that limit and I've been playing around for a few weeks fairly intensively.

so that's, yeah, but the Claude itself, so no. So one thing last, one last thing. So we've got the three main cloud providers. We know roughly what they're doing. we now have Oracle that I'm trying to find somebody to talk to, to tell me how, what their carbon footprint is. Core weave is probably up and coming as a new big one.

And then you have all the stuff that anthropic or open AI or whatever they're all running, where are they running? So we've now got some very large sources of carbon. That we need to get accounting data for. And as far as I can tell, they are not publishing that data currently. So that's currently, I'd say that's the next phase.

It's like how do we, can measure an individual GPU pretty easily, but the, GPU services we're using are not being allocated. We're not getting data from. So that's probably, where I should wrap up. That's where we are now, we need to find out how much I need to know how much carbon I'm actually generating by telling Claude to build this thing for my house.

I dunno. No clue.

Chris Adams: I think that feels like a very useful rallying cry for anyone who does want to see if it's possible to instrument any of these tools, and they're listening to this podcast. Adrian, thank you so much for giving us the time to chat with us. I really enjoyed noting out with you and going really, deep into the weeds.

If people are curious about any of these projects that you've mentioned, we're gonna share the show notes, but where would you direct people's attention? Like is there a URL or a website that you'd point people to?

Adrian Cockcroft: Well, there's the Green Software Foundation's Realtime Cloud, website, GitHub site, and my own Adrian.co. You don't have to spell Cockcroft, you just have to get the first two letters, and on GitHub, and that's where you can follow along my random rants and things like that. and I have a Medium account as well @adrianco.

And where I have my blog, if you want to chat to a, an AI copy of me. There's a service called Soopra. We'll put a link to that. Soopra.ai, and there's an, I think it's Soopra/cockcroft is my, I uploaded all my blog posts into a persona so you can go ask it, and it comes up with sometimes reasonable answers to questions about microservices and Netflix and things like that.

Chris Adams: I wanna ask you all these questions about do you ever use that to fob people off or email you? But I think I'll have to pass that for another day.

Adrian Cockcroft: mostly when somebody sends me a list of questions for a podcast, like if I have time, I feed them into it. I didn't have time

Chris Adams: Okay. Alright. Alright, Adrian, thank you so much for this and I hope you have a lovely weekend. All right, take care mate.

Adrian Cockcroft: All right. Cheers. Thank you.

Chris Adams: Hey everyone, thanks for listening. Just a reminder to follow Environment Variables on Apple Podcasts, Spotify, or wherever you get your podcasts. And please do leave a rating and review if you like what we're doing. It helps other people discover the show, and of course, we'd love to have more listeners.

To find out more about the Green Software Foundation, please visit greensoftware.foundation. That's greensoftware.foundation in any browser. Thanks again, and see you in the next episode.