The Inevitable Shift to Distributed Systems

In this episode, Astrid Atkinson, co-founder of Camus Energy, talks about her company’s “grid orchestration” work of helping utilities see, track, and coordinate the distributed energy resources in their territories.

(PDF transcript)

(Active transcript)

Text transcript:

David Roberts

One of my favorite things I ever wrote was a 2018 piece for Vox on grid architecture — the basic structure of the electricity transmission and distribution networks. It was about how a top-down system, with one-way power delivery from big power plants to passive consumers, might evolve into a bottom-up system, driven by local distributed energy resources.

Thanks to all-star illustrator Javier Zarracina, it even has awesome animated illustrations.

One person who read that piece was Astrid Atkinson, who at the time was a senior software engineer at Google. She had managed a team that shifted Google search from a top-down system to a massively distributed system, back before the term “the cloud” existed and there was no template available. She and her team had to develop the principles and best practices of getting reliable performance out of millions of unreliable, loosely coordinated machines. By doing so, they radically expanded the scale and speed of what search could do.

She thought, wouldn’t it be cool if the power grid could make the same shift? Unlike some people, though, she didn’t just blog about it — in 2019, she left Google to co-found and run Camus Energy, a software company that helps utilities see, track, and coordinate the distributed energy resources in their territories. The company calls what it does “grid orchestration.”

Atkinson has been a thought leader in pushing for a new grid architecture. (See Camus’ white-paper series on “the rise of local grid management.”) So I was super-excited to geek out with her on this stuff. We talked about the conceptual shift from centralized to distributed and the drivers making that shift inevitable, plus getting more out of the grid we’ve already built through coordination and efficiency, and how the utility sector can evolve to better manage local resources. I really loved this one.

Okay, then. Astrid Atkinson. Welcome to Volts. Thank you so much for coming.

Astrid Atkinson

Thank you so much. I'm really excited to be here.

David Roberts

I'm so excited for this. Astrid, I have to tell you just by way of preface that I had a weirdly difficult time preparing for today's pod because I'm just so excited by this whole area, and I'm so jazzed. I have so many things to ask you about, so many things I want to say about all this stuff, and I'm kind of overwhelmed and fried my circuits. But let's start here: Let me describe for listeners what you did at Google and tell me if this is an accurate description. So you were part of a team, I think, leading a team that was shifting the way Google did things away from a model where computing was done on a relatively limited set of high-quality, extremely reliable data centers, tightly centrally controlled, to a model where computing is done not on a small set, but on thousands, millions of distributed computers living all over the place, any one of which might be unreliably connected or off periodically or weak or otherwise glitchy.

So basically, moving from a model of tightly coordinated, central control, limited number of entities, to loosely coordinated millions of entities, somehow getting aggregate reliability out of massively distributed, individually unreliable machines. Is that more or less accurate?

Astrid Atkinson

Yeah, that's about right. So my role was in the site reliability engineering team at Google, which is a function that nobody's ever heard of outside of the kind of tech industry. But you can think of reliability engineering as being basically Google's systems engineering function. It's the entity that's kind of responsible for pulling all of the pieces together between sort of software and operations and networking and hardware and everything, and making sure that you can get them to kind of work as a reliable system overall. And I was part of the original team that kind of — it wasn't a function that existed in the industry before Google made that transition.

I was part of that original team at Google and then led a lot of Google's work around scaling out that model.

David Roberts

And so now the idea, more or less is to oversee or encourage a parallel evolution of the electricity grid, basically, from a limited number of tightly centrally controlled entities to a loosely coordinated, massively distributed, huge number of smaller entities, basically.

Astrid Atkinson

Yeah, I mean, that's definitely the hope. And that's partly derived from the utility industry and the grid space's sense of the changes that are needed and also partly derived from my sense that there are a fair number of parallels between some of the approaches that we took and kind of had to make up on the spot to support massive growth and really significant changes in the way that we managed systems for that work at Google. And there are parallels with the changes that we need to go through on the grid side. So it's less like "there's this one piece of technology that we built at Google that will totally solve the problems" and more like "we had to develop a set of approaches and a set of kind of integrative and system level perspectives to figure out how to make that change happen."

And I think a lot of those can be helpful in the grid space.

David Roberts

Yeah, I mean, I think one of the most intriguing things about this is in doing that work, you extracted a set of principles for how to design systems like this such that they are reliable, et cetera. And it's those principles, I think it's like conceptually those principles apply to the grid. Obviously, the individual technologies might be different, circumstances are different, but the principles of how to make it work, I think are a weirdly neat fit. The reason I think we should maybe pull out, the reason that this is not something that the utility really can choose to do or not to do.

Utilities kind of have to do it for a couple of reasons. One is all these distributed energy resources, Volts listeners are familiar with these distributed — with these solar panels and hot water heaters that can store energy and batteries, et cetera, et cetera. All these sort of behind the meter distribution side, distributed resources are coming online. Whether the utilities like it or not, they are swarming —

Astrid Atkinson

It's happening.

David Roberts

online, it's happening. And right now utilities are just like kind of hoping it works out, we'll get into that. That's one reason. But the other reason is we're expecting, like you could say of Google, like it couldn't have scaled to the size it got, it couldn't do the amount of computing it's doing without going through this transition.

You just can't at a certain point with a centrally controlled system where you're tightly controlling a limited set of entities, you just can't get big beyond a certain level. You run into sort of computational limits of your computational resources for a central controller. And we're expecting a lot more out of the electricity system in coming years, as Volts listeners are also very familiar with. We're going to two or three x the demand that it has to satisfy in a much more complex way. So I think it could be argued, and you have argued, and I think it's pretty self-evident, that the electricity system cannot achieve the scale we want out of it without going through this evolution.

There's just no way for the way it's currently run to get as big as we want it to get.

Astrid Atkinson

Yeah, and if you want some simple examples of why that is, utilities have spent a lot of money and a lot of time in the last 10 to 15 years installing smart meters. Right? They were supposed to give us the kind of universal visibility into customer activity that I think we can all intuitively think that we would need to manage a rapidly changing grid with a lot more complexity. But most utilities don't really have particularly high-scaled data infrastructure. And so the idea of actually being able to do something useful with all of that meter data, SCADA data, kind of everything at scale on an ongoing basis in real time, using that as a foundation for analysis and visibility and those kinds of things.

It's really hard for them because usually the software systems that gather and process that data, they're on premise within the utility's own data center. They're typically not really using any kind of modern scalability approaches beyond downsampling the data, which means losing some of it. And they're usually running on a single machine. So it gets really difficult to incorporate very large amounts of data when you can only use one computer.

David Roberts

Yeah, if you talk to them, they're like, "Oh my God, we've got all this data. It's overwhelming. The reason we're not doing more with it is like, we can't do more with it, we're overwhelmed." And then you look at the scale of data involved, and to a Google it's a tiny amount of data. The scales are completely different.

Astrid Atkinson

Yeah. Just as a back-of-the-envelope calculation, I was trying to figure out how many data points do Google's monitoring systems collect on a daily basis? And it should be somewhere in the order of 6 to 10 trillion.

David Roberts

Good God. That's like whatever atoms in the universe. I don't know what the right comparison is.

Astrid Atkinson

Google is where I learned the term exabyte. They sort of just came into this with a perspective on scale that was sort of orders of magnitude larger. And you don't really think about the amount of computing work that's required to get you the right answer in under a second to any question that you might have through search, but the short answer is hundreds of thousands of computers all at once.

David Roberts

Right.

Astrid Atkinson

You can just do more.

David Roberts

Right. And so the current way utilities are running this with sort of a single machine on premises chewing through this data serially is just not going to get anywhere close to that scale.

Astrid Atkinson

Yeah, and to be fair, they'll scale up to five or ten machines. But the systems that we had to build to kind of manage the Google scale tech computing, they weren't five to ten machines, they were 50 to 500,000 machines.

David Roberts

Right. Which you're not going to do on premises. You have to basically move to the cloud to widely distributed resources. So, Astrid, all this sort of preface, conceptual preface, is so my jam. You probably know this because I wrote a big article about grid architecture a few years ago.

Astrid Atkinson

Yes. And that article, if you're talking about the one from 2018, was really influential in how we started thinking about the space a little bit before starting the company.

David Roberts

I mean, that was literally my dream, writing that article, is that some smart person who understands how this works with computers will come over to the utility industry and bring it to the electricity system. Literally. Like, you're the answer to my prayer. But what you may not know is that this is my jam going even way farther back, like back in the midst of time, back in the midst of prehistory. I was getting a philosophy degree, I was working on a PhD in philosophy, and my whole thing was studying cognitive science and consciousness and sort of cognition and how cognition works.

And one of the things I was deep into is this shift of people describing cognition as distributed and describing the human identity as distributed. Sort of this illusion of a unity that actually comes out of massively distributed, relatively dumb nodes coordinating to produce this emergent behavior that we call consciousness. So the whole idea of moving from centralized to distributed, that was in my bones even as far back as grad school. So now it's like popping up all over the world. These are really just thrilling times for me personally, I guess, is what I'm saying. It's so cool to see all this stuff come together.

So one of the things you say in your presentations and such is the first — step one, if you want to do this, if you want to coordinate massive numbers of nodes to get a desired set of behaviors as an outcome, the very first thing you need is knowledge of the nodes. You need to know what's out there and what it's doing and what its capacity is. And in the computer world, we have the system on the Internet, we have the system of IP addresses, an individual address for every single computer that's connected to the Internet. And there are, as we say, millions and millions of them.

So right there, it seems like right there, your attempt to pull this over into electricity looks like it grounds out because we just don't have that. Like when I hook a water heater up to the grid, I don't think it gets an individual identifier, does it? Or a solar panel, or an EV battery or EV charger, et cetera, et cetera. This is the whole problem, right, is all these devices are hooking up to the grid and utilities don't know where they are or what they're doing or when they're going to do what they're doing. And I can't — have trouble imagining a system that would individually identify them.

So right there, right off the bat, how do you overcome this first and most difficult problem? Because as you say in one of your presentations, the pyramid, you show the pyramid of reliability. And the base level, the bottom, the level upon which the rest of the levels are built is monitoring. You have to know what's going on out there on the edge of your network. And in electricity we don't. So how the heck do we overcome that problem to even get to the other problems?

Astrid Atkinson

Yeah, so there's really, from a technical perspective, kind of two questions buried in the one that you just asked. One is about addressing. And it's true that we don't really have an IP address system for the grid. But the reason that we need that for the Internet is that we ultimately need each individual piece of work to end up at a specific location. Like an email needs to end up at a certain address or a DNS query needs to go to a certain server in order to get a response. Whereas in the electrical system, we're really sort of thinking more generally about managing need for work to ability to do work.

So, managing supply to demand, it doesn't really matter which particular electron serves the need. One of the primary goals is just that there are always exactly the right number, always exactly the right amount. So the problem space is a little bit different in that perspective. It doesn't mean we don't need to know what's on the grid and where it is. But you don't have quite the same need for an absolutely universal addressing system. Just as an example: One of the ways that one could tackle this is there's a bunch of really interesting work happening in the grid management and grid architecture space in Australia, which I'd love to talk about a little bit more when we get further through this.

But as part of that, they basically extended a universal identifier system that applies both to meters for one set of identifiers and then to generators for another set. And that's just kind of the country deciding to do things a certain way. And it definitely, I think, can be a helpful enabler to have that. In the absence of that, you can correlate it across multiple sources because within any given utility, they do actually have a way to identify specific assets. Typically that's something associated with the meter number. So it's usually like a site identity or like a site number or something like that that identifies a particular customer location that associates with a billing relationship and associates usually with a specific physical meter and is like kind of the identity component that ties those things together.

David Roberts

Right, but isn't the problem that behind any one of those meters might be multiple devices that behave very differently and have very different timing and etc.?

Astrid Atkinson

Yes, and should you want to be able to tie together the activity of those devices with the utilities identification mechanism, you're going to need to write some software. So one could do this today by basically just like writing software to link together the program identifier or the device asset identifier. Let's say you're talking to ChargePoint chargers. You can get like a ChargePoint charger ID, and with a modest amount of software, you can link that together with a meter ID. It's all kind of a pain in the ass, but it's doable.

David Roberts

Would it not be easier with something like an IP system, though? I mean, is there any talk about is that a gleam in anyone's eye?

Astrid Atkinson

Yeah, I do like the Australian model, I think they call it. It's like a distributed generation ID and then there's like a meter ID. It's basically just an agreement that those things are going to be unique across multiple systems. I think one of the sorts of things that could be easily extracted as an example and it certainly would help. I think there's a lot you can do without it, but it would help. The other thing that's kind of buried in your question, though, is "how can we see what's out there and know what's happening?" And when we talk about monitoring in the Internet space, as you mentioned from my diagram, which is actually drawn by a former colleague of mine, Mikey Dickerson, who went to go lead up — I don't know if you remember, but the healthcare.gov rescue team under the Obama administration.

David Roberts

Heroes.

Astrid Atkinson

Mikey was a colleague of mine at Google, and he went to go lead up that rescue effort. And so the diagram you're mentioning, which is a lovely one. It's just like a little pyramid of — kind of like Maslow's pyramid of needs, but for reliable computing — it does have monitoring and system visibility as its foundation. And what that looks like in the Internet computing space is that for every systems component, whether you're talking about an individual server or a network router or a piece of software or whatever that's operating within the construct of a larger system, you kind of want to know what it's doing.

That's why Google collects so many data points for their systems, because they have quite a lot of them. But this is common across the entire modern computing industry. If you're running a large system, you would typically have a monitoring system that would pick up ongoing telemetry, so like heartbeat information and activity information from literally every component and then assemble that. Firstly store that data and then assemble it into some operator friendly dashboards and ways to keep an eye on what the system is doing. We don't really have that, particularly the distribution grid today. We do have it on the transmission side, but that's not really how the distribution grid rolls at the moment.

David Roberts

Yeah, and so absent a system like that being put into place with unique identifiers for every little piece, ultimately it seems like you're in the end modeling, basically estimating on some level, are you not?

Astrid Atkinson

It's a mix usually. So in practice what that looks like is you do end up with unique identifiers for those assets because if you're bringing in a data stream from an EV charger or a battery or a solar inverter or something like that, you want to know which one it comes from. So in the absence of like a universal addressing scheme, the way that we would do that from a technical perspective is we'd say we've correlated this battery with this site ID, so its identity is now site-id.battery or something like that in our database. So you're actually creating that notion of identity to do that.

And then we're pulling in data, or whoever is pulling in data from ideally every one of those things. Now those things are all exporting data all the time anyway. Like their vendors, their installers are always installing them along with some kind of remote monitoring capability.

David Roberts

Right. Just not to the utility.

Astrid Atkinson

No. And as mentioned, partly that's because the utility in general is not very good at dealing with very large amounts of data, but it's also because that's a lot of vendors to talk to and a lot of devices to talk to.

David Roberts

Yes, and I would imagine a lot of different data formats and it must be a mess of information.

Astrid Atkinson

This is where the modeling stuff comes in because in general, there's lots of data available, but it's almost never universal, it's almost never consistent across every utility or device type or whatever. Sometimes you'll have utilities that have smart meters with 20% of their customers, but not the others. Or you might have ones where they have meters that give you a monthly read for most customers and then a small subset that can give you more frequent data. You might have utilities that know where all their rooftop solar is and have even production meters that tell you what it's doing, but most have an interconnection record that's kind of separate from any other data set.

Maybe it's in their GIS. And most don't have a production meter to tell you what's happening there. And so if you want to make sense of all of that, you do need to use a fair bit of modeling so that can be from everything. Like this meter data has a bunch of drops where the network dropped out and I need to fill it in so that the time series is complete to like the meter data has a different frequency, like it's sampled every 15 minutes and SCADA data is sampled every minute and I want to compare them.

So now, I need to fill in minutely data points using a model. Or it might be I want to have some idea of what rooftop solar is up to, but I don't have any production meters. I do know where there are rooftop solar installations on my system though. And we do have modeling tools that can do a forecast for rooftop solar. So then, what you can do is forecast the likely output of a panel that's yay big at location X, match that against the data that you actually have from the meter, and then form more complex and more complete machine learning models, which can then give you a model-based view of what's probably happening.

David Roberts

Right. So this is where the machine learning and the AI comes in.

Astrid Atkinson

That's right.

David Roberts

Taking the data you do have and extrapolating to a more complete —

Astrid Atkinson

Absolutely, and that's really valuable, not just for things like forecasting, which is kind of the most common use case for it, but it's also useful for understanding what might be happening right now because most utilities don't really have great visibility in real time. To anything that's sort of below the substation or feeder circuit or maybe they have reclosers or something on the line but they don't really have any visibility at the distribution transformer or at the meter that is anything like real time. Meter data usually trails by between 2 and 48 hours.

David Roberts

As I was reading about this and thinking about this, my mind kept bumping up against one thing, which is that a model like this in, I don't know, like global shipping of goods, where maybe you lack certain information sets and you're deriving them from other information and making a model. A sort of like extremely educated guess. And you get close enough for government work, you get close enough to make the thing work. But in electricity systems, you really got to get it exactly right. Like the supply and demand have to be exactly matched at every second. Can you get from the limited data we have now, add a bunch of machine learning and AI to derive a global picture of what's happening? Is that global picture going to be accurate enough to meet the sort of reliability standards we require of electricity specifically? Like people are not very forgiving if you have little gaps.

Astrid Atkinson

So the key question is "reliable enough." Is it going to be perfectly reliable? Absolutely not. And for high fidelity low latency applications, you need high fidelity low latency data. But a lot of our applications are not high fidelity and low latency. And so there's a lot to unpack in terms of how much data you need for particular use cases. But broadly speaking, for things like maintaining frequency of the bulk system, that's something that needs to be done on like a subsecond basis. Obviously, it's kind of a 60 Hz basis, but we have that visibility on the transmission system, and we have transmission operators whose job it is to maintain that frequency.

So if you don't have that kind of fidelity on the distribution side, that's not necessarily going to impact stability from a frequency maintenance perspective, especially if you do have fast response assets like batteries and stuff like that, that the transmission operator can call on to provide services. And there the distribution operator, even if it's distribution connected, doesn't really need to know about that as long as somebody knows that they need it and can call on it. So when we think about what we need in terms of data completeness on the distribution side, it kind of depends on the use cases, right? If we want to be able to just use flexible load to move our peak usage around so that you're not running up huge demand charge bills on behalf of the utility at peak usage times in the evening, you don't really need real-time data for that.

You can just kind of move it around on a day ahead basis, right? If you want to be able to manage, let's say you've got a house with five powerwalls and you want to make sure that when you charge those powerwalls, they don't blow up the transformer that house is connected to. That's also something that benefits from more real-time data, but doesn't strictly require it because you can just kind of put a bit of a margin for error on your calculation of transformer capacity and make sure you operate within it. What you lose is a lot of additional efficiency and the more you want to actually use that active management to manage reliability and grid capacity conditions, the more you do really need the actual real-time data.

But it can be a process. It doesn't have to be perfect from day one.

David Roberts

Right. I mean, a lot of this is just about how to get started with the crappy systems we have in front of us. I'm sort of like —

Astrid Atkinson

I like to refer to it as real world data.

David Roberts

Right, which anyone who's dealt with the real world knows is generally pretty crappy. So what's the sort of balance of efforts of trying to organize people to produce more data on these things, right. Or harmonize data, or come up with some sort of harmonized transferable, mutually communicatable data systems versus effort put into the machine learning and AI that can make hay out of existing data? Does that make sense? Are you just running with the data you got? Or are there also efforts underway to produce better data or to harmonize data across all these systems?

Astrid Atkinson

Yeah, there's really both. So there's a bunch of really good work that's happening in the industry to try to get much higher fidelity data collection and reporting at either the meter or an equivalent kind of customer side component. So companies like Sense do this, SPAN panels can do this. The meter manufacturers are all working on it.

David Roberts

Are the smart meters that got installed in that first wave of smart meter installs, are they useful for this? Like, are they producing good data for this kind?

Astrid Atkinson

The real issue with smart meters, especially in the US, at least from my perspective, is not actually the meters themselves: They collect data every 15 minutes, which is pretty sparse in general —

David Roberts

Smart meters do?

Astrid Atkinson

Yeah, sometimes. Often it's every hour and sometimes it's every month.

David Roberts

That's not super smart.

Astrid Atkinson

It's not very fast. No. And a lot of times it really has to do with the amount of data that the head end system, which is like the software system that picks up the data, can bring back. It also has to do with the bandwidth of the network that's available to the meter to bring data back to the meter as well. So in general, we collect and store a lot less data than we could. And a lot of that has to do with limitations of the communications network and of — not enough computers to bring in the data.

David Roberts

So you could theoretically get more out of those in the future if you built up the infrastructure.

Astrid Atkinson

Yes. The big issue with the data collection from smart meters today, from my perspective, is that most of them in the US communicate via an RF mesh network, which basically uses kind of a — I don't know, you could think of it as like a bucket brigade for data. It's like kind of tossing from one meter to a repeater to an upline repeater, to an upline repeater. And God knows what it's doing out there on the network, but it takes like often 6 hours to get a message back. I like to picture the data, like, having a little picnic out there on the line because it's very slow.

David Roberts

It's like The Hobbit. It's like trudging through fields and over mountains.

Astrid Atkinson

Yeah, and most of those systems will let you get a single point read from a single meter more quickly. But if you wanted universal, somewhat real-time visibility into what's happening at every meter today, for anyone who has those systems, the only way to get that is to forecast it. And so that's where you're really sort of looking at the role of real-time forecasting to try to fill that gap. Now, the issue with getting more instrumentation out there or even getting those kinds of technologies in place isn't really a technical one. It's more that most utilities have never had it and they haven't really felt a need for it.

Right.

David Roberts

Like, moving around large amounts of data is a solved problem, let's say technologically.

Astrid Atkinson

It is something that we have great technologies for.

David Roberts

But they're just not using it. And just as a general comment — and this is something I'd say frequently — I feel like ordinary people who are out there very familiar with tech and the Internet and all that kind of stuff would be surprised if they knew how comparatively low tech the grid is compared to the systems that they're using day to day life. It's wild that there are still people finding out about outages by getting a phone call from Bob, who's walking his dog, and that causes Jerry to have to go throw a physical switch somewhere. Like, the whole thing just seems incredibly like the more you learn about it, you're like, still in 2023?

I don't know if this was your impression when you started getting into it.

Astrid Atkinson

I remember going to do a tour at San Diego Gas and Electric for their operations center not that long after starting the company. And the difference between data fidelity and kind of amount of real-time operations on the transmission side versus the distribution side is pretty shocking. The transmission side is everything you'd expect. Right. I remember when we did that tour, they walked us into a conference room, gave us a little advanced talk, and then flipped a switch and the entire frosted glass wall of the conference room went transparent. And you could see their transmission operations room with their gigantic screens and all these operators in their little consoles with each one has half a dozen computers around them and it looks really high tech.

You go over to the distribution side and it's like a bunch of desks people maybe have two monitors, and it's mostly people taking calls and making calls to roll trucks to fix outages. We have the technology. We even have it in the grid. We just need to scale it.

David Roberts

Yeah. Distribution systems basically have been neglected and badly need now to be beefed up in a number of different ways because of the aforementioned flood of DERs coming.

Astrid Atkinson

Yeah, and if I can mention one of the parallels with my past life doing operations at Google, because part of my job was actually like on-call operations. I led the team that was responsible for Google's homepage for about five years. So you went to Google.com to see if your Internet was on between 2007 and 2012. That was my team and I carried a pager myself for it. One of our big strategic efforts as we were doing that work was really thinking critically about building tools to help operators understand system scale. So if you think about not only a very large system, but also one that's undergoing constant change, where there might be dozens or hundreds or thousands or tens of thousands of individual systems interacting, changes are happening all the time that you don't know about.

And that's true even in a Google type environment where you'd think it would be all coordinated. But Google has like 100,000 software engineers. It might be 200,000 by now. And good God, they can generate new things quickly. And so from the operations side, one of your big challenges is making sure that what's happening within that changing system is comprehensible to a small group of people.

David Roberts

Yes. Can't change human bandwidth.

Astrid Atkinson

That's right. And so one of our kind of core philosophies about that was we wanted to have tooling that would let us support exponential growth in the system with sublinear growth of the operations workforce required to support it.

David Roberts

Right.

Astrid Atkinson

And so that means you're making a lot of ongoing investments in tooling and stuff like that. But it also means that that technology investment is a fundamental part of how you scale the system and that operations function is kind of intrinsically tied to how and whether you can scale the system. So, you know, when I saw that room at San Diego Gas and Electric of their distribution operators working really hard to manage the complexity of the system that they have, I kind of look at that from a systems engineering perspective and kind of a large systems perspective.

And just think, like, wow, making sure that those folks have the tools that they need to understand what's happening as we go through all these changes. That's actually really close to the heart of the problem.

David Roberts

Yeah. And is there an industry doing that? Do the utilities just cook all that stuff in house? Where are the 100,000 engineers that are helping utilities?

Astrid Atkinson

Well, I have a company of about 30, so we're short a few. No, there are people, of course, across the industry that are working on this. And that's typically a mix though, of the existing kind of large scale vendors and folks within the utility trying to kind of roll their own, which is good in the sense that they know the problem space, but bad in the sense that very large scale, real-time operations, cloud computing is sort of a specialty that doesn't have a lot of overlap, necessarily with utility systems training. So we could use a lot more person power on this problem, for sure.

David Roberts

So the point of all this, and this is a point I thought you made it really well in one of your talks is one way we could deal with the growth in demand that we know is coming is just to build more and more grid. But building grid is hard. There are NIMBY problems and capacity problems, money problems. And our ability to build out physical grid is rapidly going to be outstripped by rising demand. So our only real alternative here is to make more use of the grid we've got, is to use the grid we've got more efficiently.

And basically what that means is coordinating the behavior of all of these millions of distributed devices so that they work in concert with generation and everything works in a big happy system together. But right now, I think Volts listeners will be familiar with crude proto versions of this, like demand response systems where the aforementioned phone calls are like someone calls you and says, "hey, will you not run your boiler on x hours of x day?" And they're like, "okay," so that's a form of coordination of those machines. But I think anyone can tell with a little bit of thinking that that's not going to scale up to millions of them.

So the key here, the heart of all this is beginning to automate these things is to beginning to automate the behavior of these thousands and millions of distributed devices. And so here again, I think your experience in Google transfers pretty well since automating tasks, automating work, work routines, and subroutines is kind of at the heart of what you're doing. So talk a little bit about the hierarchy of work and the ladder of automation.

Astrid Atkinson

Yeah, so there are a couple of things in that. First, I guess it's maybe helpful to have a concrete example of what that looks like at Google. So one of the things that we went through at Google was reaching a point where the global load distribution system, which manages all data center traffic, was needing to be adapted to manage really rapid growth in demand for network bandwidth. And so before that, we were really just kind of moving search requests around across a set of global data centers. And that has its own constraints. You need to be able to move that traffic quickly.

You need to be able to respect capacity constraints. You don't necessarily blow up substations if you get it wrong, but you actually can overload and crash servers. And it has a lot of analogs to the grid system in the sense that once you overload servers, you can cause cascade failures that will take down service globally. So there are consequences to getting that wrong even when it's just search traffic.

David Roberts

To put it in electricity terms that the audience will be very familiar with, you're just trying to avoid peaks in computing load. Basically, you're trying to make the demand for computing look like a nice smooth line, even though there's tons going on beneath that line. If you can coordinate it all just right, you get a nice smooth line. And spikes, because you have to build, as you point out in your papers, any network is built to the size of its tallest peak.

Astrid Atkinson

That's exactly right.

David Roberts

So the more you can smooth out those peaks, the more you can grow the whole thing without demanding more infrastructure. And that just transfers very straightforwardly to electricity. You're trying to move load demand around and coordinate it, such as to create a nice smooth demand line.

Astrid Atkinson

Yeah, that's exactly right. And that's kind of the fundamental capacity planning constraint for any large-scale system like that is really what's your peak demand and can you absolutely meet it, or do you have a way to shed some of that demand if you're going to cause system problems? Right, so when we transitioned to needing to be able to scale the use of network bandwidth, it was when Google bought YouTube and we needed to be able to serve lots of cat videos. We started to think about not necessarily taking every user request all the way from Azerbaijan to a data center in California, but rather could you put capacity out in Azerbaijan to serve the most popular Azerbaijani cat videos?

And so we went through this really big transition of how we manage the network towards much more distributed capacity management and also software-based network automation. So really moving towards software-based management of where and how peaks occur, because you're exactly right. And then also trying to move as much work as possible to the edges of the network so that you could get more work done on the network as a whole. So the parallel in the energy system is moving as much work as possible, as close to the user as possible. So that's moving it to distribution, connected resources, and to DERs.

And what that gets you is a bunch of tools for optimizing your network capacity, which, if we want to get our grid to do more without building four times as much of it, is really what we need to do now. So thinking about like, okay, what's the set of technologies that makes that kind of transition possible? If you need 10,000 times more capacity from an existing network, or let's just say like ten times more, or even four times more, which is probably about what we're going to need, you need some tools to be able to handle that. So firstly, the more of that work you can do close to the source, the more network you have to work with.

Secondly, understanding what the network constraints are in a really nuanced way, and also what the need for capacity and the available supply to meet that is in a really nuanced way. The monitoring piece, like what's out there? What's available not just for the system as a whole, but on my street or your street or my substation, is a really important part of that. And then the analog to what we did at Google, which was to put caching, which is basically computers that store information close to the user. So we'd like preload the cat videos that are most popular near your house is basically storage in the grid.

Right. And what that gave us was the ability to do some work locally, which increases the reliability of the system as a whole, because you don't have to have the whole network between Azerbaijan and California working perfectly all the time. A lot of times you can just do the work locally. But it also gave us a lot more flexibility around being able to handle really extreme variations in load because you could have some of that load soaked up locally, you could potentially shed a little bit of it, you could potentially move it around to other assets that could soak it up somewhere else.

And so when I think about that from a grid perspective, the sorts of changes that would support that, they're not huge. We need a better sense of what's going on. We need more control over load showing up, when and how it shows up. And ideally, the ability to smooth out peaks and some reasonably large amount of storage distributed broadly across the grid to like, substation or lower would give us a lot of flexibility in how we manage that system. So if we're not exactly right about exactly how much the planned demand is going to show up, you can just kind of soak it up with local storage and not have to worry about it too much.

David Roberts

Get you a little buffer there.

Astrid Atkinson

Yeah.

David Roberts

So the equivalent here is if I need an electron to run my toaster and the grid can be coordinated, such as to provide that electron from my neighbor's solar panel, let's say. You're using the minimal amount of the network and leaving as much of the network possible open for other things, basically, like you're minimizing total network load work.

Astrid Atkinson

Absolutely. And to some degree, you do that already, right. Like, let's say that your neighbor, like me, has an oversized solar panel, I have 9 kW on my garage. During the day a lot of times I'm back feeding into the system, so I'm powering my neighbor's toaster. The reason that doesn't really get us very far in terms of saving money on grid upgrades or really contributing to the health of the system today is that all parts of the grid still need to be sized as if that didn't exist because we don't have any way to guarantee that the peaks don't show up.

David Roberts

Right. And this is the thing, is when you're building a network, if you've eliminated 99% of the peaks, you still got to build the network as big as the last remaining peak is.

Astrid Atkinson

Or you need a last-ditch alternative to be able to shed that one remaining peak. There's a couple of different ways to do it, but yes.

David Roberts

There's so much in here, but this gets to the grid architecture question that I was writing about back in 2018 with my cool diagrams. That my —

Astrid Atkinson

They were so cool.

David Roberts

I know. I've seen that you're using variations of them in your presentations and it makes me so happy.

Astrid Atkinson

I'm glad it makes you happy because we thought that they were amazing and really explanatory and found them very inspiring.

David Roberts

That's Javier at Vox. I'll put a link to that piece in the show notes. But basically, the idea here is that you go to the lowest level, the edge of the grid. Let's say the distribution node, a node on the distribution grid. The idea is you satisfy as much of the demands of that distribution node with resources within that distribution node. Net that out. So the only power that the distribution node is asking for from the transmission system is whatever is left over once it has subtracted its own supply from demand or demand from supply. And then you sort of move up levels.

Right. So in a sense, everything is kind of a microgrid. It's like microgrids within microgrids within microgrids. I don't even know if that's the right terminology, but islands within islands within islands.

Astrid Atkinson

Yeah. You ultimately be looking at a system with multiple abstraction layers where some amount of management is done locally, and then those nodes are connected to the broader system, and then the broader system does its own coordination. But the thing that's really nice about that model is that you don't have to worry about your toaster from the California ISO.

David Roberts

Exactly. There's someone in between the California ISO and me that's worrying about my toaster. Some entity or —

Astrid Atkinson

That's right, yeah. And technically, you could build a system that did kind of coordinate from the ISO to the toaster. But what you would lose in that kind of system that I think is really important is some amount of local abstraction layers and local management entities where you can optimize for different kinds of work. So optimizing for the temperature in my house is actually kind of different than optimizing for the load on my transformer and kind of different than optimizing for overall supply and demand for the state of California. It is reasonable to have those things handled by different layers, different entities, even.

David Roberts

Right. And there might be neighborhood goals or values or city goals or values that are slightly different than the state goals or values and so on. And also, just as a matter of computational complexity, I mean, this gets to what I was trying to get out of my article. It's one thing for these ISOs or RTOs — these sort of entities that are coordinating transmission grids — it's one thing when you're coordinating — call it like five big power plants in a couple of dozen distribution nodes. Right. As we were saying before, that's a finite number. You can control that more or less centrally.

Astrid Atkinson

Model it deterministically.

David Roberts

Yeah, you can model it deterministically. But once you're getting to thousands and millions of devices trying to coordinate all that at every level from a single point, from a single operator at the top. It just seems to me it's rapidly going to overwhelm their ability, their ability to — and why would you want to? It just doesn't make sense to have one giant regional organization coordinating your toaster, as you say. All of this brings us to the DSO model, which is I was delighted to find beloved by both of us.

Astrid Atkinson

Absolutely.

David Roberts

Which is you need an entity — you know, we can discuss whether it has to be a singular entity, but let's just call it an entity for now. You need an entity at the distribution level, at the level of that distribution node that is responsible for coordinating the activities of all these DERs within that distribution node. Matching supply and demand, netting it out, determining how much power is needed by the distribution system. And then that node, that DSO, the distribution system operator, does that work. And then the only signal it needs to send up is a single signal, like we need x amount for our distribution node.

Right. So it's taking all that complexity at the local level and simplifying it before it sends it upward, basically. And you have that at every level you're simplifying and sending upward. So that once you're at the top level, the ISO, you're dealing with a tractable number of signals rather than trying to talk to every toaster.

Astrid Atkinson

Yeah, definitely. And even for very large scale load balancing systems like the ones that we used at Google, they were still built that way. Right? It was like local coordination across work within a single data center that was then providing a simple interface to global coordination systems, which basically were like, "hey, how much load you got? How much load can you take?"

David Roberts

Right?

Astrid Atkinson

And then would readjust allocations every second or two.

David Roberts

Right. So from the transmission operator's perspective, a distribution node will basically just be a single machine that either has a set amount of demand or can offer a set amount of supply.

Astrid Atkinson

It would ideally look a lot like battery does on the ISO system today, I think. Where you've got a certain amount of power, need a certain amount of flexibility available, perhaps a certain amount of power available. But I think something like that would be a small enough adjustment on top of how we think about managing at the transmission level today to be practical.

David Roberts

Do you? There's a large amount of reform —

Astrid Atkinson

Don't get me wrong. There's definitely new software components, new data components, new business model components, all the above required to do this.

David Roberts

Well, it's the business part and the regulatory part that's baffling to me because that's the hardest, in my experience, the hardest part to get moving on. And right now, just to be clear with everyone, there is no such entity in the US. Basically, there's no entity responsible for handling the complexity at the local level, simplifying it and then sending the signal up. So we would have to create those out of parts. So I know Lorenzo Kristov, who I drew on to write that original piece, who's kind of the guru of all this stuff, I think has been beating his head against the wall in California, trying to get them to run DSO sort of test models or create test DSOs.

So who out there is doing that? And what does the DSO model look like? Is it up and running and working anywhere or is it all experiments?

Astrid Atkinson

Yeah. So this is where I get to talk about Australia, as promised, because Australia has a really nice large scale demonstration of this type of model that they're actually pretty far into at this point. You know, like many regions, including the UK, a couple years ago, Australia sort of took a look at the oncoming challenges associated with DERs on the electrical system.

David Roberts

Yeah. Swarming rooftop solar per some previous podcasts here, tons of rooftop solar. Way ahead of anybody else.

Astrid Atkinson

Absolutely. And so they were a little bit more motivated, maybe, than other places to try to solve this problem. And so there was a bunch of really interesting work led up there, mostly out of AEMO, but also out of the CSIRO and some of the universities and stuff as well. And I know what this is because I'm from Australia, but what they did was come up with a couple of kind of trial models of coordination between AEMO, which is the Australian market operator this is kind of the ISO, and the distribution network operators and the aggregators and retailers that were working within some of their key markets.

And one really relevant example of this work was something called Project Edge, which you can Google, although you might need to, I don't know, add AEMO or something to it, just AEMO. And what that was doing was looking at a coordination model exactly like the one that you're describing. So it was asking the distribution network operator to take on more of a DSO role. It was asking the aggregators to provide data about the location of their devices, what they plan to do with them, those kinds of things. And AMO is playing a coordinator role to kind of sponsor that and bring some of the data together.

But ultimately, it's a sort of joint function between those three parties. That's just one example of how this model could work. The UK is taking a little bit of a different approach with the central regulator and kind of central operator National Grid taking a bit more of a central role in that. But still, it's kind of looking at the idea of cutting up the system into multiple localized components and then having some entity take on a distribution system operator role to collect that data about the state of the grid and the things that are participating in it.

Coordinate that with data from the aggregators and coordinate that with data about market participation. And in Australia, they liked the results of that so much that they are planning on rolling it out nationwide.

David Roberts

Interesting.

Astrid Atkinson

Which is an advantage of being a smaller country.

David Roberts

You know, they've got so much rooftop solar now that they're having duck curve problems and even some stability problems, I think. So where they've tried this, they're solving that kind of duck curve-ish problem.

Astrid Atkinson

Yeah. So some of the early applications in Australia particularly focus on being able to curtail rooftop solar in places where they have basically oversupply issues. So it's a duck curve problem. Right. And just given the way that sort of social license and kind of public sentiment works in Australia, if you're curtailing somebody's solar, the Australian consumer citizen will kind of expect you to pay for that just as much as they would expect you to pay for the energy that they provided when it wasn't curtailed. And so they needed a market construct to support that. The other thing that is happening within that broader model that I think is really interesting is also the ability to do basically flexible interconnections.

So being able to basically say you can connect this new load, EV bus charging is my favorite example, but only if you don't exceed a certain capacity allocation. So you can connect it and it's kind of up to you to manage that. And we will give you that capacity allocation either as a fixed allocation or even better as a dynamic allocation. And that idea of like a dynamic capacity allocation that a user has to stay within is called dynamic operating envelopes. And that's a sort of technical component of that Australian work that helps the system operator to manage basically the capacity allocation to every individual DER, consumer or producer.

David Roberts

Right. And this is where again, automation comes in because you're not going to get every bus charger operator to sit there with their hand on the lever, no pulling it up and down as these dynamic constraints change.

Astrid Atkinson

Yeah, this is a job for computers, this is not a job for humans.

David Roberts

So there are DSO models out there happening. In the US, let's talk about the weird US situation. So in the US, recently, FERC issued this order is it 2222?

Astrid Atkinson

Yeah.

David Roberts

2222, which says aggregators — which just in case people are not familiar with these, that's just like a third party entity that strikes contracts with dozens or hundreds or thousands of DER owners to basically give them control over those DERs so that they can treat them as an aggregate, treat them as a big, giant generator, or treat them as a big, giant battery — and then FERC's order 2222 says those aggregators can play in wholesale energy markets, basically.

So you can take an aggregation of DERs and pretend to be a power plant in the wholesale power market, basically, or act as though you are a power plant in the power plant market. And that's a response to the need for our system to make more of these DERs, to use these DERs rather than just be a victim of these DERs, to take some control of them and use them in such a way that they're useful. But to me, that just feels cludgy, feels like a half-ass solution. Because you have this weird thing where pretty soon, like I said, these wholesale markets, depending on how many aggregators you have, and I'm not sure totally how that market is going to shake out, but there could be a lot of them eventually. Again, you're just like, why have the one central coordination of all those machines?

It just feels like that should be resolved, that the stuff among individual DERs should be resolved at the local level and not something for the ISO to have to be dealing with. And also, so you have sort of aggregators kind of speaking directly with the ISO, kind of bypassing any DSO. It's just a weird kind of hybrid model of localness and centrality, I guess. To me this seems like a temporary fix on the way to something better. How do you feel about 2222?

Astrid Atkinson

I think it's definitely a key part of a solution. I actually really like 2222 for a little bit of a roundabout reason. What it does is mandate that aggregators should have access to wholesale markets. And it is entirely silent on the interaction of the DERs that they control with the network that they're actually located on on the distribution side, which is kind of bad in the sense that you really want to know probably what's happening with any assets that are moving a lot of load around on the distribution side.

David Roberts

Yeah, if you're running a distribution grid and a bunch of the machines within it, their behavior is being coordinated on behalf of the ISO. That might not be exactly the kind of behavior you need for stability at the local level.

Astrid Atkinson

Absolutely. It might cause a lot of problems, and I think it's actually likely to cause some problems. However, there's nothing to stop a given distribution operator/utility from deciding that they want to understand what's happening, that they want to be able to play a coordinative role or at least get data from aggregators about what they're doing and to say like, "look, I want to take on a distribution system operator role within this broader system." And there are some implementations of 2222 that are kind of going that direction. The one I'm particularly thinking of is PJM has a filing for their implementation that does include coordinative function with the local utility.

And I think that's really interesting. Now it doesn't necessarily 100% meet the needs of aggregators because it basically says that the local utility should have visibility into and perhaps dispatch control over aggregate resources.

David Roberts

Kind of a veto, right? Because it's those local needs. That ultimately are the that's the actual toaster coming on.

Astrid Atkinson

So, you don't really want to veto either. But one thing that I think is really important to remember about those aggregators and the resources that they're managing on the customer side is that they want to get paid for the services they can provide. And one way to get paid for that would be to bid them into the wholesale market. But the value of the service that those local DERs could provide is actually a lot higher to the distribution utility that they're located on. If we look at kind of all up and down the value stack, selling energy on the market is part of that value stack.

But selling peak shaving at the local transformer or substation level theoretically is very valuable in practice. No one's paying today.

David Roberts

Right. If there were a market at the local level, this is the thing. It almost seems like the DSO should be running a local market.

Astrid Atkinson

Absolutely.

David Roberts

And the DSO should be sort of serving as the aggregator. Right. So you let the local market do its thing, and then you bid for whatever's left. You bid up into the wholesale market. Does that make sense?

Astrid Atkinson

Yeah. And so I think what this does is provide an opening to start having that conversation. FERC 2222 doesn't really provide the structure for that kind of DSO/ISO aggregator coordination function as we see in other countries. But it does provide kind of a stick in the kind of carrot and stick sense, in the sense that you could either choose to take a leadership role in coordinating the behavior of assets on your network as a utility, or you could just let that happen and kind of deal with the stability and cost implications on the back end.

And don't get me wrong, there's probably a bunch of utilities that will end up doing the latter, but there is an opportunity to do the former, and some will go that way.

David Roberts

Yeah. So in your efforts to cajole utilities into doing things differently, I feel like the landscape is littered with the exhausted husks of people who have spent their lives trying to get utilities to do things differently. You're focusing specifically on co-ops and munis as kind of places to experiment with this more local model, this DSO model. Why is that?

Astrid Atkinson

So, we do also work with investor-owned utilities, but primarily with ones who have a vision for taking that leadership role and really kind of want to move that direction. But one thing that's really nice about working with co-ops and munis, which are both nonprofit, typically local utility structures, is that they have a very local set of motivations around serving the community, keeping costs low within their specific community. And they're also nonprofits, which makes things a lot simpler from a business model perspective. They also are pretty sensitive to the cost of energy, which for most investor and utilities is financially speaking, it's a pass-through.

It's not really part of their profit model.

David Roberts

Right.

Astrid Atkinson

So for a coopera-muni to save a bunch of money on the cost of the energy that they procure for their customers, it's like kind of a direct benefit because that goes back to the customer in the form of lowered bills or for co-op, they even sometimes send checks back to their members, which I think is really cool, but they care a lot about this and that's a business motivation for them. Likewise, if they are looking at substantial system upgrades because they're going to see a lot of load growth that might be from electrification. But actually for these utilities today it's more often because somebody is like planning on putting in a factory or a data center or something like that.

They're open to looking at non-wireless alternatives and kind of using smart management to avoid doing a very expensive substation upgrade again because they would have to pass that cost back along to a very limited number of members. And it's not like investor and utilities don't have a similar broad motivation or care about it. But if you're a co-op executive or staff member, people come up to your front desk asking about like "hey, why is my bill $15 higher?"

David Roberts

We'll run into you in the supermarket.

Astrid Atkinson

That's right. And it's just a very different relationship with the community for those utilities.

David Roberts

So you think they're more open to this. Have you gotten movement? Are there US co-ops and munis that are setting the standard here?

Astrid Atkinson

Yeah, so there are a bunch of utilities that have been really interested in kind of moving down this path, towards a distribution system operator model and taking on a more active kind of local system operator role. And that's true across kind of all of those segments from co-op to muni to some IOUs as well. The thing that's nice about the co-ops is that they can move faster because they're small, and sometimes also they will just decide that something's working and decide to roll it out broadly. Whereas if you're at an investor and utility and you had the greatest thing in the world, you'd probably still be stuck going through a kind of regulatory approval cycle for scaling it up.

David Roberts

Process, process, process.

Astrid Atkinson

There's good and bad to that. But the co-ops are often self-regulating or at least minimally regulated by the Public Utility Commission because they are deemed to be acting in the public interest so they can move quickly. And so there are a subset of co-ops which are rural electric utilities that have been experimenting with these models and are moving very quickly in this direction and it looks different depending on the utility as to what that looks like for them. Some of them have been pushing really hard on generating energy locally and avoiding the wheeling costs of shipping it across transmission and putting that money back into the community.

So there's one that we work with in northern New Mexico that does that, and they serve the area around Taos, and they decided that they wanted to get to 100% of their daytime electricity load served by local solar. They're actually at about 120% now.

David Roberts

Oh, wow.

Astrid Atkinson

And they're starting to look at models where they can export power out of the community, which I think is a really interesting economic growth opportunity for communities like that.

David Roberts

Yeah. Again, if you can imagine these local markets, you can also imagine markets in between distribution nodes. Right. Like, imagine if my distribution node, I'm handling all the complexity within the DSO, handling all the complexity within the distribution node, comes up with a certain amount of leftover demand that it needs satisfied. It could procure that from the next distribution node over. Right. And it's in a peer-to-peer transaction and not have to involve the ISO at all, theoretically.

Astrid Atkinson

Yeah. And there's probably a bunch of different ways that this might play out, but you certainly could see a world where that's the case. And it has a delightfully localized kind of quality to it in the sense that the community is mostly self-supplying in local electricity and including batteries and flexibility that get them through the night. But they also have this opportunity to potentially export a resource that they have a lot of, which is pretty cool because it's sun.

David Roberts

Part of the delightful locality of it is that some communities might value resilience more, so they'll want to, say, bank more of their excess solar and batteries. Some communities might want the money more so they'll be more likely to sell it across to a different distribution. You can see communities will actually have much more fine-grained control over their energy. And one of the points Lorenzo makes a lot of times is this would make it much easier for local electricity and energy policy to be coordinated with local building policy and local transportation policy, and basically, like, your local — becomes part of how you want to run your local area.

Astrid Atkinson

Yep. And ideally, that would include members of the local community getting paid for the flexibility that they provide, too, because that's a big part of how we get this done. One other example I'd want to give for that is there's another utility I know pretty well. It's in Colorado. They're expecting to double their load in the next five years.

David Roberts

Just population growth?

Astrid Atkinson

No, it's actually a mix of factories, clean energy production facilities, delightful also, and data center loads, which are really growing due to all those big cloud computing facilities that we mentioned earlier. So they're looking at doubling their current energy usage, and there's no way they're going to get enough transmission additions built out to support that.

David Roberts

Right. Especially since it takes 10 to 15 years to build one line.

Astrid Atkinson

Yes. Transmission is one of the few industries where you can pass along a single project to your children, which is not a compliment.

David Roberts

A legacy project.

Astrid Atkinson

Yeah. And so they're looking very closely at models where they can trade generation and flexibility and storage amongst their large users, particularly within their territory. Because the usage profiles for a refrigeration facility are different from that of a shipping facility. Maybe a lot of EVs charging and different again from a factory facility. A lot of those would also be inclined to put local generation into the commercial and industrial site and getting all of that stuff signed up to become part of the future overall supply profile for that territory, it requires a DSO function. There's kind of no other way to do it.

David Roberts

Yeah. Somebody's got to be in charge of all that.

Astrid Atkinson

Yeah. And so that's a place where the utility taking that leadership role really makes a lot of sense because I don't know how else they would manage it.

David Roberts

Yeah. What do you think are the prospects of basically just local distribution utilities right now which are just sort of running the wires right now and billing customers growing into the DSO role? Like when you bring it up to them, do they just blink at you? It's such a sort of cosmic upgrade of their role and importance and responsibilities, etc. Are any of them eager to do that?

Astrid Atkinson

Yeah, I mean, there are a lot of utilities that see this as being part of their future. I think the tricky part is it's a really big change for an industry that hasn't had load growth in 20 years. Being able to suddenly adapt to all of the technology and organizational and business model changes required to support that is going to be a lift.

David Roberts

Yeah. It's wild. Like 20 years without load growth and without really substantial change in the industry and all of a sudden now it's like new tech, new models, new regulatory models, new legal models like boom boom boom boom boom. It's a lot.

Astrid Atkinson

Yeah. And so there are utilities that are definitely thinking about this. Like SCE has a public roadmap that's really nice that covers a transition to a DSO model. The only thing that's a bit of an issue with the normal utility process for this kind of change is that it tends to be very slow and in many cases we're going to need to see rapid adaptation in the next like three to five years. And for most utilities, they're used to kind of planning on a five to ten-year time horizon.

David Roberts

Yeah. Utilities and PUCs, I mean, I'm not sure that PUCs are exactly legendary for being agile either.

Astrid Atkinson

They do compare notes. So if you get something that works well in one place, a lot of times they will try to spread that more broadly. And so, from my perspective, one of the best things that we can do to make this happen is just show it operating at scale in as many places as possible. Not all of those will be right the first time, but it's not the kind of problem that you can really sit down for five years, come up with a solution to and then implement.

David Roberts

Right, which is the utility way, right. You make this point in one of your talks too. It's like the whole software world model, which is that you sort of build, iterate, test, learn, rebuild, reiterate, test, learn that's foreign to the utilities.

Astrid Atkinson

And it's not that, in the software world, it's not like we don't plan ahead or design things. Of course we do. But you design with the idea that rapid iteration is going to get you closer to the goal more quickly.

David Roberts

Right.

Astrid Atkinson

And we've seen really good success with this in adjacent industries. Right. Like one of my old bosses at Google was really interested in skydiving and decided that he wanted to jump from space from a weather balloon.

David Roberts

Such a Google guy.

Astrid Atkinson

This is definitely a Google problem. And so in order to do this, he acquired — this is kind of post-Google for him, but he got really interested in spacesuit design, and then worked with the company in the US, that's a premier designer of spacesuits, who had not designed a new one since, like, 1973, to rapidly develop, prototype, test and deploy new spacesuits so that he could jump from space in a weather balloon. Which by the way is a total badass move. He's an engineer from the ground up. But so they worked in a really iterative kind of rapid development and testing cycle and they got that done within a couple of years and now the same company is developing the spacesuits that are like the next generation ones that are used for SpaceX and Blue Origin and all of those.

David Roberts

Yeah. I mean, if nothing else, this is like a chance to be a hero, right? This is a wide open field and there's just so many opportunities here for innovation, for people to try new things and show successes.

Astrid Atkinson

Yeah, it's a really exciting time in the industry broadly and for the subset of folks at utilities that are really actually interested in thinking about what the future of the industry is going to look like and kind of working towards building that. Boy, it's an exciting time to be in that industry too.

David Roberts

I know. I've been writing about this stuff for 20 years now and it is still somewhat head spinning how all of a sudden it's just all happening.

Astrid Atkinson

It's changing really fast on the ground.

David Roberts

It's just waiting and waiting and then boom, all of a sudden it's all happening. So I would feel bad if we did this whole thing and I didn't give you a chance to sort of say what your company does. Your company is called Camus. Why Camus, by the way? I was thinking existentialists. This must be about the existential despair that you experience when you contemplate trying to reform utilities?

Astrid Atkinson

Not exactly, but related. So we are named for this philosopher, and the reason for that is actually really specific. So Camus wrote this essay called The Myth of Sisyphus.

David Roberts

Yes.

Astrid Atkinson

There's a short version in a longer book by the same name.

David Roberts

"One must imagine Sisyphus happy." One of my all-time favorite philosophy quotes.

Astrid Atkinson

Yeah. So the short version is only about three pages long, and I totally recommend it. But Camus is basically asking, "how do you create a sense of meaning in the face of a large, uncontrolled, potentially godless universe?" Right? Like, where do you derive a sense of purpose? And his answer for this was basically, you pick something, you work on it, you find joy in the process. Not in the notion that you're going to win, but in the everyday act of pushing a rock up a hill and following it back down.

David Roberts

I love it. At the base level, I think there are sort of like levels of your service. At the base level, you're just helping utilities be more aware of DERs and then sort of laddering up from there, like coordinating them, et cetera, et cetera. So, how high up do you go on that ladder?

Astrid Atkinson

Well, since we've gone to the trouble of describing a DSO and what it does, our goal is really to create a software platform that will enable a utility to take on that role. So a much more real time and local operations model that can include local resources as part of the supply and demand landscape and ultimately include them into capacity management and network management for the grid and let them get paid for it.

David Roberts

Got it. So you're creating the tools for the DSOs for whenever they show up?

Astrid Atkinson

We are creating the tools for the utilities that want to take on that role and working with them to figure out what that looks like in practice.

David Roberts

Right. And we should emphasize for the utilities out there listening, this is not a binary thing, it's not like —

Astrid Atkinson

It's a process.

David Roberts

jumping off the high dive into the whatever. There are pieces of this you can adopt, one at a time.

Astrid Atkinson

Yes, that's right. It's a process. Right. The question is not like, what's it take to get to the grid of tomorrow. The question is more like what is the set of reasonable steps that you can take with the data and control capabilities of today to add more sophistication, get better visibility, add coordination, talk to aggregators, coordinate with FERC 2022 deployment and 2222 deployments, all of those kinds of things. So it's going to be a process, but it doesn't have to be impossible.

David Roberts

But it's just great that this idea of making the electricity system more like the Internet goes way back. As I'm sure you're aware, Al Gore was talking about the Internet. Of course, he's trying to coin a term for it back in 2007. And I went through a period of hype for it, and being very excited about it, and DERs, and all that. And then I sort of ran up against utility and transigence and had my life force drained. But now it seems like at long last the hype cycle has come back around and it's actually happening now.

Like there's actual things happening, actual movement in that direction happening.

Astrid Atkinson

It's all happening.

David Roberts

It's very exciting. So final question then, and this is kind of a bit of a curveball, but I'm curious. So say we imagine our glorious future here in the US, where we have revolutionized the system. And we now have all local electricity being administered and run by DSOs, who, as we say, resolve the complexity of the local area before passing on a signal upward, maybe nested a couple of levels. Maybe like the level above them has three or four distribution nodes and the level above them, et cetera, et cetera. But by the time it gets to the transmission level, you've already maximized the use of local resources.

Basically, the goal here is to maximize the use of local resources before calling on large scale distant resources. So imagining that glorious future where we've made that happen, how much of net US energy do you think will come from local resources versus still coming from big utility scale power plants on the transmission system? Do you have either a predicted or desired balance of those two in your kind of perfect world?

Astrid Atkinson

So I've seen modeled estimates that put that somewhere between 30% to 50%. And you talk to other folks in the space who do modeling on this. There's some really good work from Vibrant Clean Energy that did a bunch of work on the potential role of local resources a couple of years ago. But I think somewhere in that kind of 40% ish space is likely and practical. In Australia right now, by the way, it's sometimes 50 and sometimes 70% local. So it's just like do you mean instantaneous or overall? Because if it's instantaneous, sometimes it could be 100% or even more if we're storing.

David Roberts

Right. I mean overall, like on a net yearly basis —

Astrid Atkinson

But on an overall basis, because the sun does go down and the wind doesn't always blow, as you covered very well. I feel like 40% is pretty reasonable.

David Roberts

Interesting. That'd be a good, like something to go to one of those betting sites, start a pool on, say, 2040. What's the net balance? Well, Astrid, this has been an absolute delight, as I knew it would be. I love this whole subject. I love what you're doing. As I say, it feels like you are someone that I willed into being by writing my 2018 piece, which is I'm sure you existed before that, but —

Astrid Atkinson

I certainly did. But the piece made a huge difference in how I was thinking about the space. So I'm pretty grateful for that.

David Roberts

Awesome. Well, it's a delight to have you on, and I'd love to have you back on again sometime once this stuff evolves some, because this is an endlessly, inexhaustibly fascinating topic. So thank you again.

Astrid Atkinson

Thank you so much.

David Roberts

Thank you for listening to the Volts podcast. It is ad-free, powered entirely by listeners like you. If you value conversations like this, please consider becoming a paid Volts subscriber at volts.wtf. Yes, that's volts.wtf. So that I can continue doing this work. Thank you so much and I'll see you next time.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.volts.wtf/subscribe