The Carbon Costs of AI Training and Inference

In this episode, we honor the memory of Abhishek Gupta, who was an instrumental figure in the Green Software Foundation and a Co-Chair of the Standards Working Group. Abhishek's work was pivotal in the development of the Software Carbon Intensity (SCI) Specification, now adopted globally. His tireless efforts shaped the future of green software, leaving an indelible mark on the industry. As we remember Abhishek, we reflect on his legacy of sustainability, leadership, and friendship, celebrating the remarkable impact he had on both his colleagues and the world. We are airing an old episode that featured Abhishek Gupta, Episode 5 of Environment Variables. Where host Chris Adams is joined by Will Buchanan of Azure ML (Microsoft), Abhishek Gupta; the chair of the Standards Working Group for the Green Software Foundation and Lynn Kaack, assistant professor at the Hertie School in Berlin to discuss how artificial intelligence and machine learning impact climate change. They discuss boundaries, Jevons paradox, the EU AI Act, inferencing and supplying us with a plethora of materials regarding ML and AI and the climate!

Connect with us on Twitter, Github and LinkedIn!

TRANSCRIPT BELOW:

Chris Skipper: This week on Environment Variables, we have some sad news.

We have to report the untimely passing of Abhishek Gupta. Abhishek was a key part of the Green Software Foundation. He was the co chair of the Standards Working Group from the GSF's formation to his passing on September 30th of this year. I would like to read out an in memoriam that was posted by Asim Hussain, honoring the great man, his legacy of leadership, his collaboration, and the heart that he put into the Green Software Foundation.

When the Green Software Foundation was formed over three years ago, Abhishek offered to help. The standards working group was looking for a co chair, and he jumped in to help without hesitation. He led the standards working group for over three years, with the full support of everyone involved. Leading a work group isn't about imposing your will on others, It's about finding common ground, nurturing discussions, and searching for the truth.

Abhishek excelled in all of those areas. He leaves a powerful legacy with the Software Carbon Intensity SCI specification. His tireless efforts over the years led to a consensus on the specification, which was later published to ISO in late 2023, and is now being adopted globally.

The impact of the SCI is truly global, with academics and enterprises worldwide adopting it. This widespread adoption is a testament to Abhishek's vision and dedication and his influence will be felt in every software product that bears an SCI score. His legacy is not just a memory, but a living testament to his work.

He has left an unforgettable mark on the community. And will be remembered for his contributions for years to come. Aho brother, I'll see you on the other side, Asim Hussain. To honor Abhishek, we're going to revisit an older episode of Environment Variables from the very start of the podcast, episode five, How Does AI and ML Impact Climate Change?

In this episode, host Chris Adams is joined by Abhishek Gupta, Lynne Kaack, and Will Buchanan to discuss these topics. So, without further ado, here's the episode.

Abhishek Gupta: We're not just doing all of this accounting to produce reports and to, you know, spill ink, but it's to concretely drive change in behavior. And this was coming from folks who are a part of the standards working group, including Will and myself, who are practitioners who are itching to get something that helps us change our behavior, change our team's behaviors when it comes to building greener software.

Asim Hussain: Hello and welcome to Environment Variables brought to you by the Green Software Foundation. In each episode we discuss the latest news and events surrounding green software. On our show you can expect candid conversations with top experts in their field who have a passion for how to reduce the greenhouse gas emissions of software.

I'm your host Asim Hussain.

Chris Adams: Hello there and welcome to the Environment Variables podcast, the podcast about green software. I'm Chris Adams filling in for Asim Hussain, the regular host while he's on paternity leave with a brand new baby. I met Asim on climateaction.tech, an online community for climate aware techies. And I work for the Green Web Foundation where we work towards a fossil free internet by 2030, as well as working as the co chair for the Green Software Foundation's policy group. Today, we're talking about climate change, AI, and green software. And I'm joined by Lynn, Will, and Abhishek.

Will Buchanan: Thanks for having me. My name is Will. I'm a product manager on the Azure Machine Learning team. also a member of the Green Software Foundation's Standards and Innovation Working Group. Within Microsoft, I foster the Green AI community, which now has a few hundred members, and I'm also a climate activist that's focused on pragmatic solutions to complex environmental issues.

Recently, I shipped energy consumption metrics within Azure Machine Learning, and we are about to publish a paper titled Measuring Carbon Intensity of AI in Cloud Instances, which I think we'll touch on today.

Abhishek Gupta: Well, thanks for having me. I'm Abhishek Gupta. I'm the founder and principal researcher at the Montreal AI Ethics Institute. I also work as a senior responsible AI leader and expert at the Boston Consulting Group, BCG, and I serve as the chair for the Standards Working Group at the Green Software Foundation.

So I've got a few hats on there. Most of my work, as it relates to what we're going to talk about today, It runs at the intersection of responsible AI and green software. In particular, what's of interest to me is looking at how the intersections of social responsibility, the environmental impacts of software systems, in particular AI systems, can be thought about when we're looking to make a positive impact on the world while using technology in a responsible fashion. I also, as a part of the Green Software Foundation, help through the standards working group, come up with the software carbon intensity specification, where we're trying to create an actionable way for developers and consumers of software systems to better assess and mitigate the environmental impacts of their work.

Chris Adams: Okay. And Lynn, last but not least, joining us from Berlin. Thank you very much for joining us. 

Lynn Kaack: Yeah thank you so much. I am an assistant professor at a policy school, public policy school, called Hertie School in Berlin. And I am also a co founder and a chair of an organization called Climate Change AI. with Climate Change AI, we facilitate work at the intersection of machine learning and And different kinds of climate domains, focusing on climate change mitigation and adaptation. And in my work, in my research, I am looking at how we can use machine learning as a tool to address different problems related to energy and climate policy. I'm also interested in the policy of AI and climate change. And today, actually, since we're talking about papers, I have a paper coming up. That is called Aligning Artificial Intelligence with Climate Change Mitigation, where we look at the different impacts from machine learning and how they affect greenhouse gas emissions.

Chris Adams: So we actually have some decent deep domain expertise and I'll try to keep this quite accessible, but we might drop into a little bits of like data science nerder on here, but the podcast has done that previously and it turns out to be something that we've got some decent feedback from because there aren't that many podcasts covering this. Okay so let's, uh, get into this topic of green AI and climate change. As we we know, IT is a significant driver of emissions in its own right. When we think about the climate crisis this year, the IPCC, which is the Intergovernmental Panel on climate Change in their big reports, which synthesized literally thousands of papers explicitly called out digital as a thing we, we should be talking about and thinking about.

And if you're a responsible technologist, it seems like a thing that we should be taking into account here. Now, I found it helpful to think about IT IT a little bit like how we think about the shipping industry, partly because they're similar in terms of emissions, which is around between 1 and 3%, depending on what you look at it, but also in that both of these acts like kind of connective tissue for society.

We also think of IT as a kind of force multiplier for The existing forms of activity. So if you use it, which is in line with the recommendations of the science, that's a good thing. But if you use it to do something, which is kind of rejecting some of the science, it might not be such a good thing. And within technology, AI and machine learning in particular is one of the fastest growing sectors and often seen as one of the biggest levers of all. So we're going to highlight some interesting projects we'll start off with. And out of that, we'll probably dive into some specifics about that or some other things you might want to take into account. If you're a technologist wanting to incorporate an awareness of climate into how you work and build greener software, then finally, we'll hopefully leave you with some actionable tips and techniques or projects that you may contribute to or use in your daily practice.

There's another term that we might be touching on here when you're making AI greener, and that's specifically Green AI. Is that the case, Will?

Will Buchanan: Correct. And that actually was coined by researchers a few years ago, Roy Schwartz, Emma Struble, Jesse Dodge. And it's really focused on making the development of the AI system itself more sustainable. And it's to be disambiguated from the term using AI for sustainability.

Chris Adams: Okay, so that's something we'll touch on both today. We'll talk about some of the external impacts and some of the internal impacts. We're going to start with something quite easy first because, well, why not? I'm just going to ask each of the people here to kind of point to maybe one project that they've seen that's using ML in quite an interesting fashion to ideally come up with some kind of measurable win. Well, if there was one project you'd actually look to that you think is kind of embodying these ideas of like green AI something which is really helping us essentially face some of the challenges, maybe you could tell us about what's catching your eye at the moment or what you'd look at.

Will Buchanan: I've been thinking a lot about natural capital recently, and I learned about a startup called Pachama, which combines remote sensing data with machine learning to help measure and monitor the carbon stored in a forest. I think it's really, really valuable because they're providing verification and insurance of carbon credits at scale. They've protected about a million hectares of forest. That's really when you have IoT and remote sensing and machine learning combining to help nature restore itself.

Chris Adams: Okay, cool. So if I understand that, they're using satellites to basically track forests and track deforestation. Is that the idea that they're doing there? 

Will Buchanan: Yes, Yes, and and also to verify the amount of carbon that a forest can sequester.

Chris Adams: Okay, cool. All right. I know there's a few other projects related to this. If I just hand over to Abhishek, can you let us know what's caught your eyes recently, and then we'll see what other projects come out of this.

Abhishek Gupta: Yeah, absolutely. I think one of the projects, and I don't know, I mean, what the measurable impact has been so far. In fact, it's, it's something that's come out of MILA, which is, or, you know, called the Montreal Institute for Learning Algorithms, which is Dr. Bengio's lab in, in Montreal. In fact, one of the people who led that project is a part of Climate Change AI as well, who I'm sure Lynn can talk more about too, which is SASA. And she's done this project called This Climate Does Not Exist, which I think was a fascinating use of machine learning to visualize the impact climate change will have on, you know, places around you in a very arresting and visually capturing fashion, which I think when we think about.

What impact climate change is going to have around us. Sometimes it, it feels quite distant because it's a, it's a slow rolling thing that's coming our way. And this puts it in, in, in a way that's quite immediate, quite visually arresting. And I think stirs people to action. I, as I said, I'm not sure what the measurable impact of that has been yet, but I, I certainly feel that those are the kinds of creative users of AI when we want to galvanize people into action around climate change.

Lynn Kaack: I'm happy to also talk about a application, which is also kind of difficult in terms of measuring impact, but I think it's an another interesting component of what AI can do. And this is something that researchers at the Austrian Institute of Technology do on a project called Infrared. And they use machine learning to help design new districts and cities.

And especially at the moment in many countries, a lot of new urban districts are being built and how we build these has a huge impact on energy consumption in cities, both in terms of transportation, but also how buildings are heated or cooled. And by the use of machine learning, they can drastically improve design choices because now they can approximate their very computationally heavy models and run them much faster. which means that they can also have more runs and can try out more design configurations. So this is a rather indirect application, but it has huge implications also on emissions for, for many decades to come.

Chris Adams: So essentially it's using of housing policy as climate policy there, because there's just a huge amount of emissions built into how people live and whether they need to drive everywhere in a car and stuff like that. That's some of the stuff that it's doing and making that part easier?

Lynn Kaack: So it's not really looking at housing policy, but it's looking at how districts are designed. So they take a group of, of houses, like if the new district is to be built and then they simulate the wind flow going through the cities, which are very expensive simulation models. they take the outputs of their model and approximate it with a machine learning model, which makes it much, much faster.

So from hours or days, you go to milliseconds or below seconds for one run. And then you can try out different design configurations and understand better how The built infrastructure affects natural cooling, for example, in cities or walkability, energy impacts generally of the microclimate on, on the built environment.

Chris Adams: Wow, I had no idea that was, it was actually possible. That's really, really cool.

Will Buchanan: That's very cool. That's similar to generative design.

Chris Adams: design. This is a phrase I haven't heard actually, Will. Maybe you could elucidate or share something there, actually. 

Will Buchanan: It's similar similar to to some software that Autodesk has built where you can try out many different iterations of a design and come up with optimum solutions. I think it's really cool that you're just consolidating it and running these models more efficiently.

Chris Adams: cool. And that's a bit like following, say, a fitness function saying, I have a chair, or, you know, I want to have something that works like a chair. It needs four legs and a seating pattern. Then it essentially comes up with some of the designs or iterates through some of the possibilities, something like that?

Will Buchanan: Exactly. 

Chris Adams: Oh, wow. Okay. That's cool. All right. And so we've spoken about AI, and there's a few exciting, interesting ones that we can add to the show notes and list from, and for people to look into and see how that might relate to what they do, I suppose. I want to ask a little bit about measuring impact from these projects, because quite a few different ways that you can actually measure impact here.

And in many times, it can be quite a difficult thing to kind of pin down. And this is continually a thing that's come up. When I know that. So, people have tried to come up with specs like the Software Carbon Intensity, and I'm sure Abhishek, you've had some experiences here. Will, you've mentioned a little bit about, um, Actually measuring impact internally, and it sounds like you've just had to do a bunch of this work on the ML team right now and expose some of these numbers to people consuming these services in the first place. Could you talk about some of that part a bit, perhaps?

Will Buchanan: Certainly. And so, as I mentioned, we have shipped energy consumption metrics for both training and inference within Azure machine learning. And that's really complex when you think of the infrastructure required to just report that,but that doesn't necessarily account for the additional power that's consumed in the data center, such as the idle power for devices or for the utilization of your servers.

There's so many different factors there. So you really, you could encounter scope creep when you come to your measurement methodology. So it's really necessary to put boundaries around that.

Chris Adams: and when you use the term boundaries here, you're saying, I'm going to measure the environmental impact of the servers, but not the environmental impact of building the building to put the servers in. Is that the idea of when you're referring to a boundary here? Yeah.

Will Buchanan: Yes, that's a great example.

Chris Adams: Okay. All right. I think this is actually something we've come across quite a few times in other places as well, actually, because maybe it's worth asking about this kind of boundary issue that we have had here, because automatically that sounds complicated here.

And I know that Abhishek, you've had some issues at your end as well with defining this style for deciding what's in or out, because I think this is one thing that we've had to explicitly do for the software carbon intensity spec, right?

Abhishek Gupta: Exactly. And, and I think when we talk about boundaries, it's, it's, it's trying to get a sense for what are the actual pieces that are consumed, right? From an operational standpoint, from an embodied emission standpoint, and how you make those, you know, allocations across you know, what, what your system is consuming.

And I use the word system because I think again, when we talk about software, we're not just talking about a specific piece, but we're talking about really everything that it touches, be that, you know, network, be that bandwidth consumption, be that, you know, as, as Will was saying, idle power. Even when we're looking at cloud computing, it becomes even more complicated when you have your pieces of software that are sharing tenancy across the pieces of hardware and how different consumers are perhaps sharing that piece of hardware with you and thinking about whether you've booked the resource ahead of time or not, whether it's hot or cold in terms of its availability and what implications that has.

I mean, there are so many different facets to it. And each of those decisions, what I want to highlight here is that it comes with a trade off, right? So we also don't have any standards in terms of how we should go about measuring that and what should be included, what shouldn't be included. And so the way people report out these numbers today also doesn't really make it actionable for folks who are consuming or who want to consume these reports, these metrics in taking decisions as to, you know, whether something is green or not.

And I think that's one of the places that The Software Carbon Intensity Specification is trying to help folks is to help standardize it first and foremost, but also to make it actionable so that if you're someone who's environmentally conscious, you can make the right choice by being informed about what the actual impacts are.

Chris Adams: Okay, this is question that I'm curious here, because so far we've only been speaking internally about, okay, what is the environmental impact of IT itself, like it's direct emissions. But the assumption that I have here is that there are ways we might talk about the impact that it has on the outside world in terms of activity we're speeding up or accelerating or supporting there.

Is that the only issue that we need to think about, or are there any other things to take into account about like this system boundary part that we've just been talking 

Lynn Kaack: Yeah. So the system effects are really important to look at and to consider. Maybe just to give an example, like if you use machine learning in, let's say the oil and gas sector to make a small part of the operation is more energy efficient, that first sight looks like something that could be considered sustainable and green, but you also have to realize that often then you are reducing costs as well, and that might change the way that oil and gas in this particular example is competitive, or the particular company is competitive, and that actually might shift also how much oil and gas we are able to use in the short run, how the price is changed.

So, these indirect effects can actually then have much larger impacts than the immediate effects of such an application. So drawing boundaries is really important and also opening this up to to have a broader system level view and really try to understand how does the technology also change then, then to larger consumption and production patterns. It's important.

Chris Adams: So if I understand that correctly, that's talking almost like the consequences of an intervention that we might make here. So even though we've might have reduced the emissions of say the drilling part by putting a wind turbine on an oil rig, for example, that might change the economics and make people more likely to use the oil, in which many cases they might burn, for example, or stuff like that.

Is that basically what you're saying there?

Lynn Kaack: Essentially what I'm saying is that efficiency improvements in particular, and often they can be done with data science or with machine learning or AI systems. They often come with cost reductions and then those cost reductions do something and change something. And often this is also considered under rebound effects, but it's not only rebound effects.

So there's systemic, the system level impacts that come from more small scale applications that need to be considered.

Will Buchanan: That's such a good good point, and I think I've also heard it called Jevon's paradox, too.

Chris Adams: Yes. Jevons paradox. This is stuff from like the 1800s with steam engines, right? Like my understanding of the Jevons paradox was when people had steam engines and they made steam engines more efficient, this led to people basically burning more coal because it suddenly became more accessible to more people and you end up using them in a greater number of factories.

So there's a kind of rebound, I think, that we need to take into account. This is something I think has been quite difficult to actually capture with existing ways of tracking the environment impact of particular projects. We have like an idea of, say, an attribution based approach and a consequence based approach. And maybe it's worth actually talking about here, about how Some of the complexities we might need to wrestle with when you're designing a system here. I mean, Abhishek, I think this was one of the things that was an early decision with the software carbon intensity part to not try to have an attribution approach versus a marginal approach. And if we're not diving too deeply into jargon here, maybe you might be able to kind of share a bit more information on that part there, because It sounds like it's worth expanding or explaining to people, to the audience, a bit better here.

Abhishek Gupta: Indeed, you know, the reason for making that choice was again, our emphasis on being action oriented, right? So as we had started to develop the software carbon intensity specification, one of the early debates that we had to wrestle with and, and, you know, Will and Will was of course a crucial part of that as well as whether folks who were a part of the standards working group was figuring out how.

For example, the GHG way of going about doing that, you know, accounting doesn't really translate all that well for software systems and how perhaps adopting a slightly different approach would lead to more better. More actionable outcomes for the folks who want to use this ultimately to change behavior because, you know, without getting into specifics of, you know, what marginal is and what consequential approaches are, and, and if we want, I mean, I'm, I'm sure you know, will, would, would be happy to dive into all of those details as would I.

But the, the, the thing that we were seeing was that we're doing all of this great work around, you know, talking about scope one, two, three emissions, et cetera, but it's not really helping to drive behavior change. And that's really the crux of all of this, right, is that we're not just doing all of this accounting to produce reports and to, you know, spill ink, but it's to concretely drive change in behavior.

And that's where we found that adopting a consequential, adopting a marginal approach actually helped make it more actionable. And this was coming from folks who are a part of the standards working group, including Will and myself, who are practitioners, who, who are itching to get something that helps, helps us change our behavior, change our team's behaviors when it comes to building greener software, broadly speaking.

Chris Adams: Okay. So that helps with explaining the difference between a consequential approach and a marginal approach, as in the consequences of me building this thing will mean that this is more likely to happen. And if I understand it, the GHG protocol that you mentioned, which is the greenhouse gas protocol, and this scoped emissions approach, this is the kind of standard way that an organization might report It's kind of climate responsibility, as it were, when, and when you say scoped emissions, that's like scope one, which is burning burning say that's, that's emissions from fossil fuels burned on site or in your car.

For example, scope two is electricity and scope three is your supply chain. If I understand what you're saying, there's like a kind of gap there that doesn't account for the impacts of this, perhaps. I mean, some people who've referred to this as scope zero or scope four, which might be, what are the impacts an organization is happening to. Essentially, we mentioned before, do something around this systemic change, or as Lynn mentioned, like this is changing the price of a particular commodity to make it more likely to be used or less likely to be used. And this is what I understand the SCI is actually trying to do. It's trying to address some of this consequential approach, because the current approach doesn't capture all of the impacts an organization might actually have at the moment. Right?

Will Buchanan: Yeah, that's a good summary. One challenge that I have noticed is that until it's required in reporting structures like the greenhouse gas protocol, then organizations don't have an incentive to really take the action that they need to avoid climate disaster. Something I encounter on a daily basis, and I think broadly we need to bring this into the public discourse.

Chris Adams: I think you're right. I think it's worth actually, Lynn, I think that when I've seen some of the work that you've done, you've done previously, this is something that's come into some of the briefings that I think that you've shared previously with the climate change AI work and some of the policy briefings for governments as well. Is there something that you might be able to add on here?

Lynn Kaack: Yeah, so something that comes to mind is, for example, like a concrete legislation that's currently being developed is the EU AI Act. And that's a place where, for the first time, AI systems are being regulated also that scale. And climate change almost didn't play a role for that regulation in the first draft.

So here it's also really evident that if we don't write in climate change now as a criterion for evaluating AI systems. It will probably be ignored for the next years to come. So the way that legislation works is by classifying certain AI systems as high risk and also just outright banning some other systems, but as high risk systems could, the original legislation stood, weren't really explicitly classified as high risk, even if they had like a huge environmental or climate change impact. And that's that I talked about a lot with policymakers and trying to encourage them to more explicitly make environmental factors and climate change a factor for evaluating systems. So that'd be a very concrete case where making climate change more explicit in the AI context is important also in terms of legislation.

Abhishek Gupta: So there's, there's a lot to be said about the EU AI Act, right? And a ton of ink has been spilled everywhere, I think, as, as, you know, it's, it's, it's, it's called the Brussels effect for a reason, right? Where the, whatever happens in the EU is taken as gospel and sort of spread across the world, which I think, as already Lynn has pointed out there, It's not, it's not perfect, right?

I think one of the things that I've seen being particularly problematic is, is the rigid categorization of what, you know, high risk use cases are and, and whether the EU AI act, as we'll see, hopefully with some, with some you know, revisions that are coming down the pipe is whether we'll have the ability to add new categories and, and not just update subcategories within the existing identified high risk categories.

And I think that's where things like considerations for environmental impacts and really tying that to this you know, societal impacts of AI, where we're talking about bias, privacy, and all the other areas, is going to be particularly important because we need multiple levers to, to try to account for, or to push on getting people to consider the environmental impacts.

And given that there is such a great momentum already in terms of privacy considerations, bias considerations. I think now is the time where we really push hard to make environmental considerations an equally first class citizen when it comes to, you know, thinking about the societal impacts of AI.

Will Buchanan: This is something I'm incredibly passionate about. I think needs to encompass the full scope of harms that are caused by an AI system. So that could be the hidden environmental impacts of either the development or the application. The application could vastly outweigh the good that you're doing, even just expanding oil and gas production by a certain percentage amount. I think it just must account for all of the harms for both ecosystems and people.

Chris Adams: And there's this idea of like a risk thing. Does this categorization actually include this stuff right now? What counts as a high risk use case, for example, when mentioned here?

Lynn Kaack: So I haven't seen the latest iteration. I think that there has been some update on, there's been a lot of feedback on the version that was published in April last year. I haven't seen the latest iteration. I think a lot of things changed in, yeah, in the first version, there was It's high risk systems where when, that affect personal safety, like human rights in the sense of, of personal wellbeing, but the completely overlooked environmental protection aspects aspects of human rights.

Chris Adams: Wow, that's quite a large one, especially when you take into account the human rights. Okay. We've spoken about the external impact, but I am led to believe there is also an internal impact from this as well. Like the AI has, has some direct impact that we might want to talk about as well. As I understand it, we spoke about 2 to 3 percent of emissions here, but if we know there's an external impact, why would we care about any of the internal impacts of AI we might be doing or why you might want to care about the internal impacts of AI as well, for example, like the direct emissions.

Will Buchanan: So by direct emissions, you're talking about, let's say, the scope, too, of the operational cost of the model.

Chris Adams: Yeah, there'll be things that we have, there's an external impact or there is a, we use this phrase scope 4, for example, to talk about all the other things that induce in the world. But there is a, a kind of stuff which happens inside the system boundary that we've spoken about. And presumably that's something we should be caring about as well, right? So there'll be steps that we can take to make the, the use of AI, particularly like say the, the models more efficient and more effective and more, all these parts here, this is something that we should be looking at as well, presumably right?

Will Buchanan: Totally. And so in our paper, which is going to be published, I think on Monday, we've calculated the emissions of several different models. And one of them was a 6 billion parameter transformer model. And the operational carbon footprint was equivalent to about a rail car of coal. And that's just for training. So it's really imperative that we address this and provide transparency this

Lynn Kaack: Is that Is that for developing a model or for training at once? I mean, is that with grid search, architecture search?

Will Buchanan: For for a single training run. So it does not account for sweeps or deployments.

Chris Adams: All right, so there's a, there's some language that we haven't heard for here, so, but maybe it might be worth asking, maybe Will, could you maybe talk about, just briefly, you said a rail car full of coal, and I don't actually know what that is, I mean, in metric terms, what does that look like?

Will Buchanan: A hundred million grams. I don't have the conversion handy, but we took the U.S EPA greenhouse gas equivalencies. And I should add the methodology that we applied was the Green Software Foundation's SCI. So we calculated the energy consumed by the model and multiplied it by the carbon intensity of the grid that powers that data center.

Chris Adams: Okay, cool, and that was per training run? So that wasn't the, in the, the equation of the entire model, is that correct?

Will Buchanan: Correct. 

Abhishek Gupta: That's the other interesting part as well, right? When you're thinking about the life cycle is, or life cycle of the model, I should say, because life cycle has multiple meanings here, which is that once that model is out there, what are the inference costs, right? And are we, if this is something that's going to be used you know, hundreds, thousands, tens of thousands of times, if it's something, you know, if it's, if it's a large model that's, you know, now being used as a pre trained model and is going to be fine tuned on by, by other folks downstream, are we able to then, you know, talk about amortization of that cost across all of those use cases?

And again, I think what becomes interesting and, and is how do we account for that stuff as well, right? Because we, we don't have complete visibility on that as well. And, and I know Lynn's nodding here because her paper that's, I think coming out, being released in an hour and a half, actually the embargo gets lifted on our paper, actually talks about some of those system level impacts.

So maybe, maybe Lynn, you want to chime in and talk a little bit about that as well?

Lynn Kaack: Yeah, thank you so much. Exactly. So I think what's a crucial number that we're currently still missing is not what is emitted from a single model in a well known setting, but what is emitted overall from applying machine learning? So what are the usage patterns and practice? Like how often do people develop models from scratch?

How often do they train or retrain them? People I mean, of course, organizations and typically large organizations and companies. And how do they perform inference on how much data, how frequently? And there are some numbers out there from Facebook and Google and in their large scale applications actually inference outweighs their training and development costs in terms of greenhouse gas emissions.

So inference might become a bigger share depending on the application. So we really need to understand better how machine learning is being used in practice also to understand the direct emissions that come from it.

Chris Adams: An inference is a use of a model once it's in the wild. Is that what an inference is in this case? So there's an environment, so you could think of the making part, and then there is the usage part from the inference, right? So is that how that part works?

Lynn Kaack: Exactly. So if you use a model on a data point, we call that inference. So you feed in the data and it gives you a result. Then training means you sort of train a single configuration of the model once on your training data set, and then development is what I refer to as if you search over different configurations of the model.

So there are lots of hyperparameters that you can adjust to achieve better performance. And if new models are being developed, then there's an extensive search over those hyperparameters and architecture configurations that then of course gets really energy intensive because we are training the model thousands of times essentially.

Will Buchanan: Yeah, one of the figures that really resonated with me, I think Nvidia posted on their blog that inferencing for about 80 to 90 percent of the carbon cost of a model. think Lynn on one of your papers, it was, Amazon had also claimed around 90 percent. So these are really non trivial costs, and I'm not aware of any framework to measure this.

Lynn Kaack: Yeah. So that Amazon number just to be clear is costs or monetary costs that came from a talk, but there are numbers now published by Google and Facebook, but they look at some applications of theirs where inference outweighs training in terms of energy consumption. They're not exact numbers. It's not entirely clear which applications those are, but there is some data at least that shows that.

And I think it just highly depends on the application that you're looking at. And sometimes, you know, you build a model and then you do inference once and you have the data set that you, and then in other types, you build a model and then you apply it billion times a day. So, of course, that can then add up to a lot more energy consumption.

Chris Adams: Wow, I didn't realize that was actually an issue, because most of the numbers I've seen have been focusing on the training part. So, Will, I think this is something we spoke about before, that training, there's a kind of trend in the use, in the energy use from training already. Is this something, because I've seen figures from OpenAI, but my assumption was that basically computers are generally getting more efficient. About twice as efficient every two years, or so with like Moore's Law or Kumi's Law or things like that. But if you're seeing an uptick in usage here, is, does that mean that they're staying about the same? Or is there, is there a trend that we should be taking into account there?

Will Buchanan: So I think the computational costs of training have been doubling every 3.4 months or so, and so I think the trend is only accelerating. The models are just getting larger and larger, and you've got, I think, GPT 3 is one of the largest ones around at this point. We might challenge Moore's Law.

Chris Adams: Okay. So if Moore's Law is doubling every once every two years, I mean, what is the impact of doubling every 3.4 months? I mean, over a few years, what does that work out to be? Because I don't think I could do the exponential numbers, the exponential math, but it sounds like it's, it sounds like a pretty big number, basically, if something is doubling on every three or four months, right?

Will Buchanan: I also don't have the math handy, but I think it's important to note here, and Abhishek was talking about this earlier, models are very flexible, so you can train them once and then provide some fine tuning or transfer learning approach on top of them, and then repurpose these models for a number of different applications. And then you can even compress them, let's say using ONNX Runtime. You can be very efficient. You can really amortize the cost of that model.

Abhishek Gupta: So yeah, just building on Will's point, there's a lot of work on quantizing the weights of a trained network, applying distillation approaches, using teacher student model approaches that actually helps to shrink down the model quite a bit, especially with the whole push for TinyML, trying to shrink down models so that they can be deployed on edge devices has been something that's helped to manage to a great extent, the computational impacts.

One of the other things that I wanted to highlight as, as, you know, Will was talking about more models getting larger is, there's this almost fetish in the world today to continuously scale and keep pushing for ever larger models and in chasing SOTA, as they would say, so chasing state of the art, you know, which is great for academic publications where you get to show, "Hey, I improved state of the art performance on this benchmark data set by 0.5 percent or whatever," right? And in performance, I think what's being ignored is that that has a tremendous, tremendous computational cost. In fact, one of the hidden costs that I think doesn't get talked about enough is there's this statistic out there that, you know, 90 percent of the models don't make it into production.

And that kind of relates to things like, you know, neural architecture search and, you know, hyper parameter tuning, where you're constantly trying to refine a model to achieve better performance. A lot of that actually goes to waste because that stuff doesn't make it into production. So it's actually not even used.

And so there's a whole bunch of computational expenditure that is done that actually never sees the light of day, never becomes useful. That obviously has environmental impacts, right? Because of the operational and embodied carbon impacts, but none of that actually gets talked about, reported, documented anywhere because, well, who wants to know that, hey, I trained, you know, 73 different, you know, combinations to get to where I'm at.

You just talk about the final results.

Chris Adams: Okay, let's say that if you don't want to go down one of those rabbit holes, what should you be using or where would you start if you wanted to start applying some of these ideas about greener AI in your work on a daily basis? Does anyone have anything that they would lead with, for example?

Will Buchanan: Bigger is not always better. Sometimes you really should choose the right tool for the job. We've had some really great graduate student projects from the University of Washington's Information School, and they built some case studies and samples around green AI. As an example, a project led by Daniel Chin was comparing a sparse or a dense model of a green AI model to a dense model for an anomaly detection setting.

And they found that using sparse, meaning less trees and a shallow, smaller depth per tree, random forest would save a massive amount of carbon and provide the equivalent accuracy. So I think it saved about 98 percent in terms of the monetary cost and energy.

Chris Adams: Okay, wow, that's bigger than I was expecting. What would you say to people if they're in production, they're trying to do something now?

Lynn Kaack: I think a big goal should be to not only develop more energy efficient machine learning models, but then also ensure that those are actually being used. And surprisingly, even sometimes within the same company, certain model developments are not being passed on to other parts of the company. So really trying to develop standard models that then are also being used in practice is important.

So interoperability of energy efficient machine learning models.

Chris Adams: If people, someone does want to look at this stuff, and they do want to apply some of these ideas, you spoke a little bit about using some other models. would you suggest people look if they wanted to operationalize some of the kind of wins or some of the better ways to make green software greener, for example?

I realize you've got a paper coming out and you work on this day to day. So yeah, what would you point us to?

Lynn Kaack: So, I mean, as I understand, there's a lot of ongoing research in the machine learning community for more energy efficient machine learning. So I don't have any names on top of my head in terms of workshops or community resources where one can see what are the most energy efficient model types. For a specific application.

I know that there are some very comprehensive papers also that summarize all the different research approaches that are being taken, but I would encourage if you are looking for using like a deep learning model of some kind, just inform yourself quickly if there's also a leaner version of it. So much of the like widely used models like BERT, for example, smaller versions that can almost do the same thing.

And maybe your performance doesn't suffer much. If you're using a much lighter model.

Chris Adams: Okay, so light up models and looking around what we have there. And Will, is there a paper or a source you might point to?

Will Buchanan: I was actually going to talk about the Carbon Aware paper that we're about to publish. I think that's a slightly different track.

Chris Adams: That's up next week, right? So that'll be the 13th or 14th of June. That's when that'll be visible, correct?

Will Buchanan: Exactly. 

Chris Adams: Okay, cool. All right, then. There's a load more that we could dive into. We've got copious, copious, copious show notes here. So what I'm gonna do is I'm gonna say thank you everyone for coming in and sharing your wisdom and your experiences with us, and hopefully we'll have more conversations about green software in future. Thank you folks.

Asim Hussain: Hey everyone, thanks for listening. Just a reminder to follow Environment Variables on Apple Podcasts, Spotify, Google Podcasts, or wherever you get your podcasts. And please do leave a rating and review if you like what we're doing. It helps other people discover the show and of course we want more listeners.

find out more about the Green Software Foundation, please visit greensoftware. foundation. Thanks again and see you in the next episode.