Intentional AI Integration

“The potential is boundless, but it doesn’t come automatically; it comes intentionally.”

–Paula Goldman

What you will learn

• Redefining ethics as trust in technology

• Designing AI with intentional human oversight

• Building justifiable trust through testing and safeguards

• Balancing automation with uniquely human tasks

• Starting small with minimum viable AI governance

• Involving diverse voices in ethical AI decisions

• Envisioning AI that enhances human connection and creativity

Episode Resources

Transcript

Ross Dawson: Paula, it is fantastic to have you on the show.

Paula Goldman: Oh, I’m so excited to have this conversation with you, Ross.

Ross: So you have a title which includes your the chief of ethical and humane use. So what is humane use of technology and AI?

Paula: Well, it’s interesting, because Salesforce created this Office of Ethical and Humane Use of Technology around seven years ago, and that was kind of before this current wave of AI. But it was with this—I don’t want to say, premonition—this recognition that as technology advances, we need to be asking ourselves sophisticated questions about how we design it and how we deploy it, and how we make sure it’s having its intended outcome, how we avoid unintended harm, how we bring in the views of different stakeholders, how we’re transparent about that process.

So that’s really the intention behind the office.

Ross: Well, we’ll come back to that, because I just—humane and humanity is important. So ethics is the other part of your role. Most people say ethics are, let’s work out what we shouldn’t do. But of course, ethics is also about having a positive impact, not just avoiding the negative impact.

So how do you frame this—how it is we can build technologies and implement technologies in ways that have a net benefit, as opposed to just not avoiding the negatives?

Paula: Well, I love this question. I love it a lot because one of my secrets is that I don’t love the word ethics to describe our work. Not that—it’s very appropriate—but the word I like much more than that is trust, trustworthy technology.

So what happens when you build—especially given how quickly AI is evolving, how sometimes it’s hard for people to understand what’s going on underneath the hood and so on—how do you design technology that people understand how it works? They know how to get the best from it, they know where it might go wrong and what safeguards they should implement, and so on.

When you frame this exercise like that, it becomes a source of innovation. It becomes a design constraint that breeds all kinds of really cool, what we call trust patterns in our technology—innovations like we have a set of safeguards, customizable safeguards for our customers, that we call our trust layer.

And this is one of our differentiators as we go to market. It’s things that allow people—features that allow people—to protect the privacy of their data, or make sure that the tone of the output from the AI remains on brand, or look out for accuracy and tune the accuracy of the responses, and so on.

So when you think about it like that, it becomes much less of this mental image of a group of people off in the corner asking lofty questions, and much more of an all-of-company exercise where we’re asking deeply with our customers: How do we get this technology to work in a way that really benefits everyone?

Ross: That’s fantastic. Actually, I just created a little framework around trust in AI adoption. So it’s like trust that I can use this effectively, trust that others around me will use it well in teams, trust that my leaders will use it in appropriate ways, trust from customers, trust in the AI. And in many ways, everything’s about trust.

Because a lot of people don’t trust AI, possibly justifiably in some domains. So I’d love to dig a little bit into how it is you frame and architect that ability—this ability to have justifiable trust.

Paula: Do you mean the justifiable trust from the customers, the end users?

Ross: Well, I think at all those layers. I think these are all important, but that’s a critical one.

Paula: Yeah, I think a lot of it is about—I actually think about our work as sort of having two different levels to it. One is the objective function of reviewing a product.

We do something called adversarial testing, where we’ll take, let’s say, an AI agent that’s meant for customer service, and we’ll try all these different variations on it to see if we can get it to say things that it shouldn’t say. We’ll involve our employees in that, and we’ll take people from lots of diverse backgrounds and say, “Hey, try to break this product.”

And we measure: How is it performing, and what are the improvements that we can make to get it to perform? That’s a big part of trust, right?

When we think about AI, is the product doing what it says it should do? Is it doing what we’re asking it to do? And with a non-deterministic technology like this wave of AI, that’s a very important question.

You want to harness the creative potential of AI—its ability to generate and communicate in human-sounding terms—but also marry it to accuracy and outcomes that are more predictable.

So that’s one side of it. But the second side, the second part of the job, is really a culture job. It’s about listening—listening to our employees, our customers, our end users.

It’s about participating in these multi-stakeholder groups. I was a member of the National AI Advisory Committee in the US. In many jurisdictions, we’re part of these multidisciplinary forums where people are bringing up different concerns about AI, whether that’s about how work is changing or particular questions about privacy.

We integrate those questions into the work itself and integrate solutions into the work itself, but really have it be so that everyone owns it—so that the solutions are generated by everyone.

That’s, I think, the cultural part of it. I’m an anthropologist by training, and I always think about it like that. If you want technology to serve people, people have to be involved in determining those goals.

Ross: Which goes to the next point. This is Humans Plus AI podcast, and I’ve heard you use the term “human at the helm.” AI capabilities are pretty damn good, and they’re getting better all the time. So how do we architect that humans remain at the helm?

Paula: We coined the phrase “human at the helm” a couple of years ago, as we realized there were these older frameworks about having a human in the loop for consequential decision making.

Back with machine learning, you have predictions or recommendations on a consequential decision. You want a human to take responsibility for that decision and exercise oversight.

We realized that with agentic AI, and with AI increasingly empowered to take tasks autonomously—not just make a recommendation, but carry out a task from start to finish—we needed a new way of conceptualizing how people work alongside AI but remain in control.

Know how to get the right outcomes from AI. Know what to ask for and what not to ask for, and what tasks should remain uniquely human.

I think it’s an ever-evolving framework. I know you’re deeply looking at those sets of questions. I honestly think, going back to the ethics exchange, that’s one of the most important ethical questions of our time: How do people work alongside AI? How do we implement AI in work in a way that keeps people at the center of that?

So that’s what we’re doing, discipline by discipline. For example, going back to AI in customer service—AI is very good at answering questions that are routine, that have been answered a number of times before, like “Where’s my order?” or “Where’s my return?” or “Have I gotten the money for my return?”

When it comes to unusual circumstances or emotionally challenging circumstances for the customer, human touch can make the world of difference between a terrible interaction, an interaction that does maybe okay, or an interaction that leaves a lasting impression and causes a customer to go talk to 10 other customers about how important your company is to them.

That is the kind of marriage we’re talking about between the capabilities of AI and the capabilities of people. It’s a very simple example, but we see them across every single discipline.

I think the more clear we are about how these combinations work and how people can—we have a function called “command center,” a feature that allows people to see exactly what’s going on across hundreds of agents and millions of interactions and summarize what’s going on and find anomalies—the more that people can stay in control and understand what’s going on, the more trust they’ll have, the more they’ll use AI. And it’s sort of a virtuous loop.

Ross: Yes, absolutely. The phrase “human in the loop” kind of suggests all they do is press “approve, approve.” Whereas a reframing I heard recently, a little while ago, which I think is really lovely, is “AI in the loop”—as in, humans are there, AI is in the loop, as opposed to humans just being an afterthought.

Paula: Yeah, that makes sense, right? And it’s consistently going to be a give and take. We talk a lot about, and we could design a lot for, a key question: How and when should AI escalate to a human?

The truth is, as this matures, it’s going to be AI escalating to people, and people handing it back to AI, and back and forth and back and forth.

We need to be able to observe all of those transactions. We need to be able to know when it was actually AI making a decision versus a person.

I think these are the types of changes that are going to be transformative for businesses, but it requires really understanding the workflows. Really understanding AI is not a substitute for every human task—it’s just not—but it can do a lot of powerful things.

So where is it that you use AI to perform certain tasks? Where is it that you don’t? And what does that mean for how roles evolve in time?

Ross: So whenever an organization says, “All right, we need to get some AI in, let’s build an AI roadmap,” basically the very first thing, pretty much, is governance. So I’d love to know—Salesforce is a very unique organization, so perhaps looking more at your client organizations—what is the on-the-ground reality of saying, “Okay, we’re going to set up our AI governance”? What does that look like?

Paula: It’s an excellent question. One of the things I often say in customer conversations about this is that, I know it can sound like a cliché, but the most important thing is just to get started.

In reality, there are a number of different building blocks that are important, and none of them are necessarily going to be absolutely polished and perfect. They’re iterative.

For example, it’s very important to clean up data—especially to label sensitive data and make sure that privacy controls are in place. Data is obviously the foundation of getting good results from AI, but that is a continuous process. It’s not a once-and-done thing.

We end up talking a lot about the types of features I was mentioning earlier—that is to say, having people that really know and understand how to make sure that there isn’t unintended bias in the results of different AI applications, or ensuring and judging outputs from AI and making sure that they’re accurate and looking at measures of precision and recall.

You start small and look at the results, and as you trust the results more, you grow whatever application it is that you’re building and build from there.

The other thing I would say, and that I often talk to boards about, is: How do you have the right board-level oversight of AI?

I do think we’re unique at Salesforce—we have had a board committee that our Office of Ethical and Humane Use has reported to every quarter for years. And we use that conversation to talk about emergent issues, what’s going on, what’s a snapshot of what’s going on with AI, what are the questions that we’ve got under control, and what are these emerging questions that we really are thinking about.

Given the technology is evolving so quickly, how do we set new standards for new technologies that maybe we hadn’t dreamed of two years ago? I think that board-level conversation, that sort of bottom-up and top-down awareness of what’s going on, is also very important to governance.

Ross: I mean, are there any particular structures, as in an AI Oversight Committee, which requires a board, or anything else which you have seen?

Paula: Yeah, I think so. We have an Ethical Use Advisory Council, and I think many companies have some variant of this kind of structure where we have frontline employees, and that’s intentional. We want people who are very, very close to the work.

We have executives from many different functions of the business. We also have outside civil society groups that are contributing to bringing in concerns that are coming in from various communities around the world.

Then we bring to that council a number of items, including policies. We have an AI acceptable use policy that applies to our customers. It sort of sets a floor for what responsible use looks like, and we will kick the tires on all of these questions with this group.

We’ll kick the tires with other stakeholders, as you can imagine, depending on what question comes up. Then we’ll bring recommendations up our executive chain as we want to make changes. That’s how we came up with our AI acceptable use policy, for example.

So I do think it really is all about who’s at the table, and I think creating these big tables for lots of people to have ideas and to have questions really enriches the quality of the solution.

I’ll add one more thing there too, which is, I mentioned culture as part of the job of AI ethics, one thing that’s really important is making sure that we have channels for people to ask questions, and not just if you happen to be part of a council.

We have anonymous channels where employees can ask questions. We have Slack channels where people can ask questions. We have workflows for people to ask questions, and so on.

All of that also remains very, very important. As we get more and more technologically sophisticated, these very deeply human structures of how we listen to each other, and how we make each other feel heard, and how we acknowledge questions that come up all become even more important.

Ross: That’s fantastic. Yeah, no, it’s—well, going back to the trust point, if people feel that they are listened to, then that’s a pretty good starting point. If they don’t feel listened to, that’s not going to engender lots of trust.

But going back, you said, “You’ve got to get started on the AI journey.” All right, we’re going to plan for six months and then do something. So what is minimum viable governance? As I say, we’ve got to get started. . We do need something. What’s that minimum to be able to say? All right, we’re going to try some stuff, but we have some atleast some guidelines around it.

Paula: Yeah, that is an excellent question, and I’ll answer it off the cuff, but it’s one I want to think more about.

I think you want to know, if you’ve got a significant project going, you want to know about it. You want to know what its goal is. You want to start with some, at least elementary, measures of what’s going on with it—how is it doing, and what are the critical benchmarks, and if anything unexpected has come up from it.

It really is kind of that sort of elementary. And yet, you also want to make sure that there are certain domains of AI that have higher obligations for diligence.

So if you’re using AI, for example, to give tailored financial advice, or you’re using AI to determine who gets a job, those are things that have higher levels of scrutiny. So I would say that’s also a second thing—think about the requirements for those types of forms of AI.

But the reason I started with that first example of—and it sounds fairly elementary—but if you’re measuring success and looking out for anomalies, and being able to fine-tune that, it ends up actually being very important. You can start with some sort of basic measurements, and as you get more sophisticated with it—let’s say you’re using AI for marketing personalization—you can start with looking at marketing segments and looking at how click-through rates on particular campaigns.

Obviously, there’s a whole world of things you can do with AI agents in marketing. As you gain confidence with it, those measures become much more precise, and the measures almost evolve automatically from the work that you’re doing.

I guess I’ll add one more thing, which is that I think the culture of transparency and documentation is very important. So that, I think, is a huge part of ethics and AI—being able to say, for example, when we publish AI models, when we put them out there in the world, we have model cards, we have system documentation for all of our products.

That documentation will have all the things you expect about what the product is and what it does and how to use it, but it will also say, “Here are some of the risks, and here’s how we tested for those risks, and here are ways not to use the product, or here are best practices to try to avoid those risks.”

I think that level of transparency is what helps humans to use AI in ways that get them the results that they expect. And that’s also, I think, a very important piece of it.

Ross: So the future of work is here, yeah, moving pretty fast. And I suppose, of course, Salesforce is a leader in generative AI, and this idea of agents being able to collaborate, form teams, do an extraordinary array of tasks and capabilities.

So how do we design the use of these wonderful tools and agentic AI so that they are augmenting people, developing skills, creating more opportunities for people in this future of work?

Paula: Well, this is the question that animates me every day, and it is a question that I’m personally paying attention to. It’s a question that we’re paying attention to in our product development life cycle. It’s a question that we’re living every day as we ourselves use AI in Salesforce.

And I think it really does come down to having a shared understanding of what tasks we do ask AI to take on, and what tasks we leave to humans, and where those boundaries get made, and how people collaborate with AI.

Just to give you a couple of examples, I was thinking about one of our customers, 1-800 Accountant. This is a firm that does, amongst other things, as you would guess by its name—it helps with tax preparation, helps with financial planning, and so on. We’ve been working with them on their customer service AI.

And, particularly as tax season arises every year in the United States, you can imagine the company gets inbound requests that can exceed capacity to respond to in a quick way. AI is really able to handle some of those routine requests.

But for a number of reasons—regulatory, ethics, and otherwise—personalized financial advice is something that you want now for people to be giving. You want to make sure, to the extent that AI is augmenting people and giving that advice, you also want to make sure that it’s accurate and that it’s looking out for risks.

So 1-800 Accountant was very deliberate in creating that division of labor where the less routine requests are going to their experts to handle. And because there were those less routine requests, they have more capacity. They have less of the “Where am I? Where’s my order?” type requests, and they’re able to respond more effectively and reduce call volume by significant percentages.

There are so many stories like this. The orchestration has to be intentional. These are intentional decisions we have to make about how we prepare people to work with AI, and how we allow them to oversee personal AI agents working on their behalf—whether that’s drafting emails to customers, or orchestrating a whole marketing campaign.

It’s really that intentionality around it, and how people oversee it, and what jobs and roles and tasks within those roles are reserved for people, that I think is one of the most crucial questions of our day.

Ross: So always, always get very practical. So, where’s the intentionality come from? As in, so who’s who in the organization?

So intention, I think, is—I always talk—is a really critical part of how it is we create this positive future. So intentionality, certainly. So where does that intentionality reside? Is there the chief AI officer or chief ethics officer? Or is it people who are delegated to have the capabilities to implement their intention around how they use the agents? How do we cascade?

Paula: It is interesting because you use the word “cascade,” and that’s what I was going to say—yes, right.

So a chief AI officer, head of IT traditionally, would be kind of thinking about those big bets on AI and setting goals around them, and making the technology available to customize for different groups and different departments within an organization.

But increasingly, we’re seeing the sort of matrix approach where people that are closer to how the work is getting done are in the position to say, “Here’s how it can help me, and here’s how it shouldn’t help me.”

It’s interesting. I was on a business trip last year, and I was in the UK, and I met an IT person, an executive from a local town council, and he was telling me the story about how he began trying to implement—for their social workers—a note-taking tool that would basically take notes during a site visit with a person in need and then summarize the conversation so that social workers didn’t have to go back to the office to type up the notes, which was taking hours, and they could just go home.

And he told me that the team rejected it. They wanted to go back to the office. Why? Because they wanted to talk to their colleagues, and they wanted to talk to their colleagues to trade notes, because they didn’t want to burden their spouses with this.

And so, because they missed how humans actually wanted to use the technology, they had to put the whole experiment on hold for a while. But a small tweak—observing what is this flow of work, and which are the tasks we should and shouldn’t delegate, and how do we make it so that it actually makes people better at their jobs—would have made the difference to a successful implementation.

And that is not rocket science, but it can sometimes be overlooked because it’s not just about the technology; it’s about the technology plus the people working together that gets the best results.

Ross: Yeah, so essentially giving people the knowledge and the power then to be able to make their choices or use it the way which they see fit.

Paula: I think so. And maybe there’s a balance, as there’s AI that can help us individually in our roles, and there’s AI that helps us as teams do better work. And I think there’s a place for both of those things, but you want the people that are closer to the ground to be able to impact how those technologies get deployed.

Ross: So to round out—a nice big question—what is the positive potential for humans plus AI?

Paula: Well, in my mind, I think it’s about enhancing human connection and enhancing human creativity and innovation—I’m thinking mostly in a work context. Right?

There’s a number of other societal potential benefits—there’s deep science, which I’m deeply excited about, and medicine, and so on. In a work context, it’s about being able to free people up so that they are able to make these deeper connections with each other.

And I think about that. I think about stories that are sort of way predate AI. But I also think about, for example, in medicine, I think about the doctor that doesn’t have to take notes, but is able to look someone in the eye and understand if maybe there’s something that’s not being said or a question that needs to be asked.

Or I think about—well, you and I have talked a bunch, Ross, about strategy—and I think about how AI would enable us to ask better, deeper, more incisive questions about where organizations should be going and to see unmet needs.

Let’s say we’re running a nonprofit and see unmet needs in our community and find better solutions around it. I mean, it’s really—it’s almost one of the hardest, because it’s so boundless. The potential is boundless, but it doesn’t come automatically; it comes intentionally.

And I think I might round out my answer by saying, when I first joined Salesforce, one of the things that attracted me to the company was its long history of being a values-led company. And Marc Benioff, our CEO, often says, “Technology is not good or bad, it’s what you do with it that matters.”

And here we are at the inflection point of one big technology leap forward with this wave of AI. And yet that adage, I think, remains true. It’s what we do with it, and that is why the intentionality that we talked about is so deeply important.

Ross: That’s fantastic. Thank you. Paula, is there anywhere people can go to find out more about your work or Salesforce’s framing of ethics?

Paula: Yeah, for sure. Well, we have a site—the Office of Ethical and Humane Use site for Salesforce—and you can just Google it, and there’s lots of material there to look for. You could ChatGPT it, you can do whatever you’d like, but it’s all there for the taking.

Ross: Fantastic. Thank you so much for your time and your insights and your work in creating a positive future for humans plus AI.

Paula: Ross, thank you for your leadership in this domain.