Validating AI outputs and spotting hallucinations

Today on the Salesforce Admins Podcast, we talk to Denise Carbone, Director of Delivery at ImagineCRM. Join us as we chat about her journey from business analyst to external consultant admin and why AI makes BA skills even more valuable.

You should subscribe for the full episode, but here are a few takeaways from our conversation with Denise Carbone.

Why business analyst skills are so important for AI

Denise was working as a business analyst when she was first handed the keys to a Salesforce org. She remembers getting a plaque in the mail for being one of the first 500 people to become Salesforce certified.

Obviously we’ve come a long way since then, but the skills needed to understand a business problem and map out the requirements for a solution haven’t changed. As Denise explains, you have to be “process first, technology second.” I sat down with her for this episode to find out how these BA skills are even more important with Agentforce.

The power of asking why

In order to really do your job well, an admin needs to be more than just an order taker. But if you’re spending your time running through a list of requests without having conversations with users and really understanding the business process, well, that’s a tough row to hoe.

Instead, you need to ask why. That’s where your business analyst skills come into play. You need to have a full understanding of how things currently work, where they could be improved, and who cares about it the most.

Admins in the age of AI

Another key business analyst skill is change management. You may have built the coolest solution in the world, but how do you get folks to actually use it?

As Denise explains, if you want your solution to be adopted, you need to make it adaptive. As you’re building, you need to keep going back to your users for input. If they feel like their feedback is a part of the process, they’ll have ownership over the results.

While Agentforce has greatly expanded what the platform can do, it’s still just technology you’re using to help solve a business problem. As long as you understand the why behind a request, you can build solutions that transform your organization.

Be sure to listen to the full episode for more from Denise on business analyst best practices and the importance of establishing AI governance policies. And don’t forget to subscribe to the Salesforce Admins Podcast to catch us every Thursday.

Podcast swag

• Salesforce Admins on the Trailhead Store

Learn more

• Salesforce Admins Blog: Why Every Admin Should Adopt a Consulting Mindset
• Denise’s Dreamforce Session: Why Admin & BA Skills Matter More Than Ever in the Age of AI
• Admin Trailblazers Community Group

Social

• Denise on LinkedIn
• Salesforce Admins on LinkedIn
• Salesforce Admins on X
• Mike on Bluesky social
• Mike on Threads
• Mike on X

Full show transcript

Mike:
This week on the Salesforce Admins podcast, I’m joined by Denise Carbone, longtime admin, Salesforce MVP, and all-around champion of business analysis. We unpack her journey from BA to external admin through the power of asking why, and why AI makes those foundational skills more valuable than ever. So, whether you’re like me and you’ve just figured out dependent picklists, or you’re leading delivery teams, I promise Denise’s insight on governance, process and career growth is a must-listen. So with that, let’s get Denise on the podcast. So Denise, welcome to the podcast.

Denise Carbone:
Awesome. Thanks, Mike. I’m really happy to be here.

Mike:
I know, I’ve been to a whole bunch of Chicago user groups, I can’t believe I haven’t had you on the podcast. So, I’ve tipped the scale a little bit in your favor, but tell us how you got into the ecosystem and what you do in Chicago.

Denise Carbone:
Yeah. No, thank you. Thank you. Yes, longtime listener of the podcast so happy to be here. So, I actually started in the ecosystem in 2004. I was a BA turned external admin. I was working for a technology company in Chicago, and I was bestowed the ownership of our Salesforce platform. I was a BA operations kind of girl. I’ve done work in CRM systems like Goldmine and Act!, so really old system. So, when Salesforce was presented to me I wasn’t totally intimidated. I knew it was a CRM platform, I just didn’t know much about it. I will be honest, I did not fall immediately in love with it. Later on I found out why. It was not because of the platform or the technology, it was because of lack of governance and processes. So, when I was doing reporting and operational metrics and things weren’t being presented, as the CEO was saying, as they wanted to see them, it turned out just a lack of the governance and the processes is what really caused those issues.

So, I put my BA hat back on and went back to the team, and explained some of these points we had to clean up and just do better with some guidance, and some structure and rigor around using the tool in the platform me. And that was in 2004, so in 2006 I attended my first Chicago user group meeting. Completely intimidated. I had major imposter syndrome because I really didn’t know much about the platform. Do I even belong here? But I needed to learn, I had that desire and curiosity to really build out my skill set. So, I got a little bit more comfortable in this ecosystem. And then, in 2008 I was presented an opportunity to help co-lead the group.

The user group leader was moving on and actually going to work for Salesforce at that time, and left a vacancy in Chicago. So I said, “Yes, sure.” At the time I agreed to helping co-lead the Metro Chicago area, so just anything in the city limits. And that same year I actually took the certification exam. So, in October of 2008 I was certified as an admin. So that solidified, put to rest slightly the imposter syndrome that I was carrying along, but felt really confident and good about that. And a few months later I received a plaque in the mail, I was one of the first 500 people globally to become Salesforce certified.

Mike:
Welcome to the club.

Denise Carbone:
Thank you. It was a really cool designation at the time, but yet looking back how many years later, I’m like, “Oh my God.” And today with the number of certified professionals out there and all the different types of certifications available, it’s a pretty cool little fun fact. So, I like to share that. And going into my career, and again, I was on the client side for the past, starting my career from 2004 on the client side, learning the platform, being that admin, external admin, utilizing my business analyst skills, always asking the questions of why. In 2013, I actually joined the MVP program, so super, again, imposter syndrome, why am I here? Appreciate all the peer support, love being there, but also completely intimidated.

I wasn’t sure if I belong, but it was a really cool opportunity for me to learn and grow, get to meet amazing people like yourself, Mike.

Mike:
Oh, thank you.

Denise Carbone:
And Steve Moe, actually, I was a huge fan of Steve’s. Met him through the MVP program, became friends to date. The MVP program just helped me solidify my career and just learning journeys I think is where it helped me the most. In 2015 I took the leap over to the partner side, so I needed the opportunity to really stretch my skillset. Working on the client side, you really are focused on industry specific workflows and processes, your growth opportunities on the industry side, at least for me at that time, were a little bit limited. So, going over to the partner side was really an opportunity for me to stretch my skillsets in both BA and admin, working with clients, getting to know why they’re building something and less about the how. So, one of my mantras I use all the time is it should always be process first and then technology second. I’ve seen situations where people love the shiny things and, “Oh, we can automate this and turn this on, and doesn’t this look great?” And the design of the layout is, it’s not intuitive, it’s hard to follow. It doesn’t make sense.

It’s over-automated, over-architected, over-engineered, so I always like to peel it back and ask the questions why, and get to learn more about the users, more about what we’re trying to solve for. I love that journey, and when I joined the partner side I had the opportunity to really upskill myself in both BA, administration, and then just focused on delivery. So on the partner side today, I am a director of delivery. I stepped into a leadership role in the past 10 years. So, I started as a director of delivery in 2015 with a team of one. I was able to build a team of 14 people, which means I manage a team of consultants, and I am accountable for all of our project success. Today I’m in the same industry, work for a smaller SI. We are focused in the philanthropy space and also nonprofits. Again, I support a team of consultants and I am very client-facing.

I work day-to-day with our teams, I meet with our clients, and I’m accountable for the overall project success. So, one of the things I really love about the job I do today is the coaching and the mentoring for learning and upskilling. In this space in the day and age that we’re in today, things change fast. I remember last summer seeing a preview of the Dreamforce keynote where they’re talking about agents, and being presented to us, and what do we think about this, and how does this work? I will say Salesforce has done an amazing job at providing enablement so that we can all start this learning journey, and you have to stay up to date with it because it changes, as you know, as the ecosystem knows, almost weekly. So, it’s been fantastic. But today, again, I focus on project delivery with my teams, and upskilling, and supporting, and mentoring folks.

Mike:
Wow, you said a lot in all of that answer. So, I want to dig into what you’re presenting at Dreamforce, but there was a theme in your answer that’s come through on quite a few podcasts, which is imposter syndrome. Do you feel that a lot of admins get into their role because they weren’t pursuing something in tech and they fell into it? Is that where you were feeling your imposter syndrome? And to be fair, I also want you to clarify, when you say BA you mean business analyst, right?

Denise Carbone:
Correct, yes. Business analyst.

Mike:
Okay. So I mean, you had a background in business analyst skills. That’s not a skill to scoff at. What do you think was giving you the imposter syndrome?

Denise Carbone:
That’s a great question. I think it was learning the technology and learning the platforms, I felt at the time when I fell into it, anything technology related you would be categorized as a developer, and it was very tech heavy. We used technology, we used business tools, but it was just a tool. I didn’t have to learn the behind the scenes. So, I felt a little intimidated that we’re learning the system, and I think the power of the platform, which really helped solidify my career and I think a lot of people’s career back at that time was the clicks not code focus and everything that you’re able to do declaratively. That was a huge lift for people’s career. Obviously when I started back in 2004 I wasn’t seeking out a role in the Salesforce ecosystem, and I’m using my air quotes here, because it wasn’t really a thing. We didn’t call it out at that time. It was a business tool, and it took you through a whole wave and phase of a career path.

And today in this space, I think it was even maybe eight years ago when people were really talking about career transitions, and they wanted to get into the Salesforce ecosystem and they wanted to learn Salesforce, and it was definitely a shift in that learning and it became a career back in 2004. And I don’t know about even yourself and folks, was that something people aim for? I think it started becoming a career path more towards maybe 2008, 9, 10, when certifications become more prevalent.

Mike:
I mean, I ask because I’m the similar path. I was hired to do something else, and along the way of improving business process Salesforce became vital. And so it was, “Well, I have to learn this because it’s really at the crux of improving our business process.” There’s only so much I can do to suggest sales stages and stuff, but without the technology supporting it. And so yeah, I mean, I accidentally fell into it, and within easily a couple years I was like, “This is my full-time job.”

Denise Carbone:
It does become full-time job, for sure. I think my skillset in being a business analyst, my superpower, if I’m getting asked that today what’s my superpower, is being able to break down complexity and talk in layman terms. That is a huge, again, superpower, because I have this ability to hear and kind of visualize in my head a process flow chart. Like, “Okay, we start here.” I always ask people if we’re trying to elicit requirements and gather requirements, understand a day in the life of the X persona, “Walk me through your day,” and they walk through stuff. And you start getting a visual and how you can build it, but outside of the technology is really getting to know their process and building out.

Because sometimes people say, “Oh, this is so different. You’ve probably never seen this before.” Well, some cases maybe because maybe really broken, or sometimes it’s just they never took time to step a few steps back and optimize, make things more efficient. So, I love efficiencies, I love optimization, I love the ability to break down complexities, I love the ability to speak in layman terms. I think that’s a learning that I’ve taken wholeheartedly through my journey because I can speak business speak, I don’t want to speak Salesforce speak if the recipient on the other end doesn’t speak Salesforce. So, I want to speak in a language that’s going to be comfortable for them, and then break down the complexity and really help and work to build systems that are going to bring the client along for the journey.

Mike:
I feel like we’ve gotten a hell of a preview of your Dreamforce presentation. Because I don’t know if admins always think, “Well, I’m doing business analysis,” how do you look at business analysis maybe different than somebody that’s never been formally trained?

Denise Carbone:
That’s a great question, and this will definitely be part of my session, is I always ask why. There’s been a mantra back in the admin world-

Mike:
Forever.

Denise Carbone:
Yeah, we’re not order takers. We should ask the question on why. I’ve heard people say, “You should say no,” but I would challenge that and say, “Well, ask why,” because that is the root of what a business analyst is doing is they have this curiosity mindset where they are just like, “Well, why? Why do you do this? Have you ever considered this?” Just not saying no per se. I like to say, “Yes, and,” but you give the options. Because you can do it, this, this or this, but what are the impacts of your decision if you go that route? So, I believe that admins inherently have some BA, business analyst built into their skills, it’s just maybe they don’t realize it yet, it’s unrealized. But they actually do in a way, if they’re asking what somebody wants, and they should also follow up with why.

I think that should be the starting question honestly, and then go forward and then do the build if applicable. But it’s definitely a learned skill. The other flip side to that is, I think there was, I’ve heard it before and it’s so not true, that business analysts are not just note takers. They should never be a note taker. I mean, they can do documentation, but they’re not glorified note takers. It’s definitely a skillset where you talk about a user story, or a requirement, or use cases. Those are all three very separate things. And the reason why you do some of this documentation upfront is to make sure that what we’re building as admins meet the technical requirements for the end user.

Mike:
What is your best advice for an admin to not be a note taker? Because I think when I first started, I knew improvements that needed to be done because I was in sales and I was working on sales. And so, very easily out of the box I was like, “Well, I know we need this. I know we need to track that.” And then it quickly became my user seeing the change and offering for, “Well, can you add this? Can you add this? Can you add that?” And I think what I realized early on was I really wanted to appease my users right away. Like, “Oh yeah, absolutely I can do this.” Or I remember the time, this is going to sound silly, I remember I figured out how to do dependent picklists, and let me tell you, I was like, “I can code the internet, because I can make a picklist where if you pick hot dogs it can say ‘Red Hots,’ and if you pick hamburgers it can say, ‘Rare, medium, well.'” And I just thought I was the cat’s pajamas.

Denise Carbone:
[inaudible 00:16:51] it was very powerful.

Mike:
Yeah. And then your users are like, “Well, can you make it do this?” I was like, “Yes, I can. Ha-ha. That’s a dependent picklist.” For a while all I wanted to build was dependent picklists. But how do you move from that to like, “No, I need to set a vision. I need to understand more of what’s going on”? How do you make that your approach versus piling up a bunch of tickets?

Denise Carbone:
That’s a great question. So, you want to make sure that it’s going to be adopted, that we always talk about adoption in the space, and adapted, being adaptable to things and change. So, there’s a whole change management aspect of what we do. I do believe that through the journey and empowering the admins, it’s making sure, so partnering with the end user. I think it’s on the consulting side we always say, “We’re a partner,” so we partner with our clients. It is a partnership because we can do the work but we need their input, and we want feedback. So, it’s like a continuous feedback loop. So, we have a hybrid methodology in how we do things today, but it’s definitely we will gather requirements, and get approval of these requirements, and do some build, and we build that. We break it up in sprints, maybe theme it out by certain features and functionality and present it back to the end user and get their feedback. “Does this work? Does this look good? Are we missing anything? Is there any key step that we did not capture?”

It’s really getting that buy-in as you’re doing the build so that it can easily be adopted, and you’re getting the feedback in the time where it’s more real time. So, if it’s features and functionalities that are, like you said, the bells and whistles of dependent picklists were amazing, but after a while it could become stale and they’re not using it. The goal is, I always say less is more. You don’t want to overkill something that’s not going to be used. Even today I was talking to somebody earlier about utilizing a tool to be able to scan all the metadata and see all the unused fields that were created at one point in time that are just sitting out there and untouched for a long time. So, we have to get rid of that technical debt.

So, my approach and my recommendation or suggestion is for people to really partner with who they’re building it for so that they can get the buy-in, get feedback. “Does it work? What would you like to see? Does the user interface still, is that intuitive?” I feel like with the new Lightning design pages, Mike, we grew up on Classic, remember that?

Mike:
Oh, I know. We’ve called it a few other things too, like Aloha. But yeah, Classic. I remember describing to my users, Salesforce is like the Amazon interface because at the time Amazon had those tabs. The world has gotten away from tabs on websites now.

Denise Carbone:
They have.

Mike:
We’re too fancy for that.

Denise Carbone:
When I log into something I’m like, “Wait, where do I go?” So, the intuitiveness of some of these apps and the design is escaping the newer, what’s being published today. But it’s really, you want to make sure whatever you’re building is going to be usable, it’s intuitive. All the things like practical foundational serum, best practices should always be applied, and I feel like it’s really partnering with who you’re building it for. And the business analyst aspect that feeds into the admin role is that why are you building it? What are you building? The how is always something, I don’t necessarily know if the end users need to see behind the curtain for it, but you want to make sure that what you’re building is, it works and it flows well.

I always still laugh because it still comes up today, we don’t want to do all these clicking. So, there’s many magical ways to make things a lot faster than to do all these clickings. The automation tools that are available today on the platform are amazing, the flow automation is just next level. So, I feel like there’s a lot of opportunity there to do really cool things, but you want to make sure the cool things are really going to support the folks that are going to use the platform.

Mike:
I couldn’t agree more. So, let’s talk about the cool things. AI is everywhere, and that’s in your session.

Denise Carbone:
It is.

Mike:
So, let’s talk about why. I mean, there’s been a lot. There’s always new features coming out with Salesforce, and sometimes we cover features too heavily but I feel like there’s AI in everything. I was at a car dealer website the other day and they had an AI chatbot on the homepage. I was like, “Man, if car dealers got it, you know everybody’s got it now.” So, what makes it different for admins about why we need to learn and understand business and analyst skills in the age of AI?

Denise Carbone:
That is a great question. There’s truly a distinction between our human skills and the strengths of AI. AI is definitely a tool. It can be your autonomous agent, it can support different tasks, but the human skills are and will always hopefully remain relevant in this space. Right now, humans bring to the table we can do critical thinking, we provide judgment. Most people have emotional intelligence. The thing about BAs too, I talk about, and I think this is a skill that crosses both the adaptability and growth mindset, we have to be able to think differently. As humans, we are, “Okay, so what’s the new shiny thing out there? Do I need to learn this? How is it impacting the work I do today?” That statement I just said about, “Do I need to learn this? How does it impact the work today?” For me, just a full transparency, I did not need to learn how to actually build flows for a while.

I needed to be able to document and explain what needed to happen based on a client’s experience, or their workflow, or their processes. So, I would document processes and I would be able to document the heck out of how to explain or the build steps to create a flow. But I didn’t have to be hands-on right away. I do know how to build flow and I can build flows, but it wasn’t a skill that I had to learn because I was able to document that, write that, and I was able to articulate it well enough so that an admin or one of our builders was able to pick it up and run with it to build. But this day and age today with AI, it is here, it is not going away. It is not like a nice to learn later, it is if you don’t know it or if you haven’t started, you’re already behind.

And that’s one of the pieces I really want to emphasize is that if you have, and I would beg to differ with the folks attending Dreamforce, if this is not on the radar, if they’re not already in this learning journey, I’m sure a good majority of them are because it is so prevalent. But if they’re not, they’re already left behind. But I do feel that the human skills, storytelling, communication, ethnical reasoning, these are things that we bring to the table to it. Using AI, AI is really good at processing, data processing, or summarizing, maybe some automation. They’re good at pattern recognition, but they’re lacking certain human emotions, human elements that go into the work that we do today.

Mike:
I can agree with that. It’s also fun if you ask it for things and it’s like, “Yep, that sounds bad.” And you’re like, “No, you wouldn’t understand.”

Denise Carbone:
Yeah, one of another piece I’m going to hone in on a bit in the session is admin skills and BA skills are needed because you have to vet it. So many times I’ve seen people generate, the generative AI, I read this thing, I was like, “This is so AI generated.” I’ll go back to my team and I’m like, “All right, so here’s my problem. What other sources did you reference this on? Is this validated? This to me,” and I’ll point out a spot, “This looks like a hallucination. Is this a real thing?” “I don’t know.” And I was like, “Well, you need to check that.”

Mike:
Yeah. I was recently on a long road trip and some people in the car were scrolling through Facebook ads and stuff, and the one was like, “Oh, hey. Hey, I think I found one.” And they read it and I go, “That was 100% generated by AI.” It was describing some 1992 used car, and it said, “With a fresh battery for those cold starts on Iowa mornings.” And I was like, “Anybody that’s selling a $2,000 car probably doesn’t have the copy editing skills to put that level of detail into the ad.” So, yeah.

Denise Carbone:
Yeah, the description and the adjectives totally give it away.

Mike:
I mean, I am starting to get to the point where you look at an article and you’re like, “That’s AI generated,” and I’ll just give it to another AI and be like, “Just tell me what this says.” So, great. And so, I have to think back that somewhere AI is sitting going, “Isn’t it fun that the humans make us type all this stuff out and then the human make us decode all of our own stuff?”

Denise Carbone:
Probably, probably. It is a little humorous in what we ask it to do. I know someone that actually wanted constructive feedback on his code, so he fed, and he actually asked the bot or the agent to change his tone to be, I don’t know, he called irrigate or snotty. So, he wanted to get the constructive feedback versus the, “Oh, this is great.” So, it was pretty funny and it was actually a pretty fun read on some of the feedback, and there’s a lot of experiments you can do with it. But yeah, in the end I think you really need to validate, at least from a solution perspective what it’s doing. It’s entertaining in some ways, but other ways you really need to validate against other true sources.

Mike:
So I’ll end on, I hope it’s not a hard question but it is something I’m curious about.

Denise Carbone:
Sure.

Mike:
As a business analyst, what is a question you ask now because of AI that you didn’t ask before?

Denise Carbone:
That is a great question. One of the things we ask people or clients today is, “What is your AI policy?” And what that actually entails is also governance, because there’s certain aspects of the platform that based on industry certain people not are allowed to use it. Or we have even some scenarios where we would use a Zoom recording and there’s an AI summary function. Some people don’t allow that. But actually when it comes to platform questions, it’s relevant in that do you have a policy? You’re thinking about a governance, there is a good 50-50 that don’t and a good 50-50 that do. It’s also what are your thoughts about using AI? Do you have any use cases today that you’re thinking about using? We also recommend maybe AI readiness, which means do you have those use cases? Do you have policies or governance in place? How do you think you want to use AI? And then we can provide use case examples based on those responses, but those are the top few questions we ask.

Mike:
Yeah, I could understand that because otherwise everybody’s just going to be, “Well, at this point I just have Bob,” because I’m waiting for somebody to call AI Bob, “Generate this,” and cool. So, isn’t that your job? Yes, it is. Yes, it is.

Denise Carbone:
It is.

Mike:
Denise, I think your session at Dreamforce is going to be amazing. I hope you can get out into the user group world and share more BA skills with admins, because I want us to be more than order takers. And I say that for all of the personas, I don’t think a developer should be an order taker or an architect. I mean, bring yourself into the role, and bring your knowledge and experience because that’s what they hired you for, right?

Denise Carbone:
Yeah, for sure. No, I love it. And we have some really great BAs out there in the space, but I’m also happy to support and share knowledge, and just talk through these scenarios, because the work is not going away. It’s only getting deeper.

Mike:
Right, I know. The work’s not going away, it’s just going to change.

Denise Carbone:
Right.

Mike:
Right? Thanks so much for being on the podcast.

Denise Carbone:
Awesome. Thank you, Mike.

Mike:
Big thanks to Denise for sharing her story and dropping some really insightful knowledge about the intersection of business analysis and admin work in the age of AI. And her message, ask why, build smart, and don’t let imposter syndrome hold you back really resonates with me. Now, if you found this episode helpful, pass it along to a fellow Salesforce admin, or a teammate, or a friend maybe. Until next time, we’ll see you in the cloud.

The post How Do Admins Use Business Analysis Skills Effectively? appeared first on Salesforce Admins.