Rebecca Nguyen: Collaborative Content Design Leadership at Indeed.com – Episode 15

Rebecca Nguyen In her work as a content designer at Indeed.com, Rebecca Nguyen is finding new opportunities to assume a leadership role on teams working with generative AI. Rebecca feels fortunate to work with teams that recognize the value of writing and design skills. She's also finding that generative AI is the perfect place for content design to take the lead. We talked about: her work as a senior UX content designer at Indeed and her recent shift to focus on product teams using generative AI how well-suited content designers are to AI products the unique challenges of working with non-deterministic large language models their process for designing prompts and how they evaluate them her learning curve around the loss of some language control that you get in conventional content design the main differences between prompt engineering (the how) and content design (the what) her ability as a content designer to lead more in the AI space than in prior design roles how they balance the use of outsourced LLM solutions like OpenAI versus developing their own models the lack of genuine intelligence in LLMs how her fear and concern about AI is eased the more she works in the LLM world how the evaluation component of designing content for AI creates more work for content folks one of the main benefits of LLMs - their ability to take on tedious rote content work the child-like nature of LLMs the surprising liberating effects of simply not worrying about whether or not you have a seat at the proverbial table Rebecca's bio Rebecca Nguyen (she/her/hers) is a Senior UX Content Designer at Indeed. She’s been part of marketing, UX, and product design teams at Bankrate, Northwestern Mutual, and LPL Financial, where she established the content strategy practice. A Confab speaker and workshop instructor, Rebecca is also an award-winning memoirist. Connect with Rebecca online LinkedIn RebeccaAnneNguyen.com Video Here’s the video version of our conversation: https://youtu.be/8WnxlXXKxeY Podcast intro transcript This is the Content and AI podcast, episode number 15. Just as content design was emerging as its own craft and profession, along came generative AI. At first it looked like ChatGPT and large language models might displace content designers (unfortunately, it appears from recent layoffs that some executives may still think this is the case), but at Indeed.com, Rebecca Nguyen has found that working with LLMs has given her more work, not less, and that her content design efforts are now more interesting, rewarding, and impactful. Interview transcript Larry: Hi everyone. Welcome to episode number 15 of the Content + AI podcast. I'm really happy today to welcome to the show Rebecca Nguygen. Rebecca is a senior UX content designer at Indeed. Welcome, Rebecca. Tell the folks a little bit more about what you do at Indeed. Rebecca: Hey, thank you so much, Larry. Great to be here. Yeah, I'm a senior UX content designer at Indeed. I've been there for a couple of years now, going on two years, and I work on product teams to make sure their content is useful and useful and accessible and inclusive and all those goodies that we're used to. And in the past six months or so, my role has really shifted and I've been almost exclusively focused on working with product teams who are using generative AI in their products. Larry: And that's why I wanted to have you on the show is we talked about this a while back. And that's one way to think... One way I think about that is all of a sudden we have new collaborators in two senses. One, we have these new, we're talking to machines in our work because they're generating some of the language we work with, but there's also a lot of other new collaborators. Tell me a little bit about how the people around you have changed over the last six months. Rebecca: Yeah, that's such a great point. So we're probably, if we're working in product content, we're used to working with product managers, we're used to working with UX designers, engineers. And that has shifted in that the team that I am partnering with now is made up of engineers and product managers, but we're also working really, really closely with data scientists and we do not have a UX designer or UX researcher on the team right now. So UX content design is really the entire voice of UX in this group, which is really cool. Larry: That's really interesting because often we're the last one in. How does that feel going in there as a sole UX person? Rebecca: It's exciting. It's been a little bit intimidating, but I haven't found myself feeling completely lost or anything. I think it's been great. As we were chatting earlier and you said we're really... We're creating a content product when we're working with these language models. The output is text and language, and so who better could be suited to drive and design the language when working with one of these models? It's been a really natural fit. And then the activities and tasks and approach has been different from anything I've done before, but it's well-suited to a content designer skills, I would say. Larry: Well, that's it. So what has that transition been like? You said the activities and the tasks differ. It sounds like it kind of rhymes with your old conventional product work, but how is it different now? Rebecca: Like that. Yeah, I sort of talk about it as if we think of a sandwich and in that content creation moment, that's the meat. That's sort of like when we're going through a design thinking process, we're doing some discovery or research or we're deciding on the problem that we want to solve, and then we get to that moment where we make the thing, we design the thing and we might be writing words. And after that we are iterating and getting feedback and seeing how it performs and measuring and iterating more, et cetera. Rebecca: The difference for me with generative AI has been spreading my focus out and becoming more of the bread. So instead of the meat, that creation moment, when you're working with a language model, the model takes on that task. They're the ones creating the content. And your focus as a human is all of that stuff on the periphery of that, so the prepping, which we would be sort of the prompt engineering and design where we're telling the model what we want it to do, and then the evaluation piece where we're looking at what the model did and saying, "Okay, was it successful? Did it follow directions? Could we do it better?" Rebecca: So it's almost like you become a teacher of content design instead of a content designer where you're actually making the thing yourself. Larry: Interesting. I have not heard it articulated that way, but that makes perfect sense because... Well, they're called learning models and you're the teacher. That's great. And you mentioned both prompts and one of the things you just said made me think that people always talk about prompt engineering, and you talked about engineering and designing prompts. Do you go into prompt creation with a designer hat on because you're working with engineers? Do you think more as a designer in that world? Rebecca: Yeah, I definitely think so. Particularly as a content designer, thinking about how does the language inside the prompt impact the output and to make sure that content design considerations are represented in the prompt as much as possible to make sure that we're getting the output where we want it to be. We're sort of preemptively correcting mistakes or anticipating mistakes that could happen. Rebecca: For example, when you get familiar with a model like ChatGPT, you can see, and we all can see as content designers sort of that out-of-the-box tone that the model assumes, the model that's been trained on the internet. So it's a very casual tone. It is, in my opinion, it's overly friendly in a way that can be kind of annoying. There's lots of exclamation points, there's a lot of celebration for small things that may not require such celebration. It helped you with a task and it's like, "You're so welcome. Awesome." And you're like, "Calm down." Rebecca: That tone and that voice isn't always appropriate for a product. And so when you're getting in there and designing prompts, you have this opportunity to modify as best you can. And the cool thing about prompt engineering is that you can do a lot of playing around and you can see how different instructions impact the outputs and then tweak and adjust from there. But that was surprising to me because I think that on my team and at my organization, at first we were sort of thinking about this on the other end, once we see the output, then let's evaluate it and give feedback. But the problem is that once the output has happened, it's too late. It's not like working with a human where you can revise it and create this static thing. It's always going to be different every time. It's that non-deterministic nature of a large language model. And so really we want to get in at the prompt stage to try and drive and direct before the output happens. Larry: That's so interesting. But you're still getting some feedback from it, too. You mentioned earlier how one of the pieces of bread is about iteration in your sandwich. And then, as you're talking there, I'm also reminded back when you said that you don't have UX researchers on the team. Are there more automated ways of getting feedback? Like for you, because you're still looking at it after the fact to see compliance with... Not compliance, but sort of alignment with voice and tone and that kind of thing. I guess tell me a little bit about that loop. Rebecca: Yes. So we have content design at the beginning, which would be the... Actually, even before we do prompt design,