Managing UX Debt from AI Design

This month we're joined by Rich Mehta, who has spent the last year helping out at Headscape and has recently been writing about how AI is changing the way people search. After a brief detour into childhood nicknames (Mr. Bogey and Lily, since you ask), we get into what it actually takes to show up when someone asks ChatGPT or Claude a question. Paul also gets very excited about a testing tool everyone else probably discovered years ago, and we finish with an article about cleaning up the mess AI leaves behind.

App of the Month: Polypane

Paul recently pushed a website live for a client, which he hadn't done in a long time, and found himself needing a decent way to check every breakpoint without losing the will to live. He started with Mobile Simulator, a free Chrome extension that shows any page at a mobile breakpoint with one click and lets you switch between a long list of device sizes. It doesn't emulate the actual devices, but for a quick sense check it does the job nicely.

The real star, though, is Polypane, which takes the same idea to the extreme in a dedicated app. Its main features include:

• Multiple viewports side by side, from mobile all the way up to 5K, all kept in sync
• Real device presets, or your own CSS breakpoints and custom sizes
• Synchronized scrolling, clicks, hovers, and even form input across every pane
• Emulation of user preferences such as high contrast, dark mode, and reduced motion
• Built-in accessibility checks and 40+ debug tools, including layout overlays, readability, and color blindness simulators
• Social and meta previews, plus automatic broken link and redirect checks
• Live CSS and JavaScript editing applied to one pane or all of them

Marcus admitted his developers mostly stick to browser tools, but seeing every breakpoint at once and acting on them all together saves an awful lot of repeated testing. Fair warning, though, show this to your team and they'll immediately ask for a bigger monitor.

Getting Found by AI with Rich Mehta

Rich has written 2 articles for Headscape that started this conversation. What happens when your law firm's clients stop Googling and start asking AI looks at how search habits are changing, and Writing law firm content for AI: what the research suggests covers good practice for getting found. They were written with law firms in mind, but the advice applies to pretty much any organization.

What is GEO?

GEO stands for generative engine optimization, although you'll also hear it called AEO, because the industry hasn't quite settled on a name yet. It's about making sure your organization appears when someone asks an AI tool a question, whether that's ChatGPT on their phone or Claude on their desktop. Rich explained that these tools answer questions in 2 ways. They draw on a huge index of training data (which is cheaper for them), and they also go out and search the web in real time to fill in the gaps.

Paul came into this with the same skepticism he has about SEO, which is to say he thinks most of it is bollocks. Rich's response was that there is far more overlap between the 2 than most people like to admit, and that the best starting point for both is the same, which is working out what your audience actually needs from you rather than churning out content because someone told you to.

FAQs make a comeback (sort of)

That led to a lively bit of back and forth about FAQs. AI seems to like question and answer content, which is a little ironic given how long we've spent telling people to get rid of FAQ pages. Paul's view is that answering frequently asked questions was never the problem. The FAQ page is where content goes to die. The answers belong in context, so a returns policy should sit on the product page, cart, and checkout rather than on some forgotten page in the main navigation. Rich agreed and added that putting answers in a question and answer format also lets you mark them up with FAQ schema from schema.org, which helps AI understand which answer goes with which question.

Are we all just guessing?

Google publishes detailed guidance on how to get listed, but OpenAI and Anthropic don't offer anything comparable. Rich pointed out that these tools are designed to make their own decisions, which makes it hard for their makers to say "do X and you'll appear." They're also much younger than Google, which took decades to become that open.

Instead, a lot of what we know comes from academic research. Rich's article draws on a 2023 study from Princeton and IIT Delhi that looked at how AI builds up a picture of an "entity," meaning a business, organization, or person. In short, it gathers everything it can find about you from across the web and uses that picture when answering questions. The mechanics haven't changed much since, even if the models have improved.

The sources each model relies on vary a lot. Reddit is a big source for many of them, while some models use YouTube content in 30-40% of responses and others ignore it almost entirely. Answers also change depending on whether you use the web or mobile app and on what the tool remembers about you, much as Google personalizes search results.

Do you even need a website anymore?

If AI can pull information from everywhere, why bother with a website? Rich argued that your website remains the source of truth about your organization. People still visit to check what AI has told them, and it's the one place where you control how you describe yourself. AI is one more channel alongside SEO, and you need a strategy that spans both.

Where to start

Rich's first recommendation is an old-fashioned content audit. In practice that means:

• Get rid of the cruft, including old pages nobody links to, because AI will find them anyway
• Update content that is out of date, wrong, or thin
• Make sure important content, like your returns policy, is easy to find and well linked
• Add published or last reviewed dates so AI can tell old work from current expertise
• Describe things consistently, because AI pays close attention to the language you use

The dates point came up with a lawyer example. If a lawyer used to do commercial work and now specializes in fraud, undated older content can send AI mixed messages. Connecting people to the content they write builds a clearer picture of their expertise, and articles published on respected external sites add to their authority.

Paul added that this drives home the need for focus. An agency that does print, web, apps, and everything in between leaves AI with no idea what it actually does, while a clear specialism puts you in a much stronger position. Firms that offer a huge range of services can get around this by highlighting the specialists on their team.

Rich also pointed out that a mention no longer needs to be a link to count. Anything said about you anywhere can shape how AI sees you, which pushes this work closer to PR. Paul's worry is that falling traffic will tempt organizations to invest less in their websites. His counterpoint is that a leaner, more focused site would also be cheaper to run, since most in-house teams spend their time updating pages nobody reads.

Tools for tracking AI visibility

Rich expects metrics to move from clicks toward mentions and links in AI responses, which matters as zero click searches grow. Tools he mentioned include:

• Google Search Console, which has started adding AI data. Rich doesn't think it's very good yet, but it's free.
• Semrush, which combines SEO and AI visibility and is built on a dataset of around 238 million responses.
• Scrunch, Rich's go-to, at around $250 a month.
• Profound and Searchable, dedicated AI optimization tools at around $400 a month for an agency license. Some can even show which AI agents are visiting your site and why.

As with SEO, no single tool covers everything, so you'll probably need a combination.

Read of the Month

The Custodial Era of UX: Cleaning Up After AI from Nielsen Norman Group looks at the UX debt that builds up when teams use AI to design interfaces at speed. It suggests 3 ways to keep control:

• Triage AI-generated features before they get built
• Make evaluation keep up with production, by matching research depth to risk, using AI to speed up research, keeping a user panel, and using heuristic checklists
• Build UX into generation, with UX context files, a design system, accessibility requirements, and a list of deceptive patterns to avoid

If AI is producing interfaces faster than you can test them, the quality problem is only going to grow.

Marcus' Joke

This month's offering comes from the Edinburgh Fringe and a comedian called Olaf Falafel, which is a magnificent name whether or not it's real.

I remember thinking as I taped a piranha to my boomerang, this could come back and bite me.

Terrible, and exactly what we deserve!

Find The Latest Show Notes