Search Off the Record Crawling Challenges: What the 2025 Year-End Report Tells Us.
Feb 3, 2026
They break down the 2025 year-end crawl report and its biggest technical headaches. Faceted navigation and filter-driven URL bloat get close attention. Action parameters and plugins that spawn endless URL variants are explored. Irrelevant parameters, session IDs, calendar plugins, and encoding bugs round out the crawl chaos discussion.
AI Snips
Chapters
Transcript
Episode notes
Faceted Navigation Dominates Crawl Problems
- Faceted navigation (filtering + sorting) produces combinatorial URL explosions that dominate crawl issues.
- Gary Illyes says roughly 50% of reports were about faceted navigation causing huge URL spaces and server load.
Use Robots.txt To Quickly Back Off Crawlers
- If Googlebot crawls a large new URL space and overloads your server, block the problematic paths in robots.txt to force Googlebot to back off.
- Use the examples on google.com/robots.txt to craft disallow rules for parameterized or faceted URLs.
Watch Access Logs And Alert Early
- Monitor server access logs to detect abnormal crawler activity early and set alerts for honeypot or suspicious paths.
- If heavy Googlebot crawling harms users, craft robots.txt disallow rules and deploy them quickly (noting ~24h caching).