SEO in 2026 Crawling and indexing management needs to be a corner stone of your SEO strategy - James McLoughlin
Dec 17, 2025
James McLoughlin, a Technical SEO specialist, dives into the essentials of crawling and indexing management for 2026. He emphasizes the need for effective strategies to ensure both traditional and LLM crawlers can discover your content without blockers. Key topics include the differences between crawling and indexing for various types of crawlers, how to surface valuable content, and the importance of integrating crawl monitoring into your workflow. McLoughlin also shares tips on optimizing internal linking and prioritizing high-value pages for maximum visibility.
AI Snips
Chapters
Transcript
Episode notes
Schedule Regular Crawls
- Run scheduled crawls and monitor crawl/index counts to spot sudden fluctuations quickly.
- Create tickets and work with developers to fix issues when discovered so crawlers can find important pages.
Remove Technical Blockers
- Ensure important URLs are crawlable with no disallow rules or server-side blocks.
- Use proper HTML links and clear indexing directives to tell search engines what should be indexed.
Weekly Or Sample Crawls
- Run at least a weekly crawl or a representative sampled crawl for very large sites.
- Use cloud crawlers for big sites to avoid local resource limits and get scheduled alerts.