Do sitemaps still matter? - Transcript

Should you publish your content as a PDF or stick strictly to standard HTML? Does Google treat PDF files differently when it comes to indexing, ranking, and AI search features? In this episode of Search Off the Record, Martin Splitt and John Mueller from the Google Search Relations team demystify how Google Search handles PDF documents. They break down some of Google's internal pipelines, explain why putting an LLMs.txt file inside a PDF is a bad idea, explore common technical traps like rasterized pages and canonicalization issues, and share practical advice on managing PDFs for SEO, AEO (SEO), and user experience. In this episode, you'll learn: Google's PDF converter pipeline: How Google converts binary PDF containers into HTML pages during indexing so the rest of the pipeline can process them normally. LLMs.txt as a PDF: Why saving an LLMs.txt file as a PDF is a bad idea, even if you like LLMs.txt. File limits & Project Gutenberg: Why crawling limits exist, and how HTML and PDF size limits function in the crawl pipeline. Rasterized pages & accessibility: How pages saved as images destroy text readability, ruin screen reader accessibility, and prevent Google from extracting text content. Canonicalization & "binary blobs": Why identical PDF downloads across different sites make canonicalization harder and how using rel="canonical" HTTP headers or embedded header links can help. PDFs for SEO in AI Overviews: How PDFs are parsed, ranked, and referenced in Google Search, AI Overviews, and broader AI systems for training and grounding. Key takeaways for SEOs & developers: HTML is preferred, and PDFs work: HTML offers better mobile responsiveness, zoom reflow, and faster loading. Internally, Google converts indexable PDF text into HTML so they can rank normally and appear in AI features. Test for text indexability: Highlight and copy text out of your PDF into a text editor; if text doesn't copy, it might be a rasterized image and can't be indexed easily. For restaurant menus: A PDF menu is better than no menu at all for restaurants, but a responsive HTML web page provides superior performance, accessibility, and user experience.     Chapters 00:00 – Introduction: Welcome and setting the stage for PDF discussion. 00:50 – Can you save an LLMs.txt file as a PDF? 01:56 – Plain text vs. Binary container: What makes PDFs different from HTML? 03:08 – Inside Google's converter pipeline: Turning PDFs into HTML for indexing. 05:08 – Extracting images and links from PDF files. 06:04 – Crawling limits, file sizes, and testing cut-off thresholds. 08:35 – When does it make sense to publish a PDF? 09:33 – Do PDFs rank in Search, AI Overviews, and AI mode? 11:22 – Technical pitfalls: Rasterized pages, scanned images, and accessibility. 12:53 – Mobile connection delays, layout locking, and zoom issues. 14:12 – Canonicalization challenges with duplicate "binary blobs". 15:57 – Metadata, password protection, and updating PDF files. 18:59 – The problem with dynamic CMS PDF generation. 21:08 – The restaurant menu debate: PDF vs. HTML web pages. 23:00 – PDFs for GEO (Generative Engine Optimization) & AI grounding. 24:14 – Outro and community question. Resources Mentioned: Official Google Search Central Documentation: https://developers.google.com/search Google Search Console: https://search.google.com/search-console     Don't forget to like and subscribe to the podcast on your favorite platform to catch every behind-the-scenes episode from the Search Relations team! Episode transcript →  https://goo.gle/sotr115-transcript Listen to more Search Off the Record → https://goo.gle/sotr-yt   Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO #GoogleSearch #TechnicalSEO #GenerativeEngineOptimization   Speakers: Martin Splitt, John Mueller  

Sitemaps have been a core element of web development and search engine optimization for two decades, yet Search Console warnings like "Couldn't fetch" continue to generate concern among site owners. Are XML sitemaps still necessary with modern search crawlers, or can search engines and AI bots figure out site structure on their own? In this episode of Search off the Record, Google Search Relations team members Martin Splitt and John Mueller break down the history, technical specifications, and some misconceptions surrounding sitemaps, RSS feeds, LLMs.txt, canonicalization signals, and Search Console indexing behaviors. In this episode, you'll learn: The Origins of Sitemaps: How early web crawling challenges led to the sitemap standard and how building a sitemap generator started John Mueller's career at Google. Obsolete vs. Active XML Fields: Why Google dropped support for the priority and changefreq tags, and how the lastmod date is actually evaluated. XML Sitemaps vs. RSS Feeds: How RSS feeds function as lightweight sitemaps for recent updates and why AI training crawlers utilize both. Technical Limits & Index Files: The exact URL (50,000) and file size (50MB uncompressed) limits, alongside best practices for sitemap index files and robots.txt declarations. HTML Sitemaps & LLMs.txt: Why user-facing HTML maps and Markdown-based LLMs.txt files cannot replace structured XML sitemaps for search engines. Troubleshooting "Couldn't Fetch": How host load management and perceived site quality (crawl demand) trigger non-technical fetching errors in Search Console. Key takeaways for SEOs & developers: Keep Sitemaps Enabled by Default: Even for small websites or dynamic CMS setups, keeping auto-generated sitemaps active carries no downside and assists with discovery as sites grow. Provide Accurate lastmod Timestamps: Avoid setting all page timestamps to the current date; Google evaluates date reliability and will ignore lastmod signals if they are inaccurate or abused. Submit Canonical URLs Only: Always place clean, canonical versions of URLs inside your sitemap file rather than parameter-tagged tracking URLs to help guide Google's canonical selection. "Couldn't Fetch" Isn't Always a Syntax Error: If Search Console reports "Couldn't fetch" on a valid, accessible XML file, the cause is often host load throttling or low crawl demand linked to perceived site quality.   Chapters 00:00 – Introduction & Welcome 00:57 – Sitemaps History & John Mueller's Path to Google 03:42 – Why Search Engines Started Using XML Sitemaps 04:46 – Deprecated Fields: Priority, Change Frequency, and lastmod 06:36 – Sitemaps for Small Sites vs. High-Volume News Outlets 08:50 – Canonical Selection & Parameterized URLs in Sitemaps 11:13 – Comparing RSS Feeds and XML Sitemaps 12:58 – Size Limits, Compression, Index Files, and robots.txt 14:15 – Custom File Naming, Privacy, and AI Crawlers 16:54 – Why HTML Sitemaps Don't Replace XML Sitemaps 19:06 – Will LLMs.txt Replace Sitemaps for AI? 22:05 – Decoding Search Console's "Couldn't Fetch" Error 24:16 – Hreflang, Image, and Video Sitemap Extensions 25:14 – Key Takeaways & Outro Resources Mentioned: Official Google Search Central Documentation: https://developers.google.com/search Google Search Console: https://search.google.com/search-console   Don't forget to like and subscribe to the podcast on your favorite platform to catch every behind-the-scenes episode from the Search Relations team! Episode transcript →  https://goo.gle/sotr114-transcript Listen to more Search Off the Record → https://goo.gle/sotr-yt   Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.  #SOTRpodcast #SEO #GoogleSearch #SearchConsol#SEO #WebDevelopment #GoogleSearchConsole #Sitemaps #SearchEngineOptimization #GoogleSearchRelations #LLMstxt #TechnicalSEO #SearchOffTheRecord Speakers: Martin Splitt, John Mueller

Is your website's internal search feature secretly acting as an open invitation for crawling lots and lots? In this episode of Search off the Record, Martin Splitt and John Mueller pull back the curtain on how internal search results pages can turn into "infinite crawl spaces" that trap Googlebot, waste your crawl budget, and spike your database load. They break down the critical technical differences between blocking search pages via robots.txt versus noindex tags, and expose the massive security liabilities of leaving these pages indexable.  In this episode, you'll learn: The "Infinite Crawl Space" Concept: How Googlebot treats internal search functions as infinite crawl spaces that can generate an endless loop of new URLs. Server Strain & Performance: Why uncached internal search pages force constant database lookups and ranking calculations, slowing down your website for real users. Robots.txt vs. Noindex: The distinct technical trade-offs of using a broad robots.txt disallow rule versus a robots meta tag or HTTP header noindex. The Spam Vector Threat: How bad actors search for pharmaceutical, adult, or casino queries on your site to piggyback off your domain authority and display spammy contact info in Google's index. Why 500 Errors aren't great: Why serving a 500 server error code to stop bots on search URLs will backfire and reduce Googlebot's crawl rate across your entire website. Category Pages vs. Search Pages: How systems like Blogger use search parameters for tag landing pages and why you should treat valuable category pages differently. Key Takeaways for SEOs & Developers: Fix Crawling at the Source: Do not use the Google Search Console Removal Tool to handle infinite search URLs; it only filters search results temporarily and does not stop Googlebot from hammering your server. Broaden Your Robots Rules: Use one broad wildcard rule in your robots.txt (like /search?) to cover all query parameters, keeping your file maintainable and clean. Build Real Category Pages: Instead of using internal search parameters as makeshift categories, invest in clean, dedicated category pages to help search engines understand your site's hierarchy. Don't Depend on Auto-Systems: While Google's systems try to automatically recognize and deprioritize infinite spaces, it is slow and unreliable—proactive manual configuration is always safer.     Chapters 00:00 - Intro & Greetings 00:45 - Defining Internal Search Results Pages 01:26 - How Googlebot Discovers Search Features & Creates Infinite Spaces 04:15 - Crawl Budget, Server Load, and Database Performance Hurdles 07:32 - Solutions: Robots.txt Disallow vs. Meta Noindex 10:09 - The Fallacy of the Search Console Removal Tool & 404 Pages 12:35 - Why You Should Never Serve 500 Errors to Bots 13:56 -  CMS Nuances: Tag Landing Pages and Blog Categories 15:26 - Crafting Broad Robots.txt Patterns and Historical Guidelines 18:17 - When (and When Not) to Allow Indexed Search Pages 20:41 - The Spam Vector Threat: Hacked Content, Casino, & Pharma Exploits 25:06 - Taking Proactive Security Measures for Clients 27:04 - Lazy Search Redirection Hack, Outro & Subscribing Resources Mentioned: Google Search Console (Removal Tool)    Do you have a legitimate reason for letting search engines index your internal search results page? Let us know in the comments below, or find us on LinkedIn to share your thoughts! Don't forget to like and subscribe to the podcast on your favorite platform to catch every behind-the-scenes episode from the Search Relations team! Episode transcript →  https://goo.gle/sotr113-transcript Listen to more Search Off the Record → https://goo.gle/sotr-yt  Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.  #SOTRpodcast #SEO #GoogleSearch #SearchConsole #SEOTips #CrawlBudget #RobotsTXT #GoogleSearchConsole #WebPerformance #WebSecurity #SearchOffTheRecord  

Should you panic when your Search Console indexing report is showing pages that aren't indexed? Is a 404 error code always a sign of a broken website? In this episode of Search Off the Record, Martin Splitt and John Mueller from the Google Search Relations team dive deep into the Page Indexing report in Google Search Console. They unpack why treating this report as a static inventory checklist to "fix" things is the wrong response, how to spot massive SEO-ruining hosting or CDN traps, and why a healthy website doesn't actually need a 100% index rate. In this episode, you'll learn: The Indexing Report Shift: Insights from the Search Console team's Hillel on why you should look for trend lines and systemic patterns rather than treating the report as a giant list of errors. When 404s are Good: Why expected 404 errors are technically correct for deleted content, and how to survive the "boss panic" of numbers that won't go down. The Domain Property Advantage: How setting up a domain property handles canonical shifts, www vs. non-www tracking, and performance data much cleaner. The Site Query vs. Search Console: Why the site: query is an artificial tool that might show old domain moves or hreflang swaps for years, making Search Console your only true source of truth. Hosting & CDN Traps: How aggressive bot protections, hidden interstitials, and "Soft 200" error pages completely destroy your crawl data and lead to malicious canonicalization. Discovered vs. Crawled: What it actually means when pages sit in "Discovered/Crawled - currently not indexed," and how to recognize holistic site quality issues over technical bugs. Key Takeaways for SEOs & Developers: Patterns over Inventories: Use the report to verify that your intentional changes (like site migrations or page removals) are processing correctly over time. Forget the Ratio: There is no magic metric for indexed vs. non-indexed pages. Even Google's own developer documentation has a massive chunk of non-indexed content due to intentional choices. Watch Out for Soft Blocks: Ensure your security layers or CDNs aren't serving "Are you a bot?" challenge screens to Googlebot with a 200 success code. Computers Fail (And That's Fine): Minor server blips, failed DNS requests, or temporary 500 errors happen. Google's systems are resilient and will just try again later.   Chapters 0:00 - Introduction: The Search Central Live coverage report confusion. 1:45 - Shifting perspectives: Treating Search Console as a pattern tracker, not a checklist. 4:36 - Tracking site migrations and processing data delays. 6:25 - Why 404 errors can be a good thing  8:00 - Handling canonical shifts and the value of Domain Properties. 11:12 - Why the site: query isn't telling you what you think it does (Domain moves and hreflang bugs). 13:38 - CDN bot protection and the absolute nightmare of soft error pages. 17:49 - How to use "marked as fixed". 20:32 - Discovered vs. Crawled Not Indexed: Is it a technical or site quality issue? 25:31 - Debunking the indexed-to-non-indexed ratio myth. 27:48 - Final verdict: How to stop fearing your Indexing Report. Resources Mentioned: Google Search Central: https://developers.google.com/search Google Search Console: https://search.google.com/search-console Search Central Live Events: https://developers.google.com/search/events    Are you actively stressing over your non-indexed page counts, or are you tracking the big trend lines? Let us know in the comments! Episode transcript →  https://goo.gle/sotr112-transcript   Listen to more Search Off the Record → https://goo.gle/sotr-yt  Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.  #SOTRpodcast #SEO #GoogleSearch #SearchConsole Speakers: Martin Splitt, John Mueller

Should you convert your website into Markdown to help Large Language Models (LLMs) understand your content better? Is "llms.txt" worth the effort for SEO? In this episode of Search Off the Record, Martin Splitt and John Mueller from the Google Search Relations team dive deep into the history of Markdown, its rise in the AI era, and whether it holds any real weight for search engine discovery. In this episode, you'll learn: The Origins of Markdown: From John Gruber and Aaron Swartz to its status as the "language of GitHub." Markdown vs. HTML: Why the "cleanliness" of Markdown is tempting for developers but potentially risky for site structure. LLMs & Markdown: Do AI crawlers actually prefer Markdown, or are they already experts at parsing HTML? The "Parallel Version" Trap: Why creating a separate text/Markdown version of your site for AI can lead to the same maintenance nightmares as dynamic rendering. Use Cases that Make Sense: When Markdown is actually superior (like developer documentation) and when it's totally unnecessary (like your shoe catalog). Key Takeaways for SEOs & Developers: Crawlers are built for the "messy" web: Google and other engines have decades of experience parsing HTML. Don't sacrifice discovery: Headers, footers, and sidebars in HTML provide critical context for site structure that a raw Markdown file might lack. Maintenance is king: Avoid the complexity of maintaining two versions of the same content. Chapters 0:00 - Introduction: Should we all be using Markdown? 3:45 - The history and purpose of Markdown. 7:15 - Why developers love it: Separation of style and content. 11:20 - Do crawlers need Markdown to understand your site? 14:50 - The danger of "parallel versions" and dynamic rendering lessons. 17:30 - Discussing the "llms.txt" proposal and AI agents. 21:00 - Where Markdown actually makes sense (Developer Docs). 24:00 - Final verdict: Stick to HTML for the web. Resources Mentioned: Google Search Central: https://developers.google.com/search Are you using Markdown for your site's frontend or just as a backend source? Let us know in the comments! Episode transcript →  https://goo.gle/sotr111-transcript Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral  Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.  #SOTRpodcast #SEO #GoogleSearch Speakers: Martin Splitt, John Mueller