Analysing Robots.txt at scale with HTTP Archive and BigQuery

In this episode of Search Off the Record, Martin and Gary turn a simple robots.txt question into a data‑driven deep dive using HTTP Archive, WebPageTest, custom JavaScript metrics, and BigQuery. They explore how millions of real robots.txt files are actually written in 2025–2026, which directives and user‑agents are most common, and what that means for modern crawling and AI bots. Perfect for beginner to mid‑level developers and SEOs, you'll learn how large‑scale web measurement works (HTTP Archive, Chrome UX Report, Web Almanac), and how to turn raw crawl data into actionable SEO insights. Subscribe for more candid conversations about crawling, indexing, and the data behind how Google Search and the web really work. Resources: Web Almanac →  https://almanac.httparchive.org/en/2025/ Robotstxt custom metric for the HTTP Archive →  https://github.com/HTTPArchive/custom-metrics/pull/191 robots.txt parser change → https://github.com/google/robotstxt/commit/4af32e54b715442bb04cd0470e99192f0ffb9792#commitcomment-178586774 Episode transcript → https://goo.gle/sotr108-transcript Listen to more Search Off the Record → https://goo.gle/sotr-yt   Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.  #SOTRpodcast #SEO #GoogleSearch Speakers: Martin Splitt, Gary Illyes

In this episode of Search Off the Record, Gary and Martin dig into what "page size" and "page weight" actually mean for developers, users, and search engines. They discuss exploding web page sizes: median mobile homepages hit 2.3 MB in 2025 Web Almanac (up 3x from 2015), key insights for developers on page weight definitions, Googlebot's crawl limits, HTML bloat from structured data/images, and why size still hurts UX on slow connections despite faster networks. If you build or maintain websites, this conversation will help you rethink how much data your pages ship, where bloat really comes from, and why page weight still matters even as connections get faster. Resources: ​Web Almanac → https://almanac.httparchive.org/en/2025/ HTML living standard → https://html.spec.whatwg.org/multipage/ How page speed helps with conversions →  https://www.thinkwithgoogle.com/marketing-strategies/app-and-mobile/mobile-page-speed-data/  Episode transcript → https://goo.gle/sotr106-transcript Listen to more Search Off the Record → https://goo.gle/sotr-yt  Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.  #SOTRpodcast #SEO #GoogleSearch Speakers: Martin Splitt, Gary Illyes    

Developers often talk about Googlebot as if it were a single program you could just run as "googlebot.exe", but that is not how Google's crawling actually works. In this episode of Search Off the Record, Martin and Gary from the Search Relations team unpack how Google's crawling infrastructure is really built and operated.​ They cover why "Googlebot" is a misnomer and how it relates to a central crawling software-as-a-service used by many Google products​, how crawl behavior is controlled centrally to avoid overwhelming sites (throttling, handling 503s, and "don't break the internet" safeguards)​ and more! If you build for the web, work on SEO, or just want a more accurate mental model of how Google crawls pages, this behind‑the‑scenes discussion is for you. Resources: ​Crawlers → https://developer.google.com/crawling  Episode transcript → https://goo.gle/sotr107-transcript  Listen to more Search Off the Record → https://goo.gle/sotr-yt   Subscribe to Google Search Channel → https://goo.gle/SearchCentral  Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.  #SOTRpodcast #SEO #GoogleSearch Speakers: Martin Splitt, Gary Illyes

Martin and Gary unpack how HTML parsing really works, why the HTML standard is so lenient, and how messy markup can silently break key SEO signals like hreflang and rel=canonical. They revisit validators and cross‑browser hacks from the Netscape/IE days, and discuss whether semantic HTML and strict validity truly matter for search. You'll also hear when link hints like preload, prefetch, and DNS prefetch help performance (and indirectly SEO), and where meta and link tags really belong. ​ Resources: HTML Living Standard → https://html.spec.whatwg.org/ Episode transcript → https://goo.gle/sotr105-transcript Listen to more Search Off the Record → https://goo.gle/sotr-yt  Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.  #SOTRpodcast #SEO #GoogleSearch Speakers: Martin Splitt, Gary Illyes

In this episode of Search Off the Record, Martin and Gary from the Google Search Relations team tackle a deceptively simple question: do you still need a website in 2026? Starting from the recurring industry claim that "the web is dead," they explore how the web has evolved through the rise of apps, AI chatbots, and social platforms, and why the answer almost always ends up being "it depends." Tune in for an engaging discussion on how websites remain relevant and what it means for content creation and discovery. Episode transcript → https://goo.gle/sotr103-transcript Listen to more Search Off the Record → https://goo.gle/sotr-yt  Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.  #SOTRpodcast #SEO #GoogleSearch Speakers: Martin Splitt, Gary Illyes

​Join Martin and Gary as they dive into Search Off the Record's Episode 103, unpacking the 2025 Year-End report on crawling issues. Discover fascinating insights on faceted navigation, action parameters, irrelevant parameters and more, highlighting the biggest challenges faced by web crawlers last year. With humor and expert analysis, this episode reveals critical takeaways for webmasters and SEO professionals. Don't miss valuable tips to enhance your site's crawl efficiency! ​ Resources: URL structure best practices for Google Search → https://developers.google.com/search/docs/crawling-indexing/url-structure   Crawling December: Faceted navigation → https://developers.google.com/search/blog/2024/12/crawling-december-faceted-nav   Xkcd (programmer humor) → https://xkcd.com/327/     

In this episode, John Mueller and Danny Sullivan continue their conversation about the changing landscape of Search. They discuss the practicalities of hiring an SEO professional in the age of AI: Do you really need one? How do you evaluate them? Danny also shares his thoughts on third-party SEO tools, "domain scores," and why creators shouldn't rush to break their content into "bite-sized chunks" just to please Large Language Models (LLMs). Listen to more Search Off the Record → https://goo.gle/sotr-yt  Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO #GoogleSearch

In this episode, John Mueller and Danny Sullivan talk about SEO and the new kids on the block: AIO, GEO and others. Danny shares how these things interact with each other and what he thinks website owners should look for in their content creation and how Google's principles align with the way people use AI and search engines to navigate the web.   Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.  

Celebrate the 100th episode of Search Off the Record with Martin, Lizzi, Cherry, John and Gary as they revisit memorable moments, touching on a bunch of topics. This special episode touches on Google's mobile-first indexing, the intricacies of the Caffeine indexing system, and a deep dive into JavaScript scoping – and hoisting. You'll hear again  about optimizing documentation traffic and the delightful world of Google Doodles. Discover key insights that impact your SEO strategies and web development practices. Resources:  Episode transcript → https://goo.gle/sotr100-transcript Chapters:  0:00 - Welcome and 100th Episode Intro 2:21 - Lizzi's favorite: SEO starter guide readings 3:01 - Gary on Mobile-First Indexing Importance 3:39 - Revamping the SEO starter guide 5:13 - Meta tags and Google's indexing system 9:11 - Cherry's use of podcasts for work 12:09 - Deep dive into caffeine indexing 13:59 - The mechanics of data conversion in caffeine 16:12 - John's favorite: The magic of Google Doodles 23:30 - JavaScript ccoping and hoisting 25:47 - Optimizing documentation traffic  29:29 - Conclusion: 100 Episodes of Search Off the Record Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO Speaker: Martin Splitt, John Mueller, Cherry Sireetorn Prommawin Products Mentioned: Search Console

Explore critical aspects of search engine optimization for web content that lives behind a login or paywall. In this episode, Martin and John cover how Googlebot interacts with these pages, the role of paywall structured data, and methods to prevent unintended indexing of private information. Get practical advice on improving your website's visibility and user experience while maintaining data security in Google Search. Resources:  Episode transcript →https://goo.gle/sotr099-transcript  Chapters:  0:00 - Intro  0:09 - Navigating SEO for login and paywalled content 2:01 - How Google indexes restricted content 10:51 - Strategies for controlling content visibility 20:54 - Testing and optimizing your site for Search Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO Speaker: Martin Splitt, John Mueller  Products Mentioned: Search Console