How Googlebot crawls the web

Explore critical aspects of search engine optimization for web content that lives behind a login or paywall. In this episode, Martin and John cover how Googlebot interacts with these pages, the role of paywall structured data, and methods to prevent unintended indexing of private information. Get practical advice on improving your website's visibility and user experience while maintaining data security in Google Search. Resources:  Episode transcript →https://goo.gle/sotr099-transcript  Chapters:  0:00 - Intro  0:09 - Navigating SEO for login and paywalled content 2:01 - How Google indexes restricted content 10:51 - Strategies for controlling content visibility 20:54 - Testing and optimizing your site for Search Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO Speaker: Martin Splitt, John Mueller  Products Mentioned: Search Console

Optimize your website's performance with lazy loading techniques. In this episode, John Mueller  and Martin Splitt discuss lazy loading, and its SEO implications. Find out how lazy loading affects indexing, ranking, and Core Web Vitals. Whether you're using native lazy loading or custom JavaScript libraries, this episode provides insights for developers on how to enhance website speed and user experience. Resources:  Episode transcript → https://goo.gle/sotr098-transcript  Chapters:  0:00 - Introduction  0:43 - What is lazy loading?  1:57 - HTTP chunking  3:13 - Primary drivers behind lazy loading  6:19 - Why not default lazy load all images?  8:29 - Lazy loading and SEO  8:39 - Impact to Core Web Vitals  9:17 - Largest contentful paint  9:58 - Affecting indexing and ranking  11:30 - Custom libraries for lazy loading 14:17 - How to recognize a problem 15:25 - Practical point of view  16:52 - Infinite scroll vs. lazy loading  19:31 - Lazy loading videos  20:51 - Small images and decorative images  23:16 - Where to chat about lazy loading images  23:55 - Outro  Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO Speaker: Martin Splitt, John Muelle, Martin Splitt Products Mentioned: Search Console

In this episode of Search Off The Record, John and Martin explore the intersection of SEO and photography. Is a website essential for photographers, or is social media enough? Martin, a technical SEO expert and photography enthusiast, dives into the challenges of building a photography website, optimizing images, and attracting the right audience. Whether you're a professional photographer or a hobbyist, this podcast offers valuable insights into how Search Engine Optimization can help you showcase your work and connect with your audience. Find out how to make your images more findable, understand your website's performance, and navigate the ever-changing landscape of digital marketing for photographers. Martin shares his experience building his own photography website and the challenges he faced with SEO for his personal website.  Resources:  Episode transcript → https://goo.gle/sotr097-transcript  Chapters:  0:00 - Intro 0:37 - The Expert?! 3:10 - Why do photographers need websites? 5:39 - Stock Photography and SEO 14:53-  Branding and Search 18:21 - Setting up Search Console 21:12 - Web pages and Social Media for your photos. 23:27 - Image Landing Pages 24:52 - Image Optimization 28:38 - Photography and Generative AI Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO Speaker: Martin Splitt, John Mueller Products Mentioned: Search Console

Martin and John dive into the world of CSS and explore its connection to SEO. Discover why CSS files matter for SEO, how modern CSS frameworks impact pages, and the SEO implications of CSS class names and pseudo elements. Moreover, learn practical tips for ensuring your CSS enhances - rather than hinders - your website's search performance. Resources:  Episode transcript → https://goo.gle/sotr096-transcript  Chapters: 0:00 - Introduction 0:56 - What is CSS? 1:37 - CSS file crawlability 3:52 - CSS flexibility and power 4:55 - Website and CSS growth 5:43 - CSS file size 9:15 - CSS class names and SEO 10:51 - Important CSS concepts 11:27 - CSS specificity 14:24 - Pseudo classes and pseudo elements 20:56 - Stop using before and after 22:47 - Viewport height 27:11 - Hiding things and background colours 28:05 - CSS images 35:55 - CSS for SEO 37:16 - Outro Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO Speaker: Martin Splitt, John Mueller Products Mentioned: Search Console

Are you a developer or small business owner struggling with Search Engine Optimization (SEO)? In this episode of Search Off the Record, Google Search Advocates John Mueller and Martin Splitt break down the essentials of SEO for small businesses. Learn practical tips for understanding your target audience, crafting a content strategy, and measuring your SEO efforts. They debunk SEO myths and explain when it's time to hire an SEO expert or agency. Get clarity on content strategy, technical SEO basics, the importance of relevant content and how to measure success. This video is perfect for developers who are curious about search and need a non-technical introduction to the SEO space, and small business owners who want to improve their website's visibility in Google Search. Resources:  Episode transcript →https://goo.gle/sotr095-transcript Chapters: 0:00 - Introduction: The SEO Questions We All Get Asked 0:35 - SEO Starter Guide 0:58 - Defining Your Website Goals & Customer Focus 2:51 - Food Truck Example 4:20 - Understanding Your Customers & Search Intent 6:21- Content Strategy: What Should You Write About? 7:06 - Technical Foundation Check: Is Your Site Okay? 7:54 - Website Name Searching 9:46 - CMS 10:03-  How to Approach Content Creation 11:52 - Getting Customer Feedback & Keyword Research 13:33 - How Serious Is My Goal 13:33: - How To Measure SEO Impact 14:30 - Monitoring Website Changes: Search Console & Analytics 16:03 - How Long Changes Take 17:41 - Staging Sites Vs Live Sites 18:44 - When to Hire an SEO Expert or Agency 21:14 - Red Flags SEO Promises 21:14 - Resources Mentioned 23:52 - How Long Does SEO Take 26:23 - AI and SEO 28:30 - Outro Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO Speaker: Martin Splitt, Gary Illyes Products Mentioned: Search Console,

Developers often have misconceptions about SEO. In this episode of Search Off the Record, John Mueller and Martin Splitt clarify what matters and what doesn't. They cover areas including: Optimizing themes for SEO, The indexing API, and common HTML mistakes. This podcast delivers essential knowledge for developers to enhance their SEO strategies and avoid common pitfalls. Resources:  Episode transcript → https://goo.gle/sotr094-transcript Chapters: 0:00 - Intro to SEO misconceptions 0:15 - Why developers should care about SEO? 0:59 - Technical vs non-technical SEO 4:38 - The importance of naming things 6:03 - Title and Meta Description 7:31 - Is bad Html bad for SEO? 12:43 - Core Web Vitals 17:22 - Javascript SEO 27:00 - Google APIs and SEO 27:42 - Google Search algorithm 31:32 - What about Html comments? 31:58 - Themes and SEO 32:13 - How to reach out to Martin 32:57 - Wrap up Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO Speaker: Martin Splitt, Gary Illyes Products Mentioned: Search Console,

In this candid episode of Search Off the Record, Gary Illyes and Martin Splitt from the Google Search team peel back the curtain to reveal a more human side of their work and challenges. Instead of diving deep into technical concepts, they reflect on their personal journeys—touching on everything from public speaking nerves to building empathy between developers and SEOs. Through playful banter and thoughtful commentary, they share how communication, context, and curiosity play vital roles in their day-to-day roles, both internally at Google and in the broader web ecosystem. The conversation also explores the evolution of Martin's career, his passion for the web, and how his developer background helps him bridge gaps in technical understanding. The duo navigates the complexities of delivering accurate guidance in public forums, the risk of being taken out of context, and the importance of clear communication across diverse audiences. It's a thoughtful, lighthearted, and occasionally philosophical discussion that reminds listeners that behind the guidance, there are real people striving to make the web better for everyone. Resources:  Episode 93 transcript→https://goo.gle/sotr093-transcript  Chapters: 0:00 – Intro & Setup 1:30 – Life at Google 3:30 – Speaking to Developers 6:30 – Early Speaking Days 10:00 – SEOs vs Devs 14:00 – Should SEOs Code? 19:00 – Context is Everything 23:30 – Clearer SEO Advice 26:30 – Why No Slides 29:00 – Wrap Up Chat Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO Speaker: Martin Splitt, Gary Illyes Products Mentioned: Search 

In this episode of Search Off the Record, Martin and Gary from the Google Search Relations team take a deep dive into how Googlebot and web crawling work—past, present, and future. Through their humorous and thoughtful conversation, they explore how crawling evolved from the early days of the internet, when scripts could index a chunk of the web from a single homepage, to the more complex and considerate systems used today. They discuss the basics of what a crawler is, how tools like cURL or Wget relate, and how policies like robots.txt ensure crawlers play nice with web infrastructure.   The conversation also covers Google's internal shift to unified infrastructure for all crawling needs, highlighting how different teams moved from separate crawlers to a shared system that enforces consistent policies. They explain why some fetches bypass robots.txt (like user-initiated actions) and the rising impact of automated traffic from new products and AI agents. With a nod to initiatives like Common Crawl, the episode ends with a look at the road ahead, acknowledging growing internet congestion but remaining optimistic about the web's capacity to adapt. Resources: Episode transcript → https://goo.gle/sotr092-transcript    Chapters:  Chapters: 0:00 - Intro  0:53 - What is a Web Crawler? 3:11 - Building a Minimal Crawler 6:12 - Ethical Crawling: Robots.txt & Host Health 7:42 - BackRub and Early Crawling Challenges 11:02 - The Anatomy of a Search Engine Paper 13:09 - Crawling Across Google Products 16:51 - New Crawlers & User Agent Strings 22:38 - Crawlers Beyond Google 23:17 - The Evolution of Crawlers 26:32 - Bad Actors and Overpowering Servers 27:31 - Reducing the Footprint on the Internet 28:44 - The Future of Crawlers 31:29- Conclusion   Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.   #SOTRpodcast #SEO #SearchOfTheRecord   Speakers: Martin Splitt, Gary Illyes Products Mentioned: Googlebot, Search 

In this episode of Search Off the Record, Gary Illyes and Martin Splitt from the Google Search team dive deep into the foundations of how the web works—specifically HTTP, TCP, UDP, and newer technologies like QUIC and HTTP/3. The two reflect on how even experienced web professionals often overlook or forget the mechanics behind these core protocols, sharing insights through technical discussion, playful banter, and analogies ranging from messenger pigeons to teapots. The conversation spans key concepts like packet transmission, connection handshakes, and the importance of status codes such as 404, 204, and even 418 ("I'm a teapot"). Throughout the conversation, they connect these protocols back to real-world implications for site owners, developers, and SEOs—like why Search Console might report network errors, and how browser or server behavior is influenced by low-level transport decisions. With a mix of humor and expertise, Gary and Martin aim to demystify a crucial part of the internet's infrastructure and remind listeners of the layered complexity that makes modern web experiences possible. Resources: Episode transcript →https://goo.gle/sotr091-transcript    Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO #Http Speakers: Lizzi Sassman, John Mueller, Martin Splitt, Gary Illyes Products Mentioned: Search Console - General  

In this episode of Search Off the Record, Google's Martin Splitt, Cherry, and Gary Illyes discuss the exciting expansion of Search Central Live with the introduction of Search Central Live Deep Dive. They share the history of Search Central events, the inspiration behind creating a more advanced, multi-day experience, and how community feedback helped shape the new format. Cherry explains how the Deep Dive events will offer in-depth sessions, practical workshops, and greater opportunities for networking within the Search community across the APAC region.   The team also offers a behind-the-scenes look at the complexities of planning international events, from budgeting and travel logistics to ensuring high-quality attendee experiences. They highlight plans for new locations including Budapest and South Africa, and outline how interested participants can apply. If you want to hear about the future of Google's Search community events and how to get involved, this episode is a must-watch.   Resources: Episode transcript → https://goo.gle/sotr090-transcript   Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral