Using Cloudflare? You might unwittingly be blocking your website from search

In July (2026), Cloudflare announced that it was going to categorise web crawlers (bots) in one of three ways: search, agent, or training.

If you want to opt out of training, you can block training bots - easy, right?

Not so...common crawlers such as Googlebot (critical for showing up in Google search) are classed as multi-purpose. They crawl for search purposes, but they also train AI models.

In this discussion, we explain Cloudflare's strange decision to hide settings within the 'security' section and how easily a brand could trap itself into obscurity if it chooses the incorrect settings.

Chapeters

00:00 Introduction to AI training and web crawling issues

01:58 Overview of Cloudflare's categorization of crawlers

03:31 Cloudflare's default settings for training and agent crawlers

04:27 The challenge of 'multi-purpose crawlers'

06:19 What is Cloudflare and how does it work?

08:14 The evolution of Cloudflare's features and internet security

09:28 The rise of AI crawlers and their impact on content security

11:36 The challenge of balancing bot access with website security

15:31 Implications for businesses and website owners

21:32 Managing crawler settings and the role of IT and marketing

24:25 Historical context and the evolution of AI and web scraping

25:48 The decoupling of impressions and clicks due to AI overviews

28:17 Cloudflare hides crawler settings within 'security'

29:39 Next steps for businesses: managing Cloudflare settings

33:20 The importance of transparency and security in bot management

38:30 The future of website visibility and AI training

42:02 Jargon busters and closing thoughts