Is Cloudflare Blocking Google From Your Adult Site? How to Check
Cloudflare is the default CDN for adult sites, and for good reason: free TLS, caching, and protection against floods of junk traffic. But one of its protection features cost one of our tubes weeks of Google visibility, and nothing in our own monitoring noticed. Here is what happened and how to make sure it isn’t happening to you.
What we saw
One of our trans tubes had 53,000 URLs in its sitemaps and zero of them indexed. Search Console said the sitemap was last downloaded almost two weeks earlier, while our other sites’ sitemaps were read daily. The few Google visitors it had were all typing the site’s name. Nobody was finding it through search.
Meanwhile the site loaded perfectly in a browser, the server was healthy and real visitors kept arriving. Every check we had was green.
The cause: Bot Fight Mode on the free plan
That zone was the only one of ours with Bot Fight Mode switched on. On the free plan, Bot Fight Mode challenges clients that look automated. A challenge needs JavaScript. Search engine crawlers fetching your sitemap don’t run that JavaScript, so Cloudflare answered 403 to the homepage, the sitemap and every sitemap file for any client without JavaScript. Only robots.txt got through.
The option to let verified bots through (“allow verified bots”) belongs to the paid Super Bot Fight Mode. On the free plan, Bot Fight Mode and being crawled don’t mix.
We proved it both ways: switching it off made the sitemap fetchable immediately, and switching it back on brought the 403s back within seconds.
The two-minute test
Run these from any server or your own computer. You are checking what a client without a browser receives:
curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/
curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/sitemap.xml
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://yoursite.com/sitemap.xmlYou want 200 on all three. A 403, a 503, or a page that says “Just a moment…” means a crawler is being stopped at your edge. Then confirm in Search Console: open Settings, Crawl stats, and look at the response breakdown and the date your sitemap was last read.
Note that the third command uses a fake Googlebot user agent. Cloudflare knows the real Googlebot by its IP addresses, so a fake one may still be blocked while the real one gets in. A 200 on the fake one is a good sign. A block on it is not proof of a problem on its own, but a block on the first two always is.
Settings that are safe
- Security level “medium” (the default) did not affect crawlers on any of our zones.
- WAF rules that block specific bad paths (for example old exploit URLs) are fine, as long as they match paths and not “everything that isn’t a browser”.
- Rate limiting is fine if the limit is far above what a crawler does. Googlebot on a small site fetches a few hundred pages a day, not a thousand a minute.
- Caching HTML is optional. We cache only static files and robots.txt, so our server logs show the true crawler volume.
Settings to avoid on the free plan
- Bot Fight Mode: blocks crawlers as described above.
- “I’m Under Attack” mode left on: same effect, every request gets a JavaScript challenge.
- Country blocks or challenges that include the United States: most Googlebot traffic comes from US addresses.
Don’t block crawlers you don’t like, either
It’s tempting to block every bot that isn’t Google. We decided not to: Yandex, Bing, Seznam and Naver all send real visitors to adult sites, and smaller crawlers are often the ones that feed directories and top lists. Block abuse (floods, exploit scans), not crawlers.
Make it impossible to miss next time
The real lesson is that a site can look healthy from every angle you check and still be invisible. Add one automated check that fetches your homepage and sitemap the way a crawler would, with no JavaScript, and alerts you on anything but a 200. It takes five minutes to set up and would have saved us weeks.