Malicious Bots Pose as Googlebot to Bypass Website Security

Chris Siebenmann, a longtime technical blogger behind Wandering Thoughts and a Unix system administrator at the University of Toronto, says he has observed a sharp increase in malicious bot traffic impersonating Googlebot.

“This June, the floodgates opened,” Siebenmann wrote on his blog.

“For weeks, I’ve been seeing hundreds of requests a day claiming to be Googlebot, and on a few days, thousands of requests. The requests come from a wide range of IP addresses across different providers, which I believe are mostly or entirely cloud and hosting companies.”

Google’s web crawlers have long benefited from preferential treatment because appearing in Google Search drives traffic to websites. As a result, many websites, including those that aggressively block crawlers, are configured never to block Googlebot.

Siebenmann discovered the campaign by chance after configuring his websites to block any Googlebot requests originating outside Google’s published IP address ranges. This made it easy to distinguish fake Googlebot traffic from legitimate crawler requests.

Attempts to impersonate Googlebot and other major search engine crawlers have been seen before, but Siebenmann says they were previously rare. Until June, only occasional fake requests reached his websites.

He believes the activity is part of a single large-scale campaign. Numerous IP addresses are used simultaneously, with each making only a handful of requests while posing as Googlebot. If those requests fail, some of the same systems retry using a different User-Agent string.

The IP addresses span multiple providers, with most requests originating from HostRoyale, M247, Latitude.sh, Web2Objects, and AWS.

• Googlebot is Google’s primary web crawler, responsible for crawling and indexing web pages before they appear in Google Search results.

Google has published resources to help developers verify requests from its crawlers and data-fetching services. Website administrators can perform a one-time DNS lookup using command-line tools or use Google’s published IP address ranges for automated verification.

“This is useful if you’re concerned that spammers or other troublemakers are accessing your site while claiming to be from Google,” Google’s documentation states.

HTML bot traffic has already surpassed human traffic and is becoming a real cost for website operators rather than just a nuisance. Bots consume bandwidth, slow websites for legitimate visitors, scrape content to train large language models, and provide little or no value to site owners.

At the same time, some website operators, including Siebenmann, are beginning to question whether Googlebot still deserves special treatment. As Google Search shifts from sending users to websites through traditional search results toward AI-generated answers, publishers receive significantly less traffic while AI responses often misattribute or misquote their content.

“Google Search is based on a social contract: their bots get to crawl our sites, index our sites, and show snippets from our sites because, and only because, they send people to our sites. Our sites, our words, with our design, our links, our context, and our aesthetics are shared the way we want them to be shared,” computer scientist Paul Khuong previously wrote on Mastodon.

Some website owners now believe Google is no longer upholding its side of that social contract.