The big news here is that Googlebot will be blocked from September 15th onwards by one the "block training" policies, because Google use the same crawler infrastructure for their search index AND for training Gemini:
> Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line with our call for transparency for website owners. Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).
Good. Google's approach here is manifestly predator, unfair, and IMO illegal. They deserve to be in court for this behaviour, and mandating owners give consent for AI training or drop out of Google; which is just a non-starter because they're a search monopoly.
That's exactly what antitrust laws are supposed to do, and I hope at least EU regulators take action. Every single Googlebot crawl in your access logs is a trace for damages.
The irony here is that the people blocking all other crawlers are the ones shoring up their monopoly. If you can't block Googlebot because you need the search traffic but you block everybody else so that nobody other than Google can index your site, how do you expect to ever get any search traffic that isn't from Google?
Search traffic is nose diving due to LLM use. The fundamental calculus with google is you install GA and it helps your SEO has changed and google is riding out what will eventually wither as people start to reevaluate the trade off.
It's nose diving due to Google putting the LLM answer box at the top of the search results. But then isn't it even worse to be blocking every search engine crawler that isn't doing that?
Even companies like OpenAI segregate their training crawlers (GPTBot for general corpus training; OAI-SearchBot for their search index inclusion; and ChatGPT-User for user-driven agentic browser use, etc).
When even OpenAI is more respectful of intellectual property and website owner control than you, that's a problme.
> Even companies like OpenAI segregate their training crawlers (GPTBot for general corpus training; OAI-SearchBot for their search index inclusion; and ChatGPT-User for user-driven agentic browser use, etc).
They have three different user agent strings, anyway. The problem is obviously that you can't tell what someone does with data after you give it to them.
You also don't know that some third party isn't crawling the web with the user agent string "OAI-SearchBot" and then using the results for training or selling the data to the likes of Anthropic or OpenAI without telling them that.
In general attempting to use the user agent string for access control is not going to work and it seems like Google is being the less disingenuous party on this one by not blowing smoke.
I think it's bad, because everybody is desperate to hold onto every last bit of google search traffic they can, so they're going to allow training to do so. Google's predatory, unfair and illegal actions will continue as they have with a few $100 million slaps in the wrist from the EU and a few more white house dinners for their CEO.
This had me in disbelief since the minute I saw it: Google's "AI overview" presumably trained on content from other websites, disincentivizes users from clicking through to those websites..
Except that officially, that is not what they do? It's a dick move not to put the two usecases under separate user agents, but their documentation says you're free to block Google-Extended via robots.txt which is used for training and grounding, while still being included in the search index.
Exclusion from grounding does mean that your site won't get sourced in the AI overview, but I'm not sure what the click through rates are like on those.
The western AI crawlers are generally very-well behaved. GPTBot and ClaudeBot are nice crawlers and I see them respecting my robots.txt.
There are of course plenty of mystery, obfuscated/camouflaged scrapers/crawlers from god knows whom. Thankfully they are easy to spot and ban, although I've definitely thought about deliberating sending them poisoned data.
it’s likely to do with the fact that the parent comment here laid out a thoughtful basis / argument for their “good”, providing some detail to their justification for it.
that’s just my take/feedback, take it or leave it. i won’t be engaging further as i already feel i’m going against the site guidelines with this!
We had googlebot blast a random customer system and almost cause an outage, this is when I first learnt that google will use it for AI training also. It's honestly kind of frustrating also because you then search on it and theres (was) nothing on how you are meant to "correctly" tell google to fuck off, and not use it like that.
Google's web scraping functionality has been acting as a ddos for more than two decades. I've seen literally hundreds of reports of them attacking websites and taking them down, where there's nothing you can do but accept the traffic, or get delisted
This is unfortunately nothing new. There's no correct way to tell them to fuck off, they do not care, and they never will do. People have even taken them to court over this
It is mostly, but it doesn't take a lot of googling to find sites getting ridiculous amounts of traffic from googlebot on google IPs. Its one of the most common complaints about google's search indexing
Lots of people abuse Google Cloud to get a "Google IP" for a fake Googlebot. Why don't you show me a single screenshot from Google Search Console showing a high number of requests to a site that would be counted as a DoS? All requests from the official Googlebot are logged there so if it's such a common problem it must be very easy for you to show me this.
Cloudflare is egregiously bad because of its marketing strategy. It tried to get everyone with any small website to use it, by selling a vague notion of security and charging no monetary price, and it worked. They'll even sell you a domain name to increase lockin. Many people recommend getting domains from cloudflare because apparently they're cheap.
Akamai, Fastly, etc only take big customers who know what they're doing. You need to sign a proper contract with them. They aren't low-friction.
Ideally yes, the TLS termination does not need to happen for caching purposes. Challenge is that in practice every business wants to be sticky and try to provide more functionalities which do require TLS termination. Most people either trust CDN's or they do not understand MitM so it does not concerns them. Plus they are getting certificate management and DDOS prevention capabilities.
> Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line with our call for transparency for website owners. Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).