Watching these two behemoths wrestle over the future of a space we all share, and wondering if they will need to loop in regulators on one side or another, convinces me that we shouldn't have gifted all of our digital infrastructure to just 2 companies. Inlcuding our economy, healthcare, government, and civil infrastructure. We've put all our eggs into only a couple of very greedy, impossible to audit baskets. We've really done this all to ourselves. We've raced ourselves all the way to the bottom.
imilk · 10m ago
There is nothing stopping other CDN/DNS providers from implementing similar services and tools to what Cloudflare offers. Part of the reason CF has become so popular is because so many of their competitors don't offer nearly the same convenience for routine tasks & protection.
philipallstar · 17m ago
> we shouldn't have gifted all of our digital infrastructure to just 2 companies
We didn't. Just as we didn't gift all our chocolate-making infrastructure to Hershey's and Cadbury's.
eddythompson80 · 30m ago
It used to be just Ma Bell
coldpie · 38m ago
After 50 years of effectively zero activity, we had some glimmers of anti-trust enforcement under the Biden admin. But then eggs were expensive in the summer of 2024, so we decided it was actually no problem for these half-dozen companies to control our speech and economy, and here we are.
thisislife2 · 49m ago
I didn't understand what Cloudflare is saying, so perhaps somebody else could clarify - Google already crawls sites to index them for its search. So why would Google need a separate AI crawler? Cloudflare (according to the article) claims they already "block" Gemini - does that mean Google does operate a separate AI crawler? Why though, when they already have a huge index? What does an AI crawler do different from a search engine (indexing) crawler?
imilk · 35m ago
Google has made it very difficult to completely block their AI crawling by also using the standard googlebot search crawlers to feed data into their AI overviews and other AI features within Google search. Google says there is a workaround but it also blocks your site from fully indexing in Google search. This also also all covered in the article though.
> What does an AI crawler do different from a search engine (indexing) crawler?
Many people don't want the extra bot traffic hitting their site that comes from AI, especially when AI chat & AI overviews in Google provide such a small amount of traffic in return and that traffic pretty much always has horrendous conversion rates (personally seen across multiple industries).
fathomdeez · 22m ago
It doesn't seem like the extra traffic is the issue. People don't want Google's AI from reading and summarizing their data and thus preventing clickthroughs. Why would I click on your site if google did all the work of giving me the answer ahead of time?
dathinab · 17m ago
> It doesn't seem like the extra traffic is the issue.
it really can be, Anubis AI crawler detection was create mainly because of "way to many AI bot requests" to quote
> This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies.
imilk · 9m ago
Both are an issue. People don't want AI overviews cannibalizing their website traffic. People also don't want AI bots spamming their website with outrageous numbers of requests everyday.
dathinab · 20m ago
which again comes back to
this is a problem which needs regulatory action, not one which should be solved by a quasi monopoly forcing it onto anyone but another quasi monopoly which can use their monopoly power to avoid it
require
- respecting robots.txt and similar
- require purpose binding/separation (of the crawler agent, but also the retrieved data) similar to what GDPR does
- require public agent purpose documentation and stable agent identities
- disallow obfuscation of who is crawling what
- do enforce it
and sure making something illegal doesn't prevent anyone from being technically able to do it
but now at lest large companies like Google have to decide weather they want to commit a crime, and the more they obfuscate that they are doing it the more there is prove it was done with a lot of bad faith, i.e. the higher judges can push punitive damages
combine it with internet gateways like CF trying to provide technical enforcement and you might have a good solution
but one quasi monopoly trying to force another to "comply" with their money making scheme (even if it's in the interest of the end user) smells a lot like you can have a winnable case against CF wrt. unfair market practices, monopoly power abuse etc...
imilk · 15m ago
I find it wild that you focus on CF being a monopoly here when they are providing tools that help publishers not have all of their content stolen and repurposed. AI companies have been notorious over the last few years for not respecting any directives and spamming sites with requests to scrape all of their data.
There is also nothing stopping other CDN/DNS providers spinning up a similar marketplace to what CF is looking to do now.
Spivak · 29m ago
In the specific case of Google would there be any additional traffic that isn't just the normal googlebot? I can't imagine they would bother crawling twice for every site on the internet.
imilk · 19m ago
There are about a dozen Google crawlers that can hit your website for different reasons:
Google-Extended is what is associated with AI crawling, but GoogleBot also crawls to produce AI overviews in addition to indexing your website in Google search.
While the number of crawlers and their overlapping responsibilities makes it difficult to know which ones you can safely block, I should also say that pure AI company bots behave 1000x worse than Google crawlers when it comes to not flooding your site with scraping requests.
Palomides · 40m ago
cloudflare wants to sell a "AI bots have to pay to access your site" service, so they need to argue it out with google one way or another
the reality of the tech is irrelevant
beejiu · 38m ago
Google does not use a separate crawler, but they do use the robots.txt semantics to control what's allowed in the AI "index" separately from the search index. From the docs:
> Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.
The assumption is that "Google" is a good player, not a malicious one. Same idea as robots.txt. It's only there to configure the "good guys" behavior not to prevent malicious actors.
So I'm assuming Cloudflare is basically asking Google to split its crawler's UA and distinguish between search and AI overview and respect something akin ot robots.txt
dawnerd · 41m ago
Google does crawl with a different crawler. Makes no sense other than clearly two different teams with different goals.
npollock · 44m ago
it could be the "grounding with google search" feature when using Gemini models
Snoozus · 49m ago
Is this how the democratic process works now?
Cloudflare threatens Google to pass a law?
imilk · 6m ago
What has been democratic about how the internet has evolved over the last 2 decades? Because as far as I can see, the internet has undergone a massive centralization into the hands of a few players with practically no regulation. Especially Google, which can make decisions such as adding AI Overviews to search results leading to millions of websites seeing a ~25% drop in organic traffic in the last few months.
thisislife2 · 49m ago
Do you have any more doubts who run the US and makes laws there?
handfuloflight · 46m ago
Hard to believe Cloudflare has more sway than Google.
deadbabe · 36m ago
Cloudflare is positioned to become a very powerful company. Well worth investing in.
procflora · 32m ago
Guess they've been pretty successful converting scummy Enterprise plan upsells to lobbyist retainers.
pjc50 · 42m ago
Tech regulation, or lack thereof, tends to be "biggest pile of money wins", but in this case there's already large anti-Google and anti-AI constituencies which CF may be able to mobilize. Especially in the EU.
eviks · 23m ago
"Always has been"
radicalbyte · 34m ago
Can we block them for customers? As they're extremely inaccurate. For example the Google "summary" for "stop killing games started by" was returning "Scott Ross". It was started by Ross Scott. The first search results provide the correct example.
CaptainFever · 34s ago
You can use the element picker in uBlock Origins.
msgodel · 42m ago
I've still never understood the complaint here. Robots are part of the web, the whole point of HTML is that robots can read it.
pjc50 · 39m ago
The whole point of publication is so that humans can read it. Robots not so much, especially if they're not paying customers. This is the distinction between how the web works technically and how it works socioeconomically.
This is the next iteration of things like the news snippet case. Publishers are not happy that Google crawls their content (at their expense) and then republishes it on their own site, while serving ads around it and getting user data, without cutting in the publisher who originally made it. And, for what little it's worth, owns the copyright.
msgodel · 37m ago
Robots don't exist for their own sake, at the end of the day they are user agents for some group of humans.
Again it sounds like the people who are upset by this really want to publish images rather than web pages.
imilk · 28m ago
> Again it sounds like the people who are upset by this really want to publish images rather than web pages.
More like people don't want to lose money because a 3rd party stole all of their content, and then repurposed it to show people before they visit their website.
marginalia_nu · 41m ago
It's a cost thing. It costs more to render a website than it does to consume it. When you have some bot traffic mixed in with human traffic, that is fine.
When you have egregious bot traffic, say 10k requests per minute sustained load, it becomes a real problem for webmasters.
eviks · 24m ago
Because you remain at the top of an extremely shallow perspective. The fact that robots are a part is not relevant, the relevant issue is how some of those robots behave, and what the consequences of that behavior are.
josefresco · 30m ago
Have you ever been responsible for the performance and security of a publicly accessible web server? I'll accept robots indexing my content if they play nicely. Unfortunately most do not, even from major vendors.
msgodel · 15m ago
Not a web server but yeah we dealt with it by black listing patterns (IPs, requests etc) from misbehaving domains.
We never distinguished automations from people though, that makes no sense on the internet.
imilk · 26m ago
Having perplexity or other AI bots go haywire and sending 10s of thousands of requests per minute to your website (despite you having a robots.txt blocking them) is a giant pain in the ass. Not only does your server costs go up, but your analytics and attribution reports start to look messed up because of all the bot traffic.
msgodel · 11m ago
Yeah obviously if they're being abusive that's a problem but that's not what the article seems to be talking about.
imilk · 1m ago
Well besides them being abusive, the other issue is that AI overviews and answer boxes cannibalize traffic to websites, leading to less conversions for the original content producers. This is pretty well established across industries at this point:
HTML was not made for robots to read (a semantic web or an internet of data), it just so happens that crawlers try to index things in meaningful ways. It's an un-ordered blob of unstructured data.
alganet · 15m ago
Machine-readable does not mean centralized.
dehrmann · 26m ago
That's why we need the Captchaweb. It's like the web, but everything is in captcha text.
mattl · 30m ago
A lot of people don't want AI slop and don't want the companies pushing it to crawl their websites.
andy99 · 29m ago
We're seeing the free (libre) internet destroyed in front of us.
imilk · 23m ago
I come to the exact opposite conclusion. A few large companies scraping and repurposing original content from publishers kills original content on the internet because it takes out the ability to earn a livelihood from creating original content or running your own ecommerce store that is not tied to a mega-company's platform.
handfuloflight · 47m ago
So this boils down to Cloudflare, supposedly having more regulatory capture than Google.
nathanyz · 28m ago
The thing that he doesn't mention is that as soon as they do something legislatively and announce routes there, etc.....well Google just won't crawl those sites. It turns into a game of whether you would like 0 traffic from Google, or allow them to use your content both for search results and AI summaries.
Google is the bringer of traffic and if you want it, then you play by their rules. I don't like that the web is in that position, but here we are.
kingstnap · 49m ago
I don't understand how AI scrapers make up such a large percentage of traffic to websites, as people claim it does.
In principle, if you post a webpage, presumably, it's going to be viewed at least a few dozen times. If it's an actually good article, it might be viewed a few hundred or even thousands of times. If each of the 20 or so large AI labs visit it as well, does it just become N+20?
Or am I getting this wrong somehow?
NitpickLawyer · 40m ago
> I don't understand how AI scrapers make up such a large percentage of traffic to websites, as people claim it does.
I think a lot of people confuse scraping for training with on-demand scraping for "agentic use" / "deep research", etc. Today I was testing the new GLM-experimental model, on their demo site. It had "web search", so I enabled that and asked it for something I have recently researched myself for work. It gave me a good overall list of agentic frameworks, after some google searching and "crawling" ~6 sites it found.
As a second message I asked for a list of repo links, how many stars each repo has, and general repo activity. It went on and "crawled" each of the 10 repos on github, couldn't read the stars, but then searched and found a site that reports that, and it "crawled" that site 10 times for each framework.
All in all, my 2 message chat session performed ~ 5-6 searches and 20-30 page "crawls". Imagine what they do when traffic increases. Now multiply that for every "deep research" provider (perplexity, goog, oai, anthropic, etc etc). Now think how many "vibe-coded" projects like this exist. And how many are poorly coded and re-crawl each link every time...
marginalia_nu · 34m ago
Yeah it seems the implementation of these web-aware GPT queries lacks a(n adequate) caching layer.
Could also be framed as an API issue, as there is no technical limitations why search provider couldn't provide relevant snapshots of the body of the search results. Then again, might be legal issues behind not providing that information.
dsign · 40m ago
If you run a website, you'll realize it's very difficult to get human traffic. Worse, trying to understand what those eyeballs are doing is a swamp; there are legitimate privacy concerns for example. Maybe all you care about is if your articles about sewing machines are getting more traction than your articles about computing Pi, but you can't get that without navigating all the legal complications of your analytics platform of choice, who wants to make sure you suffer for not letting them collect private information on your visitors to sell to third parties and to dump ads onto your visitors. Were it not for the bots, you would be fine just by running grep on your access logs. But no, bot traffic leaves noise everywhere; and for small websites that noise is more than enough to bury the signal and to be most of the traffic bill.
imilk · 31m ago
On multiple client sites that have > 1 million unique real visitors per month, we are seeing some days where ~25-30% of requests is from AI crawlers. Thankfully we block almost all of this traffic. But it is a huge pain because it add an addition load to your server and messes up your analytics data for what is a terrible return - traffic from AI sources has a horrendous conversion rate, even worse than social media traffic conversion rates.
csomar · 45m ago
I don't understand it either. I track requests and AI crawlers are there but not as abusive as people claim. Most annoying requests are from hackers who are trying to find my ".git" directory. But I highly doubt these guys will respect any rules anyway.
handfuloflight · 39m ago
Your speculation assumes a low page count.
abxyz · 32m ago
Pages are dynamic, they change often: if a page is worth scraping once, it is worth scraping again and again and again to keep up to date with any changes.
delfinom · 44m ago
I have a symbol server hosting a few thousand PDBs for a foss package.
Amazonbot is every day trying to scan every single PDB directory listed. For no real reason. This is something causing 10k+ requests each day when legitimate traffic sits at maybe 50 requests a day.
wat10000 · 49m ago
Maybe they're vibe-coding the scrapers.
mattl · 31m ago
Post a URL to a page on your website on something like Mastodon and tail your logs.
ninetyninenine · 49m ago
They are going to have to use the law. Are you kidding? You block ai for Google while every other agent is able to bypass that block and crawl your site? How will Google stay competitive for AI? Of course Google would rather have you do it by law. You block it for them you better do it for every single one of their competitors too.
Also guaranteed even if you do block by law there will be actors who will ignore the law. For example people outside of the US. Those people will as a result likely build better AI because they have access to more training data.
First example of this is of course China. One thing with China is there’s no holy sanctity behind data. Whatever is made is copy-able and goes to the public domain regardless of anything. It both causes China to both exceed the US and to be slightly less innovative at the same time.
If these laws come to pass you bet your ass China will be exceeding the US in AI like they already have with stem cell research.
delfinom · 31m ago
Ah ok, yet the megacorps continue to reap profits off stealing labor and work from the peasants.
I propose we just remove the charades and require us peasants pay these megacorps 20% of our annual income yearly at tax time.
ninetyninenine · 13m ago
Who pays the megacorps to use AI? The peasants. Business only succeeds if the peasantry want it. So if businesses are fucking over the peasantry what’s really going on is that the businesses are just middlemen for peasantry fucking themselves. In the end the megacorps are just a feedback loop to the peasantry. That’s the irony. You like to blame corporations but it’s just a mirror to yourself.
Still doesn’t hurt to lower taxes for peasants and heighten taxes for corps. That’s how you prevent wealth inequality.
We didn't. Just as we didn't gift all our chocolate-making infrastructure to Hershey's and Cadbury's.
> What does an AI crawler do different from a search engine (indexing) crawler?
Many people don't want the extra bot traffic hitting their site that comes from AI, especially when AI chat & AI overviews in Google provide such a small amount of traffic in return and that traffic pretty much always has horrendous conversion rates (personally seen across multiple industries).
it really can be, Anubis AI crawler detection was create mainly because of "way to many AI bot requests" to quote
> This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies.
this is a problem which needs regulatory action, not one which should be solved by a quasi monopoly forcing it onto anyone but another quasi monopoly which can use their monopoly power to avoid it
require
- respecting robots.txt and similar
- require purpose binding/separation (of the crawler agent, but also the retrieved data) similar to what GDPR does
- require public agent purpose documentation and stable agent identities
- disallow obfuscation of who is crawling what
- do enforce it
and sure making something illegal doesn't prevent anyone from being technically able to do it
but now at lest large companies like Google have to decide weather they want to commit a crime, and the more they obfuscate that they are doing it the more there is prove it was done with a lot of bad faith, i.e. the higher judges can push punitive damages
combine it with internet gateways like CF trying to provide technical enforcement and you might have a good solution
but one quasi monopoly trying to force another to "comply" with their money making scheme (even if it's in the interest of the end user) smells a lot like you can have a winnable case against CF wrt. unfair market practices, monopoly power abuse etc...
There is also nothing stopping other CDN/DNS providers spinning up a similar marketplace to what CF is looking to do now.
https://developers.google.com/search/docs/crawling-indexing/...
Google-Extended is what is associated with AI crawling, but GoogleBot also crawls to produce AI overviews in addition to indexing your website in Google search.
While the number of crawlers and their overlapping responsibilities makes it difficult to know which ones you can safely block, I should also say that pure AI company bots behave 1000x worse than Google crawlers when it comes to not flooding your site with scraping requests.
the reality of the tech is irrelevant
> Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.
https://developers.google.com/search/docs/crawling-indexing/...
So I'm assuming Cloudflare is basically asking Google to split its crawler's UA and distinguish between search and AI overview and respect something akin ot robots.txt
This is the next iteration of things like the news snippet case. Publishers are not happy that Google crawls their content (at their expense) and then republishes it on their own site, while serving ads around it and getting user data, without cutting in the publisher who originally made it. And, for what little it's worth, owns the copyright.
Again it sounds like the people who are upset by this really want to publish images rather than web pages.
More like people don't want to lose money because a 3rd party stole all of their content, and then repurposed it to show people before they visit their website.
When you have egregious bot traffic, say 10k requests per minute sustained load, it becomes a real problem for webmasters.
We never distinguished automations from people though, that makes no sense on the internet.
https://ahrefs.com/blog/ai-overviews-reduce-clicks/
Google is the bringer of traffic and if you want it, then you play by their rules. I don't like that the web is in that position, but here we are.
In principle, if you post a webpage, presumably, it's going to be viewed at least a few dozen times. If it's an actually good article, it might be viewed a few hundred or even thousands of times. If each of the 20 or so large AI labs visit it as well, does it just become N+20?
Or am I getting this wrong somehow?
I think a lot of people confuse scraping for training with on-demand scraping for "agentic use" / "deep research", etc. Today I was testing the new GLM-experimental model, on their demo site. It had "web search", so I enabled that and asked it for something I have recently researched myself for work. It gave me a good overall list of agentic frameworks, after some google searching and "crawling" ~6 sites it found.
As a second message I asked for a list of repo links, how many stars each repo has, and general repo activity. It went on and "crawled" each of the 10 repos on github, couldn't read the stars, but then searched and found a site that reports that, and it "crawled" that site 10 times for each framework.
All in all, my 2 message chat session performed ~ 5-6 searches and 20-30 page "crawls". Imagine what they do when traffic increases. Now multiply that for every "deep research" provider (perplexity, goog, oai, anthropic, etc etc). Now think how many "vibe-coded" projects like this exist. And how many are poorly coded and re-crawl each link every time...
Could also be framed as an API issue, as there is no technical limitations why search provider couldn't provide relevant snapshots of the body of the search results. Then again, might be legal issues behind not providing that information.
Amazonbot is every day trying to scan every single PDB directory listed. For no real reason. This is something causing 10k+ requests each day when legitimate traffic sits at maybe 50 requests a day.
Also guaranteed even if you do block by law there will be actors who will ignore the law. For example people outside of the US. Those people will as a result likely build better AI because they have access to more training data.
First example of this is of course China. One thing with China is there’s no holy sanctity behind data. Whatever is made is copy-able and goes to the public domain regardless of anything. It both causes China to both exceed the US and to be slightly less innovative at the same time.
If these laws come to pass you bet your ass China will be exceeding the US in AI like they already have with stem cell research.
I propose we just remove the charades and require us peasants pay these megacorps 20% of our annual income yearly at tax time.
Still doesn’t hurt to lower taxes for peasants and heighten taxes for corps. That’s how you prevent wealth inequality.
Cloudflare to introduce pay-per-crawl for AI bots
https://news.ycombinator.com/item?id=44432385