Ask HN: What does Cloudflare's pay-per-crawl mean for web scrapers?

3 jjangkke 0 7/1/2025, 9:58:49 PM

Cloudflare is rolling out pay-per-crawl scheme for AI Agents but this will obviously impact web scraping as 20% of the web is behind Cloudflare.

There are already numerous ways to bypass CF but curious how they would be able to stop web scraping?

Have they built up a giant database of residential proxies from BrightData and others over the years and can detect if these are proxies?

I'm curious to know what web scrapers on HN thinks about this move and what your plans are.

Show HN: Learning Rust with Neovim and LLMs (lorentz.app)

Test (leetcybersecurity.com)

IBM Lays Off 8k Employees for AI, Only to Rehire Just as Many Soon After (dailygalaxy.com)

A mostly allocation-free optional type (joachim-breitner.de)

Why Big Tech Turned Against Democrats – and Democracy (paulkrugman.substack.com)

On the Feasibility of Deduplicating Compiler Bugs with Bisection (arxiv.org)

Object Storage for KV cache offloading (acceleratedcloudstorage.com)

The Hell of FFIX Tetra Master (xvw.lol)

Using Playwright MCP with Claude Code (til.simonwillison.net)

BruhGrow Tools (bruhgrow.com)

K-Scale Labs Personal Robot [video] (youtube.com)

Logic Gates Implemented in Microsoft Paint (github.com)

When we die do we still have any of the original cells from our birth? (quora.com)

I Built Neural AST That Works – It Fixed 274 TypeScript Errors Then Went Rogue

Printing the web: making webpages look good on paper (piccalil.li)

Cutting AWS Costs by over 50%: Case Study on Moving OpenSearch and RDS to EC2 (kevinhq.com)

Changes to the Microsoft for Startups program – July 1, 2025 (learn.microsoft.com)

Building a Custom Klaviyo Subscription Plugin for WordPress (kevinhq.com)

Building a professional design tool on the web (2015) (madebyevan.com)

Large Atomic Model's checkpoint has reached over 10G (huggingface.co)

Apple Accuses Former Employee of Stealing Vision Pro Secrets (macrumors.com)

Can AI help in tackling mental health issues? (hawkradius.com)

Adding SQLLogicTest Support to Mochi (github.com)

Canada's "Strong Borders Act" (Bill C-2) Contains Four Mass Surveillance Trojans (easydns.com)

Elasti: Lightweight Scale-to-Zero for Existing Kubernetes Services (github.com)

Removing Hamas from power a condition for Saudi normalization (israelnationalnews.com)

U.S. Air Force Releases New UAP Video Depicting Metallic-Looking Sphere (uapregister.substack.com)

The AI-Native Software Engineer (addyo.substack.com)

Jury says Google must pay California Android smartphone users $314.6M (theguardian.com)

Show HN: Waste managment optimization with detectron2 and flask (github.com)

Custom fanless 7.5L AMD Strix Halo system (WIP) (smallformfactor.net)

OpenBAO v2.3.1 (github.com)

Big Tech is crushing middle managers (businessinsider.com)

John M. Cioffi (cioffi-group.stanford.edu)

A Class 99 – The UK's Most Powerful Locomotive [video] (youtube.com)

Show HN: Browser Based GPU Benchmarker (aistats.fyi)

Writing Is More Than a Skill – It's a Way of Joining the World (domofutu.substack.com)

Did the 'Deep State' Invent the UFO Craze? (nytimes.com)

Context Engineering vs. Prompt Engineering (vatsalshah.in)

Why Tiny Is the Future of Agentic AI (vatsalshah.in)

Artificial Super Intelligence: Forecasts and What Comes After AGI (vatsalshah.in)

Delete Tests (andre.arko.net)

Ask HN: From 10 –>100 customers–distribution playbook for engineers B2B founders

Intel's new CEO explores big shift in chip manufacturing business (reuters.com)

Show HN: I built an AI-powered tool to generate and schedule tweets (flowy.business)

The Rise of Empress (2021) (old.reddit.com)

Show HN: “Text me when my Congress member votes” – daily alerts, (not a app) (sites.google.com)

Optimizing typography of insect labels using free fonts and free software (2012) [pdf] (akentsoc.org)

Show HN: Lightweight Bar to Psi Converter (bartopsi.xyz)

Vibe coding a Perl interface to a foreign library (chrisarg.github.io)

Ask HN: What does Cloudflare's pay-per-crawl mean for web scrapers?

Comments (0)