Converge (YC S23) well-capitalized New York startup seeks product developers (runconverge.com)

1 points by thomashlvt 2h ago 0 comments

Great Question (YC W21) Is Hiring a VP of Engineering (Remote) (ycombinator.com)

1 points by nedwin 1d ago 0 comments

Coverage Cat (YC S22) Is Hiring a Senior, Staff, or Principal Engineer (coveragecat.com)

1 points by botacode 2d ago 0 comments

Kaizen (YC X25) is hiring engineers to build browser agents that work (kaizenautomation.com)

1 points by michaelssilver 3d ago 0 comments

Infracost (YC W21) hiring first PM to shift $600B cloud spend to proactive (ycombinator.com)

1 points by akh 3d ago 0 comments

Sei (YC W22) Is Hiring a Full Stack Engineer in Chennai, India (ycombinator.com)

1 points by ramkumarvenkat 3d ago 0 comments

Artie (YC S23) Is Hiring Founding AEs (ycombinator.com)

1 points by j-cheong 4d ago 0 comments

Cedana (YC S23) Is Hiring a Systems Engineer (ycombinator.com)

1 points by neelm 4d ago 0 comments

CodeCrafters (YC S22) is hiring first Marketing Person (ycombinator.com)

1 points by sarupbanskota 4d ago 0 comments

PAX Markets (YC W25) is hiring a founding principal hardware (RTL) engineer (ycombinator.com)

1 points by etep 5d ago 0 comments

Sendblue (YC S23) is hiring senior engineers (ycombinator.com)

1 points by rhaber 5d ago 0 comments

Thunder Compute (YC S24) Is Hiring a C++ Systems Engineer (ycombinator.com)

1 points by cpeterson42 10d ago 0 comments

Optery (YC W22) Is Hiring in Engineering, Legal, Sales, Marketing (U.S., Latam) (optery.com)

1 points by beyondd 11d ago 0 comments

QuestDB (YC S20) Is Hiring a Technical Content Lead (questdb.com)

1 points by nhourcard 11d ago 0 comments

Depot (YC W23) Is Hiring a Technical Content Writer (Remote) (ycombinator.com)

1 points by jacobwg 11d ago 0 comments

Firebender (YC W24) Is Hiring (ycombinator.com)

1 points by kevo1ution 11d ago 0 comments

Better Auth (YC X25) Is Hiring (ycombinator.com)

1 points by bekacru 12d ago 0 comments

Kapa.ai (YC S23) is hiring a software engineers (EU remote) (ycombinator.com)

1 points by emil_sorensen 12d ago 0 comments

Spice Data (YC S19) Is Hiring a Product Associate (New Grad) (ycombinator.com)

1 points by richard_pepper 12d ago 0 comments

Extend (YC W23) is hiring engineers to build SOTA document processing (jobs.ashbyhq.com)

1 points by kbyatnal 13d ago 0 comments

Piramidal (YC W24) is hiring a full stack engineer (ycombinator.com)

1 points by dsacellarius 15d ago 0 comments

Mango Health (YC W24) Is Hiring (ycombinator.com)

1 points by zachgitt 16d ago 0 comments

Resolve (YC W15) Is Hiring an Operations and Billing Lead for Construction VR

1 points by ugolino91 16d ago 0 comments

Arva AI (YC S24) Is Hiring an AI Research Engineer (London, UK) (arva.ai)

1 points by OliverWales 16d ago 0 comments

Rejoy Health (YC W21) Is Hiring (ycombinator.com)

1 points by rituraj_rhealth 17d ago 0 comments

Weave (YC W25) is hiring an AI engineer (ycombinator.com)

1 points by adchurch 18d ago 0 comments

CoinTracker (YC W18) is hiring to solve crypto taxes and accounting (remote)

1 points by chanfest22 19d ago 0 comments

Crimson (YC X25) is hiring founding engineers in London (ycombinator.com)

1 points by markfeldner 19d ago 0 comments

Martin (YC S23) Is Hiring Founding Engineers to Build a Better Siri (ycombinator.com)

1 points by darweenist 19d ago 0 comments

Meticulous (YC S21) is hiring in UK to redefine software dev (tinyurl.com)

1 points by Gabriel_h 20d ago 0 comments

Infisical (YC W23) Is Hiring DevRel Engineers (ycombinator.com)

1 points by vmatsiiako 21d ago 0 comments

Can Cloudflare's AI pay per crawl succeed? I doubt it

2 mmarian 4 8/3/2025, 12:06:55 PM developerwithacat.com ↗

Comments (4)

nabla9 · 6h ago

> Neither Cloudflare, nor any other service, will ever be able to block all scrapers. They can make their operations more expensive,

Cloudflare presents like single platform for crawlers. The get the same amount of data as platforms to bock crawlers they don't want. Other big platforms can prevent scrapers effectively when they don't want them Google, Facebook. etc. Nifty new scraper might crawl few million url's before it's detected.

mmarian · 5h ago

Hey! Sorry, didn't quite catch what you meant.

Is it that Cloudlare can always spot crawlers because of the amount of data they collect? Or is it there's always a nifty new scraper that will get away with it?

nabla9 · 3h ago

It's that Cloudflare can always spot crawlers. Few million random urls crawled is nothing, and provides no value for AI companies, they want all.

Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot.

mmarian · 2h ago

> Cloudflare can always spot crawlers

I mention in the post a scraping service that Cloudflare isn't spotting: https://www.scrapingbee.com/blog/how-to-bypass-cloudflare-an...

Plenty of open-source ones as well that could bypass, eg maybe this one that came up in search https://github.com/VeNoMouS/cloudscraper Combine with residential proxies and you're just not going to find them.

> Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot.

Just from a quick Google search:

- LinkedIn: https://brightdata.com/products/datasets/linkedin

- Amazon: https://www.junglescout.com/features/product-database/