Ask HN: Is there a service that offers Common Crawl as an API?

6 georgehill 2 5/10/2025, 3:15:42 AM

I am trying to do some data analysis work. I don't want the full dataset. I want only two things: give me the hostname, and give me all the pages or URLs with their HTML.

Comments (2)

phillipseamore · 9h ago

Not that I know of but there are various tools like https://github.com/alwalxed/wayurls

georgehill · 8h ago

thank you will check this out

Rollstack (YC W23) Is Hiring TypeScript Engineers (Remote US/CA) (ycombinator.com)

Ciro (YC S22) is hiring a software engineer to build AI agents for sales (ycombinator.com)

Artifact (YC W25) Is Hiring (ycombinator.com)

Thunder Compute (YC S24) Is Hiring a C++ Low-Latency Systems Developer (ycombinator.com)

GovEagle (YC W23) Is Hiring (ycombinator.com)

Motion (YC W20) Is Hiring a Senior Engineers (jobs.ashbyhq.com)

Tabular (YC S24) Is Hiring (ycombinator.com)

Continue (YC S23) is hiring software engineers in San Francisco (ycombinator.com)

Instant (YC S22) Is Hiring a Founding TypeScript Engineer (instantdb.com)

Jiga (YC W21) Is Hiring Engineers (workatastartup.com)

KaiPod Learning (YC S21) Is Hiring VP of Engineering (ycombinator.com)

Hightouch (YC S19) Is Hiring (ycombinator.com)

Helpcare AI (YC F24) Is Hiring (docs.google.com)

Stellar Sleep (YC S23) is hiring a product engineer in SF (ycombinator.com)

OneText (YC W23) Is Hiring a DevOps/DBA Lead Engineer

Toma (YC W24) Is Hiring Engs #3-4 (AI for Automotive) (ycombinator.com)

Waypoint Transit (YC W25) is hiring a software engineer (workatastartup.com)

GroMo (YC W21) Is Hiring (ycombinator.com)

Archil (YC F24) Is Hiring a Distributed Systems Engineer (In-Person, SF)

Modern Realty (YC S24) Is Hiring (workatastartup.com)

Hestus, Inc. (YC S24) Is Hiring an ML Engineer to Revolutionize CAD (ycombinator.com)

Activeloop (YC S18) is hiring a VP of Engineering in Mountain View (on-site) (careers.activeloop.ai)

Optery (YC W22) – Engineering Team Lead and Engineers with Node.js (U.S., Latam) (jobs.ashbyhq.com)

Extend (YC W23) is hiring engineers to build LLM document processing (jobs.ashbyhq.com)

Parity (YC S24) is hiring founding engineers to build an AI SRE (in-person, SF) (ycombinator.com)

Freshpaint (YC S19) is hiring back end and front end engineers (Remote, US only)

MobileBoost (YC S21) Is Hiring a Founding Back End/Platform Engineer (Remote) (ycombinator.com)

Gym Class (YC W22) Is Hiring Character Animation Engineering Lead (ycombinator.com)

Foundry (YC F24) is hiring – Come build a world model for the web

Bild AI (YC W25) is hiring a founding engineer in SF (ycombinator.com)

Tenjin (YC S14) Is Hiring a Senior Ad Attribution Engineer (Ruby, Go) (ycombinator.com)

Ask HN: Is there a service that offers Common Crawl as an API?

Comments (2)