Ask HN: Is there a service that offers Common Crawl as an API?

7 georgehill 3 5/10/2025, 3:15:42 AM

I am trying to do some data analysis work. I don't want the full dataset. I want only two things: give me the hostname, and give me all the pages or URLs with their HTML.

Comments (3)

pluto_modadic · 19d ago

there's index.commoncrawl.org where you can ask for a domain with wildcards.

phillipseamore · 31d ago

Not that I know of but there are various tools like https://github.com/alwalxed/wayurls

georgehill · 31d ago

thank you will check this out

Mac adds native support for Linux containers (appleinsider.com)

Floppy disks and paper strips lurk behind US air traffic control (theregister.com)

Apple's most sweeping software redesign disappoints mainland Chinese consumers (scmp.com)

Ask HN: Any Way to Sidestep Stripe's "alternate currency payout fee"

Lifelike Dolls Cause a Frenzy in Brazil (nytimes.com)

AMD abandons proprietary 3D graphics drivers for Linux (heise.de)

Peter Thiel-backed crypto group Bullish files for U.S. IPO (ft.com)

Mario Kart World review: Getting there is half the game (arstechnica.com)

Show HN: Just vibe coded a TypeScript/Go social app demo (github.com)

io_uring Shows Promising Potential for Linux Accelerator Drivers (phoronix.com)

Green Flash (en.wikipedia.org)

Supercharge your Python library using AST parsing [video] (youtube.com)

First Tesla Robotaxi Spotted Driving Around Austin (insideevs.com)

QuadParts Drone Inventory (github.com)

Meet Liquid Glass [video] (developer.apple.com)

Trump says FEMA to be wound down after hurricane season (reuters.com)

In China, fears grow of an EV financial crisis amid pricing war (cnbc.com)

Custom MCP Client

Apple's NEURLFilter API (textslashplain.com)

Drug deaths plummet among young Americans as fentanyl carnage eases (npr.org)

NSButton is a SwiftUI view now (federated.saagarjha.com)

Our Knowledge System Has Collapsed. Can We Survive Without It? (thefp.com)

Denmark gets more serious about digital sovereignty (world.hey.com)

Comparing Insect ID Apps for Budding Entomologists (commonbynature.com)

Show HN: Make stories for your stuffed animals (stuffiestories.ai)

Where Have the IMO Gold Medallists Ended Up? (xquant.substack.com)

Tell AI your macros, Get a Recipe, Instacart checkout in one click (iamhungry.io)

Think Deeply, Write Clearly: Language, Laughter, and Sensemaking (enterlabyrinth.com)

Rousseau and the Performance of Being Yourself (fee.org)

Slip Sliding Away (cse.umn.edu)

The Diff Cookbook (2023) (naseraleisa.com)

Xiaomi SU7 Ultra sets new Nürburgring record for production EVs (carnewschina.com)

Psychiatric Disorder Diagnosis System Using Wearable ECG Monitors (arxiv.org)

GenAI Criticism and Moral Quandaries (lucumr.pocoo.org)

Ask HN: Are we bike-shedding with prompt engineering?

How ASML, Maker of 'Most Complex Machine Ever Created,' Handles Trade Fights (nytimes.com)

Designing the Ledgers API with Optimistic Locking (moderntreasury.com)

Apple Execs Defend Siri Delays, AI Plan and Apple Intelligence [video] (youtube.com)

Ask HN: Which tools, skills, and knowledge are your alternative to LLMs?

Kerf Bend Wizard (kerfbendwizard.com)

Oodle 2.9.14 and Intel 13th/14th gen CPUs (fgiesen.wordpress.com)

No More Liquid Ass (theverge.com)

The Political Journey of Thomas Mann (hedgehogreview.com)

RepliBench: Measuring autonomous replication capabilities in AI systems (aisi.gov.uk)

US Employment for Prof., Scientific and Tech Svcs: Computer Sys. Design/Related (fred.stlouisfed.org)

Why Rideshare Pricing Should Be a Market (lucaspauker.com)

Advancing underwater energy storage with seabed power solution (waterpowermagazine.com)

Can We Afford Large-Scale Solar PV? (construction-physics.com)

Why Large Language Models Are Still Learning to Find Bugs (jazzberry.ai)

Ryo Minemizu Illuminates the Diversity of Plankton Off the Coast of Japan (thisiscolossal.com)

Ask HN: Is there a service that offers Common Crawl as an API?

Comments (3)