Techniques scraping vulnerability data with over 100 crawlers (without LLMs)

Comments (1)

louisstow · 3h ago

I wanted to share some of my techniques for managing and scaling 100+ unique web crawlers. I've tried a few approaches but these techniques stand out as having the most impact in terms of quick iteration and being resilient to breaking changes. Interested in hearing different approaches.

The six-month recap: closing talk on AI at Web Directions, Melbourne, June 2025 (ghuntley.com)

GitHub Pages the Easiest Way and the Hard Way (jackd.ethertech.org)

ChatGPT agent might be a big deal (augmentedswe.com)

AI safety index on model scoring released today [pdf] (futureoflife.org)

Hacktoberfest 2025 (hacktoberfest.com)

Supporting Faster File Load Times with Memory Optimizations in Rust (figma.com)

Introduction to ChatGPT Agent (youtube.com)

Stablecoin Giant Tether Is the 'Genius Act's Big Loser (wsj.com)

Latest Intel Engineering Layoffs Lead to an Intel Linux Driver Being Orphaned (phoronix.com)

NIH Is Far Cheaper Than the Wrong Dependency (lewiscampbell.tech)

Show HN: UI builder for React Native with real-time preview and code export (clickly.app)

Bloomberg for Luxury Assets (chr0mata.com)

Asm-Lessons: FFmpeg Assembly Language Lessons (github.com)

Fixing a Direct3D9 bug in Far Cry (2018) (houssemnasri.github.io)

Use Pipx to Install Aider and Other Non-System-Managed Python Packages on Linux (supracarol.github.io)

Firefox Dev: Intel Raptor Lake Crashes Increasing in European Heat Wave (tomshardware.com)

Automate any workflow from sales to videos of your kids growing up (tryospreyai.com)

Hard Tech Businesses Often Look Like Gold Mines (blog.abolyn.com)

Naked Scientists SOS (thenakedscientists.com)

USB-C hubs and my slow descent into madness (2021) (overengineer.dev)

MCP for Fast AI Filesystem Edits (morphllm.com)

Will AI end cheap flights? Critics attack Delta's "predatory" AI pricing (arstechnica.com)

Phenotypic plasticity in cell elongation among closely related bacterial species (nature.com)

CBS Canceling 'Late Show with Stephen Colbert' After Next Season (nytimes.com)

How to go from your target audience to where they are no friction drop feedback (soya-ai.vercel.app)

Show HN: Quickly Generate Creative Banner Designs with AI (bannergraphic.top)

When cells rush to repair DNA, they also know when to stop (thehindu.com)

We're not 'appy. Not 'appy at all. (2013) (gds.blog.gov.uk)

The Purpose of the App Store (asymco.com)

DltHub: LLM-native data platform for Python devs to build/run pipelines (dlthub.com)

China puts new restrictions on EV battery technology to consolidate dominance (cnn.com)

Show HN: JsonDB – A Redis-like KV store with MongoDB query in Go (github.com)

Magic Mushrooms could lengthen your lifespan (independent.co.uk)

Differences in satellite-era sea surface temperature trends across datasets (nature.com)

Toughen Up Dems Says Barack Obama (democracy360.substack.com)

Why the Dalai Lama's succession matters for Trump's China policy (lowyinstitute.org)

LiDO: Discovery of a 10:1 Resonator with a Novel Libration State (iopscience.iop.org)

Sam Altman Outfoxed Elon Musk to Become Trump's AI Buddy (wsj.com)

NIH study finds toxins in 90% of US toddlers (pubs.acs.org)

New Google Chrome Filter May Kill Web Push Notifications (pushpad.xyz)

Has China Become Cool? (chinauncensored.tv)

Reverse Mode Automatic Differentiation (eli.thegreenplace.net)

Why Your AI Coding Assistant Keeps Suggesting Dead Code (and How We Fixed It) (github.com)

Poor Passwords Tattle on AI Hiring Bot Maker Paradox.ai (krebsonsecurity.com)

Out Run: Amiga Edition – Launch Trailer [video] (youtube.com)

Forgejo v12 (codeberg.org)

The problem with 'stabilising' China relations: it hardly makes a difference (aspistrategist.org.au)

Epstein, Trump, and the Smokescreen (victorwynne.com)

Apollo-Soyuz 50th Anniversary: A Handshake that Transformed the Space Race (nytimes.com)

Threads Exporter – Export Threads Followers and Following (chromewebstore.google.com)

Techniques scraping vulnerability data with over 100 crawlers (without LLMs)

Comments (1)