Concurrent Web Crawling in Ruby with Async

Comments (1)

vidarh · 6h ago

I've written several crawlers in Ruby over the years, and in terms of raw performance, you can do just fine with just Thread for most uses, but the various tools in Async saves you from writing a lot of the niceties to make it behave well yourself...

Is the UK's energy storage growing fast enough? (theconversation.com)

What's an Agent, Anyway? (frontierai.substack.com)

Show HN: Nuxtbe SaaS Kit – AI Features, Directory Template, and First Customers (nuxtbe.dev)

VideoGameBench from Princeton: Can vision-language models play 90s video games? (vgbench.com)

Show HN: The unified playground for AI image generation (imageninja.ai)

'E-tattoo' could track mental workload for people in high-stake jobs, study says (theguardian.com)

How to do some magic with indexedDB (2014) (codeproject.com)

Show HN: LLM structured outputs hurt performance, we built a model to fix it (osmosis.ai)

Games on ARM64: Introduction to FEX EMU, a fast usermode x86-64 emulator [video] (youtube.com)

Fine-Grained Authorization: Developer Tradeoffs – Gabriel Manor Authcon 2025 [video] (youtube.com)

Wext4 (WextNet): The Decentralized Internet of the Future (cont.ws)

AI-powered fanfiction blurs political reality (semafor.com)

Tinybird: The Supabase for ClickHouse (tinybird.co)

How to self host n8n workflow automation framework (popularowl.com)

The End of the Internet (endoftheinternet.com)

The Role and Functionality of Llms.txt in LLM-Driven Web Interactions (tryprofound.com)

The Legend of John Henry [video] (youtube.com)

Two AIs Compete at Variant Sudoku [video] (youtube.com)

Meta Fired Palmer Luckey. Now, They're Teaming Up on a Defense Contract (wsj.com)

Felt's Sam Hashemi on when to ignore feedback, work trial, designers doing sales (designerfounders.substack.com)

U.S. Cancels Contract with Moderna to Develop Bird Flu Vaccine (nytimes.com)

The Quiet Web: My Guide to Avoiding the Internet's Constant Noise (medium.com)

Kea DHCP: Local Vulnerabilities in Many Linux and BSD Distributions (security.opensuse.org)

Apple to Launch iOS 26, macOS 26 in Major Rebrand Tied to Software Redesigns (bloomberg.com)

The Urgency of Interpretability (darioamodei.com)

Bug Finding Bot vs. Gruyere (jazzberry.ai)

WireWatch: Security of proprietary network encryption in the Android ecosystem (computer.org)

FLUX.1 Kontext – Edit Images with Prompt (bfl.ai)

Tesla shareholders demand Elon Musk commits to 40-hour week (thetimes.com)

A Spyware App Compromised Assad's Army (newlinesmag.com)

My Guidelines for Vibe-Coding (thetechenabler.substack.com)

Show HN: Website Does Not Exist (thiswebsitedoesnotexist.net)

A MEMS grating modulator with a tunable sinusoidal grating (nature.com)

What we know (so far) about CSS reading order (css-tricks.com)

The Knowledge Economy Is Over. Welcome to the Allocation Economy (every.to)

Bittersweet Beginnings of Vanilla: Commercial Success from the Isle of Réunion (smithsonianmag.com)

Show HN: DiscordTCG – A TCG platform for Discord in early development (discordtcg.com)

Discord lures users to click on ads by offering them new Orbs currency (arstechnica.com)

Ethics Guidelines for Trustworthy AI (ec.europa.eu)

Running MCP on a partner bank (griffin.com)

How to Pack for a Long Trip (atlasobscura.com)

Lalrpop: A Parser Generator for Rust (2015) (smallcultfollowing.com)

Meta and Anduril defense startup partner on VR/AR project intended for U.S. Army (cnbc.com)

Self-driving Tesla fails school bus test, hitting child-size dummies (fuelarc.com)

Show HN: MCP Defender – OSS AI Firewall for Protecting MCP in Cursor/Claude etc. (mcpdefender.com)

FLUX.1 Kontext (bfl.ai)

AI in SMB Manufacturing: What Worked and What Did Not (fredlybrand.com)

Cory Doctorow – PyCon 2025 keynote (youtube.com)

Show HN: logtrees, a Blockchain Economic Model Producing UBI and Debt Reduction

Show HN: ClickStack – open-source Datadog alternative by ClickHouse and HyperDX (github.com)

Concurrent Web Crawling in Ruby with Async

Comments (1)