Concurrent Web Crawling in Ruby with Async

Comments (1)

vidarh · 15h ago

I've written several crawlers in Ruby over the years, and in terms of raw performance, you can do just fine with just Thread for most uses, but the various tools in Async saves you from writing a lot of the niceties to make it behave well yourself...

Ask HN: Management wants to talk to my Datalake. What's the best way to do this?

Can we take a moment to appreciate what kind of web experience we are building?

Ask HN: How are people using ChatGPT to increase productivity in personal life?

Ask HN: Career Plateau: Looking for Advice on How to Break Through

Ask HN: What are you working on? (May 2025)

Ask HN: What projects do you donate to?

Ask HN: What's the best tool you've used for sprint planning in 2025?

Ask HN: Are AIs intentionally weak at debugging their code?

Ask HN: Ban streaming services from playing the same commercial constantly?

Google will delete OAuth clients falsely flagged as unused

Al-LLM powered eBPF based security platform

Ask HN: Anyone struggling to get value out of coding LLMs?

Reverse engineering Twitter's new WASM-based "X-XP-Forwarded-For" antibot header

Ask HN: Why that many more US-based companies are hiring "US-only" remote?

Ask HN: Am I old?

Ask HN: Is anyone using AI conversation partners?

Ask HN: How do I start my own cybersecurity related company?

Best Buy is selling a $400 "digital Ethernet" cable for "cleaner, clearer sound"

Lost in AI-translation

SQLite is 25 years old today

Ask HN: Any modern mobile friendly family tree / genealogy builders?

Ask HN: Building LLM apps? How are you handling user context?

Ask HN: Go deep into AI/LLMs or just use them as tools?

What's your "coworker didn't lock screen" procedure?

Ask HN: Is this Tetris variant novel? Would it be fun?

Ask HN: Would you be interested in OpenRouter for MCPs?

Ask HN: What HN posts inspired or changed your perspective the most?

Ask HN: Merge Request Review Agent?

What's been your biggest technical bottleneck as a small startup lately?

It Survives Because It Collapsed: Three AIs, One Residue

Every wondered how Facebook spoofs Gmail message list snippet text?

Ask HN: Why does MCP need to exist?

Ask HN: Do you have a side project you're getting tired of?

Ask HN: Can you give tips on writing a README for project?

ESP32 MCP Server

OSINTGraph – Map Your Target's Instagram Network and Uncover Hidden Circles

Concurrent Web Crawling in Ruby with Async

Comments (1)