Scrapy_cffi: Async-first, modular web scraping utilities

Comments (1)

funnyStrange · 1h ago

scrapy_cffi keeps a Scrapy-style crawler architecture, supporting async execution and both HTTP & WebSocket requests. CLI support is minimal—Python API is recommended for running spiders.

The real highlights are its utility extensions:

• JSON Extractor – handles standard, embedded, and malformed JSON

• Media Downloader – segmented downloads for videos and large files

• Async Database Managers – Redis, MySQL, MongoDB with automatic retry and reconnection

• Multi-process RPC – quickly register functions, classes, and objects for rapid prototyping without MQ/Redis

These utilities can be used independently or combined into full async crawlers, offering flexibility, rapid prototyping, and easy extensibility.

'It feels like, almost, he's here': How AI is changing the way we grieve (reuters.com)

Fungi found on dying buckthorn could control the highly invasive species (phys.org)

Show HN: We Built AI Sales Chatbot for SaaS Websites (getfluxyai.com)

A store that generates products from anything you type in search (anycrap.shop)

Show HN: Neuralrad Mammo Free tool for breast patients to get secondary opinion (mammo.neuralrad.com)

UK Gov AI 'hit squad' set up to drive efficiency struggles to hire top talent (ft.com)

Is user onboarding better now that we have AI? (medium.com)

Show HN: Silksong map with 280 POI locations,boss fights/collectibles (silksong-map.org)

Show HN: I made cc-filter after Claude kept bypassing its own rules (github.com)

CISA program gave out $20k+ payments to unqualified employees, auditor says (theregister.com)

Aaron Patterson – Rails World 2025 Closing Keynote [video] (youtube.com)

Show HN: I built SaaS churn calculator (free, no login, 1-click JPG export) (sidemail.io)

RL's Razor: Why Online Reinforcement Learning Forgets Less (arxiv.org)

VC Treats Investment Memos Like Short Fiction – Here's What We Can Learn (chiefwordofficer.substack.com)

Program verification is not all-or-nothing (lawrencecpaulson.github.io)

From Snapshots to Standards: Measuring AI Visibility (zenodo.org)

Cursive Reader – Free Handwriting to Text Converter (cursivereader.net)

Bring back your old Mac: 5 ways to refresh the OS on elderly Apples (theregister.com)

Zodiac Sign Is 2k Year Out of Date (nytimes.com)

K2-Think: A Parameter-Efficient Reasoning System (huggingface.co)

Yes, Your Passkeys Can Be Hacked–New Attack 'Breaks the Myth' (forbes.com)

Underused Techniques for Effective Emails (refactoringenglish.com)

School cell phone ban creates surge in JCPS library visits (wave3.com)

How 'overworked, underpaid' humans train Google's AI to seem smart (theguardian.com)

Are EVs causing car sickness – and what can be done? (theguardian.com)

I Made a Mechanical Laptop (youtube.com)

Never steal a hacker's girlfriend's phone: A global network of thieves exposed (english.elpais.com)

Firing the Lorentz Plasma Cannon [video] (youtube.com)

Builtwithhare.org – various projects built with the Hare programming language (builtwithhare.org)

James Webb Space Telescope reveals something strange on comet 3I/ATLAS (livescience.com)

Show HN: AI Roast – Fun AI-generated roasts of websites (ai-roast.jamatrix.io)

Æthelstan 1100 (diamondgeezer.blogspot.com)

Improving state machine code generation (trifectatech.org)

People are losing jobs due to social media posts about Charlie Kirk (npr.org)

Why not analyzing a dataset as if it were an image (with brushes, erasers,)?

Inventor says most humanoid robots today are 'terrifying' (theregister.com)

Lobsters Interview with Susam (lobste.rs)

Betrusted – Building transparent, secure computing from the silicon up (betrusted.io)

Ace and DAZN Shut Down a Major Sports Piracy Site in a "DMCA Ignored" Country (torrentfreak.com)

Evolutionary Software Quality (youtube.com)

Gen-Z Protestors in Nepal Used Discord and TikTok to Overthrow Their Government (twitter.com)

We compared 6 AI models for coding (unscientific, but fun) (blog.kilocode.ai)

New Zealand's plan to save birds? Kill invasive animals (npr.org)

Scrapy_cffi: Async-first, modular web scraping utilities (github.com)

Vimeo to be acquired by Bending Spoons for $1.38B (theverge.com)

Show HN: PromptGit – Git diff and rollback for LLM prompts (github.com)

Seedream 4.0 – Professional AI Image Generator and Editor (seedream-4.app)

The Mass Shooters Are Performing for One Another (theatlantic.com)

Next-generation graph computing with electric current-based approaches (nature.com)

Pivotal Helix: A Quiet Leap in Emergency Response (spectrum.ieee.org)

Scrapy_cffi: Async-first, modular web scraping utilities

Comments (1)