Ask HN: How to build services for FOSS projects when ToS forbids scraping?

3 ATechGuy 6 8/20/2025, 9:50:10 PM

I'm working on a service for FOSS developers to help enforce code license compliance and make projects more sustainable.

The challenge: many websites' Terms of Service explicitly prohibit scraping, crawling, or automation. At the same time, the information needed (repos, dependencies, metadata) is often available only through those sites.

For those who've built tools around open source ecosystems:

* How do you navigate ToS restrictions while still delivering value to users?

* Do you focus on official APIs only, even if they're limited?

* Are there established legal/technical best practices for this situation?

* How to balance compliance with ToS and the mission of supporting FOSS?

Curious to hear what others have done (or seen work) in this space.

Comments (6)

like_any_other · 3h ago

The law around scraping is unfortunately complex, but it's plausible your use of it (legal compliance) could be legal regardless of ToS ("you're not allowed to use automated tools to check if we're breaking your license" is quite fishy):

https://legalclarity.org/is-web-scraping-legal-a-look-at-the...

https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

Or you could base your business in Denmark: https://en.wikipedia.org/wiki/Web_scraping#European_Union

ATechGuy · 3h ago

Thanks for sharing the links! You're right, the legality around scraping is nuanced :( Use cases like license compliance might fall on the defensible side.

toomuchtodo · 3h ago

I’m unsure what your jurisdiction is, but if you’re in the US, speak with an attorney and get an attorney opinion letter. You’ll at least be able to demonstrate that you were acting in good faith.

ATechGuy · 3h ago

Thanks!

bigyabai · 3h ago

You're making a business, right? Pay for official API access.

ATechGuy · 3h ago

Right now the cost is a blocker (I'm bootstrapped, no funding yet). On top of that, sometimes official APIs either don't exist or don't expose the full dataset needed for license checks, which makes things tricky.

Ask HN: Why does the US Visa application website do a port-scan of my network?

Ask HN: Where are the best online gathering places for humans?

Dagger and opencode and agnostic agents and SSH app = most portable dev kit

Tell HN: Somebody please make a faster task management system

Warp sends a terminal session to LLM without user consent

Ask HN: How do you get your devs to understand your customers?

What services or apps did you see abroad and wonder: why don't we have them?

AI might worsen diagnostics for physicians

Ask HN: What to do with a pure mathematics degree?

Ask HN: How can I use AlarmKit at expo/React Native?

When to Open Source

Can someone review my Laravel code?

VC-backed company just killed my EU trademark for a small OSS project

F-Droid build servers can't build modern Android apps due to outdated CPUs

Ask HN: Do you still bookmark websites?

Ask HN: Are we allowed to discuss Israel on HN?

Ask HN: SaaS Bookkeeping and Accounting

Ask HN: Tell me about the best programmer you worked with

Ask HN: Are there any good educational computers for children that aren't toys?

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

Ask HN: Imagine coding LLM's 1M times faster; what uses might there be?

Ask HN: Has there been an increase in online censorship due to AI search's rise?

My First $500 MRR – My thoughts and learnings

Ask HN: Founding Engineer/CTO (equity) for music rights/data infra (UK/US)

AI Is Not a Dev

Ask HN: How do you tune your personality to get better at interviews?

Riff.nvim: Play YouTube music inside Neovim

Ask HN: How to build services for FOSS projects when ToS forbids scraping?

Comments (6)