Show HN: Help improve language coverage in Common Crawl

8 ccgreg 0 6/24/2025, 3:36:53 PM

Hey HN, the Common Crawl Foundation is trying to expand the coverage of our crawl to more languages, regions and cultures, and if you speak a language other than English (LOTE) you can help!

By validating Language Identification data (LangID or LID): https://dynabench.org/tasks/text-language-identification

By contributing urls for our seed crawl: https://github.com/commoncrawl/web-languages

We're also organizing a Workshop on Multilingual Data Quality Signals (WMDQS) with MLCommons and EleutherAI where we have a call for papers open (https://wmdqs.org/cfp/) and a upcoming shared task on language identification (https://wmdqs.org/shared-task/)

I built an app to backup Live Photos from iPhone to external hard drives

Ask HN: Anyone using augmented reality, VR, glasses, helmets etc. in industry?

The scam that is Visa Account Updater

Ask HN: Anyone interested in taking over my indie app?

Ask HN: How to get rid of Gemini?

Tell HN: Beware confidentiality agreements that act as lifetime non competes

Tell HN: Meta developer account suspended

Ask HN: Anyone got advice for someone like me nearing 40s?

A literary magazine accessible only via telnet

Ask HN: How do you keep your SWE skills sharp outside of work?

Ask HN: How to regain the ability to read with focus and learn

Ask HN: How Do You Actually Use Claude Code Effectively?

Ask HN: How do you learn Spanish guitar?

DTN in Rust

Ask HN: What is the maximum indoor temperature beyond which you can't work?

Is "MIT Software License but No AI" Possible?

Free Virtual CS Classes and Tutoring

Are we overfitting our code to trends instead of problems?

Ask HN: Could a "social mode" in AI chats replace social media?

Ask HN: How much would you pay to solve the largest problem you face?

Ask HN: What is your main measure in life?

Ask HN: How is a company like Astral viable, financially?

Ask HN: Anyone using OpenAI's Agent SDK in production?

Ask HN: Are you hesitant to open source your project because LLMs may steal it?

Ask HN: Which low-code app creator should I learn how to master?

Ask HN: Feedback on "QSS" – A Quantized Vector Search Engine in C

Alan Turing's birthday today – born 1912 in London

Ask HN: Tips for hiring? It has been difficult

Ask HN: What useful AI tools do you use every day?

Ask HN: Do you think switching between apps hurts your productivity?

Handling safety/compliance in edtech apps?

Show HN: Help improve language coverage in Common Crawl

Comments (0)