Teaching a small embedding model with LLMs to deliver GPT-like semantics in 10ms

Comments (1)

mattmmm · 21m ago

Hey folks! I’m Matt, CEO at Instant Domain Search. Quick summary: we distilled LLM judgments into a 22.7M-parameter embedding model and optimized CPU inference to deliver sub-10ms latency for semantic domain matches (correlation ≈0.87 with GPT-4).

The post walks through our training signal, distillation choices, quantization, index layout, and production latency/CPU learnings.

We’re a small team of 4 engineers building free, wicked fast search tools. AMA or feedback welcome!

Ask HN: I can't remember the name of a AI CLI tool I saw a few weeks ago

Not caring enough about money?

Claude Code fails to fetch or create the GPL v3.0 license when asked

Ask HN: How to take notes and learn from them?

Ask HN: Good resources for DIY-ish animatronic kits for Halloween?

Ask HN: How much can we trust open-source projects or our hardware?

Ask HN: Looking for headless CMS recommendation

Ask HN: Who wants to be hired? (September 2025)

Ask HN: Who is hiring? (September 2025)

Ask HN: How to avoid passive use of AI?

Ask HN: Can an amateur make contributions to pure math or theoretical physics?

Raku.org Chooses Htmx

Ask HN: Are api.nasa.gov, data.nasa.gov down or shutdown?

Why the Technological Singularity May Be a "Big Nothing"

Ask HN: Significant reduction in AI related submissions?

Ask HN: Is Reddit going the way of Stack Overflow?

Automated Workday check in/check out and Microsoft Teams messages monitoring

If AI agents take the jobs, who buys the stuff?

Ask HN: Why does Google word privacy settings like you agree even when off?

Ask HN: What do you think of the new Digg?

Ask HN: Moving from Dev to PM

Ask HN: Is your company still hiring junior engineers?

Ask HN: How long did it take you to learn Git?

New Member Alert

Ask HN: LLM struggles to center div too?

Tell HN: My advice after I applied to 450 positions before getting hired

Ask HN: Useful AI applications in regular businesses?

Ask HN: Why do LLMs struggle with word count?

A16Z scouting ambitious Swiss founders for $1M accelerator

ASIC: Proof-of-Concept Binary Optimizer Reduces Size, More to Come

Ask HN: How do you fight YouTube addiction and procrastination? I'm struggling

Ask HN: When was the last time you visited Stack Overflow?

Ask HN: VSCode AI Autocomplete Woes

File protection: anonymous, open source and fast

Teaching a small embedding model with LLMs to deliver GPT-like semantics in 10ms

Comments (1)