Teaching a small embedding model with LLMs to deliver GPT-like semantics in 10ms

Comments (1)

mattmmm · 25m ago

Hey folks! I’m Matt, CEO at Instant Domain Search. Quick summary: we distilled LLM judgments into a 22.7M-parameter embedding model and optimized CPU inference to deliver sub-10ms latency for semantic domain matches (correlation ≈0.87 with GPT-4).

The post walks through our training signal, distillation choices, quantization, index layout, and production latency/CPU learnings.

We’re a small team of 4 engineers building free, wicked fast search tools. AMA or feedback welcome!

Signal Secure Backups (signal.org)

OpenWrt: A Linux OS targeting embedded devices (openwrt.org)

Job Mismatch and Early Career Success (nber.org)

Firefox 32-bit Linux Support to End in 2026 (blog.mozilla.org)

Experimenting with Local LLMs on macOS (blog.6nok.org)

iPhone Dumbphone (stopa.io)

Will Amazon S3 Vectors Kill Vector Databases–Or Save Them? (zilliz.com)

Dietary omega-3 polyunsaturated fatty acids as a protective factor of myopia (bjo.bmj.com)

NPM debug and chalk packages compromised (aikido.dev)

The story of how RSS beat Microsoft (buttondown.com)

Color NPM Package Compromised (fasterthanli.me)

Browser Fingerprint Detector (fingerprint.goldenowl.ai)

Immich – High performance self-hosted photo and video management solution (github.com)

Escaping the Internet (ryanckulp.com)

Meta suppressed research on child safety, employees say (washingtonpost.com)

A complete map of the Rust type system (rustcurious.com)

Building an acoustic camera with UMA-16 and Acoular (minidsp.com)

14 Killed in anti-government protests in Nepal (tribuneindia.com)

Using Claude Code to modernize a 25-year-old kernel driver (dmitrybrant.com)

The MacBook has a sensor that knows the exact angle of the screen hinge (twitter.com)

The Key Points of Working Effectively with Legacy Code (understandlegacycode.com)

Why Is Japan Still Investing in Custom Floating Point Accelerators? (nextplatform.com)

VMware's in court again. Customer relationships rarely go this wrong (theregister.com)

Formatting code should be unnecessary (maxleiter.com)

Australian startup joins race to build local ChatGPT (afr.com)

Indiana Jones and the Last Crusade Adventure Prototype Recovered for the C64 (gamesthatwerent.com)

We Rarely Lose Technology (2023) (hopefulmons.com)

Integer Programming (2002) [pdf] (web.mit.edu)

GPT-5 Thinking in ChatGPT (a.k.a. Research Goblin) is good at search (simonwillison.net)

Intel Arc Pro B50 GPU Launched at $349 for Compact Workstations (guru3d.com)

Writing by manipulating visual representations of stories (github.com)

After nearly half a century in deep space, every ping from Voyager 1 is a bonus (theregister.com)

What if artificial intelligence is just a "normal" technology? (economist.com)

Look Out for Bugs (matklad.github.io)

Google gets away almost scot-free in US search antitrust case (computerworld.com)

Writing Code Is Easy. Reading It Isn't (idiallo.com)

New Google Court Doc: Open Web Is in Rapid Decline (seroundtable.com)

Creative Technology: The Sound Blaster (abortretry.fail)

Analog optical computer for AI inference and combinatorial optimization (nature.com)

How many dimensions is this? (lcamtuf.substack.com)

The AI bubble argument misunderstands both bubbles and AI (danielmiessler.com)

How inaccurate are Nintendo's official emulators? [video] (youtube.com)

Raspberry Pi Connect (raspberrypi.com)

Garmin beats Apple to market with satellite-connected smartwatch (macrumors.com)

Microdosing GLP-1 drugs became a longevity 'craze' (washingtonpost.com)

How many SPARCs is too many SPARCs? (thejpster.org.uk)

Big business and capitalism are becoming less and less popular with Americans (businessinsider.com)

Exploring Grid-Aware Websites (nicchan.me)

Taking Buildkite from a side project to a global company (valleyofdoubt.com)

How to make metals from Martian dirt (csiro.au)

Teaching a small embedding model with LLMs to deliver GPT-like semantics in 10ms

Comments (1)