Reduce cache memory on avg 39x

Comments (1)

jonathanehrlich · 3h ago

What if we trained a smaller KV cache for our documents offline? Using a test-time training recipe we call self-study, we find that this can reduce cache memory on avg 39x (enabling 26x higher tok/s and lower TTFT) while maintaining quality. These smaller KV caches, which we call cartridges, can be trained once and reused for different user requests!

Tobacco CEOs testify: "Nicotine is not addictive" (1994) (pbs.org)

Show HN: RenderDay: A GPU-only render farm for Blender (renderday.com)

Triangulate – Turn-based triangle drawing game for 2 to 4 players (laisrast.github.io)

Containerization is a Swift package for running Linux containers on macOS (github.com)

WWDC 25 Keynote Thoughts (taoofmac.com)

Why Icon Rebranded to Sodax and Abandoned Its Layer-1 (coindesk.com)

Apple: AI suffers "complete accuracy collapse" in face of complex problems (theguardian.com)

Omni-Path is back on the AI, HPC menu in a new challenge to Nvidia's InfiniBand (theregister.com)

I Want to Hack to See WHO He Likes

Cal Newport: What Isaac Asimov Reveals About Living with A.I. (newyorker.com)

Baking the Y Combinator from scratch, Part 2: Recursion and its consequences (the-nerve-blog.ghost.io)

Take That, You Hockey Puck (slate.com)

You Can Be Anything [video] (youtube.com)

Agentic AI Summit 8/2 UC Berkeley-Early Bird Registration Now Open Until 6/30

Container: Apple's Linux-Container Runtime (github.com)

The urgency of saving our wallets (bluematt.bitcoin.ninja)

The Secret World of Luxury Real Estate in DC (washingtonian.com)

Headspace competitor Nomadful expands worldwide on the App Store (nomadful.io)

Bears, mice, and moles aren't enough: a better approach for preventing fraud (stytch.com)

Push Science [pdf] (paulrcohen.github.io)

New revolutionary device – 24 EEG channels with Raspberry Pi (pieeg.com)

Domains I Love (ahmedsaoudi.com)

Lovart AI-Powered Visual Design Assistant (lovart.ai)

Show HN: ForkOff – Live Fork and RPC Desync Monitor for Ethereum and L2s

As war rages in Gaza, archaeological looting in the West Bank has spiked (science.org)

Pavel Durov Speaks Out Since His Arrest in France [video] (youtube.com)

Xcode 26 Beta Release Notes (developer.apple.com)

Report claims voting machines were tampered with before 2024 elections (msn.com)

AMD EPYC 4345P 8-Core CPU Performance Review (phoronix.com)

iOS 26 Gets New 'Adaptive Power' Option to Extend Battery Life (macrumors.com)

Fact Sheet: Donald Trump Reprioritizes Cybersecurity Efforts to Protect America (whitehouse.gov)

NASA's Top Technical Challenges Countdown: #2: More Power – Universe Today (universetoday.com)

Ask HN: What are the pros and cons of different AI coding background agents?

The Making of 'New York Street Diaries' by Phil Penman (aboutphotography.blog)

Why Tesla's full self-driving is a scam (carsandhorsepower.com)

My video game sprite generator startup's "stats" page (gametorch.app)

Swift Binary Parsing (github.com)

Show HN: Chrome extension uses VSCode core to run algo trading with AI alerts (github.com)

Genomics of extremotolerant bacteria from spacecraft assembly cleanrooms (microbiomejournal.biomedcentral.com)

The Evasive Evitability of Enshittification (apenwarr.ca)

Show HN: Connect – One SDK, 1000s of integrations for your AI agent (pipedream.com)

China starts mass production of first non-binary AI chip (scmp.com)

How Social Media Brings Out the Worst in Us (greatergood.berkeley.edu)

Apple's "Illusions of Thinking" Paper Isn't Fair to LLMs (theahura.substack.com)

Aircela demonstrates prototype device that turns air into fuel (dezeen.com)

Scientists suggest that cancer is man-made (2010) (manchester.ac.uk)

Updates to Apple's On-Device and Server Foundation Language Models (machinelearning.apple.com)

Should a court break up Google? (npr.org)

OpenJK: Game and Engine Powering Jedi Academy and Jedi Outcast (github.com)

My best advice about preparing for difficult conversations (shaungallagher.pressbin.com)

Reduce cache memory on avg 39x

Comments (1)