Log-Linear Attention

Comments (3)

btilly · 3h ago

I think it would be very good if they can make this work. I suspect that we do something not entirely unlike this, and that is why spaced repetition is so good for stuffing things into our long term memories.

iknownothow · 3h ago

> Log-linear attention replaces the fixed-size hidden state with a logarithmically growing set of hidden states

Does this mean the models can be smaller too (on top of the primary benefit of being faster)?

Lerc · 3h ago

Reduced memory consumption for context perhaps, but hidden state is different from weights. I don't think this would improve the model's capability per model parameter (but as with everything with ML, I wouldn't bet against it until it's been tested)

Ask HN: What is something you deeply care about?

Ask HN: What's your open source stack?

60–70% of YC X25 Agent Startups Are Using TypeScript

Ask HN: Any good tools for viewing congressional bills?

Ask HN: Startup getting spammed with PayPal disputes, what should we do?

Ask HN: What powerful but lesser-known Python libraries are out there?

Best place for small remote gigs?

When Profit Overshadows Community: A Look at Golang Conferences

Ask HN: In-house or outsourced data annotation? (2025)

Ask HN: Has anybody built search on top of Anna's Archive?

Ask HN: What would you work on if you couldn't fail?

Ask HN: Who is hiring? (June 2025)

Ask HN: How do I learn robotics in 2025?

Ask HN: Anyone else feeling increasingly alienated from the industry?

Ask HN: Anyone making a living from a paid API?

Get Your Dev Tool Mentioned by ChatGPT, Gemini Not Just Ranked on Google

Ask HN: How do I learn practical electronic repair?

Ask HN: Do we need a language designed specifically for AI code generation?

Ask HN: Who wants to be hired? (June 2025)

Ask HN: Options for One-Handed Typing

Ask HN: Running AI agents in isolated environments

Ask HN: A Tetris variant with greater tactical and strategic depth?

Tiptap open-sources 10 formerly Pro extensions under MIT license

Ask HN: Is synthetic data generation practical outside academia?

I Built an AI Agent with Gmail Access and Discovered a Security Hole

Summer projects (preferably open source) for college sophomores

Ask HN: What is the best LLM for consumer grade hardware?

Ask HN: What do you put in claude.md and what you leave out?

Ask HN: What Does Your Self-Hosted LLM Stack Look Like in 2025?

Ask HN: What are some good resources for coding best practices?

Ask HN: Should I build a directory product?

Ask HN: How are parents who program teaching their kids today?

O(1) memory, no-preprocessing reachability algorithm for 2D grids

Ask HN: What are your fav/goto decision making hacks/heuristics?

Ask HN: Where do you go for cutting-edge dev news and info?

Reaching my first 100 users without money or audience (at 10K users now)

Ask HN: Walking while working and having meetings

Ask HN: Dealing with Vibe Coding Depression?

Ask HN: Who's Using the Origin Private File System?

How do you store and maintain your CV/resume over time?

Ask HN: What non-AI projects are you working on?

Ask HN: What tools are you using for AI evals? Everything feels half-baked

Ask HN: What's with the repeated job posts on "Who's hiring"?

Log-Linear Attention

Comments (3)