There's not much incentive to subsidize prices for OpenRouter providers for example, and the prices are much lower than the $6.37/M estimate from the article.
Llama is not even a good example, as the recent models are more optimized using Mixture Of Experts and KV cache compression.
apsec112 · 3h ago
This ignores batching - token generation is much more efficient in batch - and I strongly suspect is itself written by AI, given the heavy use of bullets
twoodfin · 1h ago
The “X—not Y” pattern is also a dead giveaway.
biophysboy · 2h ago
is it common for adjacent tokens to use the same weights in a memory cache?
PaulHoule · 4h ago
When the AI hype train left the station I said "we don't understand how these things work at all and they're going to get much cheaper to run" and that turned out to be... true.
Already vendors of legacy models like ChatGPT-4 have to subsidize inference to keep up with new entrants based on a better foundation. It's likely that inference costs can be brought down by another factor of ten or so so of course you have to 90% subsidize these to get where the industry will be in 2-3 years.
impure · 1h ago
I’ve been playing around with Gemma E4B and have gotten really good results. That’s a model you can run on a phone. So although prices have been going up recently I suspect they will start to fall again soon.
mrtksn · 3h ago
Subsidized is probably not the correct word here, it's probably more like loss leader in the race of the land grab.
It's like the early days of the internet when everything was amazing and all the people who put money into this thing were "losing" their money.
It's going to be like this until monopolization and moat becomes defensible and then they will enshittify the crap of it and make their money back 10x, 100x etc.
GaggiX · 3h ago
This calculation doesn't account for batches, it makes no sense.
BriggyDwiggs42 · 2h ago
On average how much does batching bring costs down?
GaggiX · 1h ago
It balances the computing and memory bandwidth bottleneck so by a lot, with continuous batching you can easily see a x10, x20 or more.
revskill · 3h ago
No lol. The quality is mostly bad. Basically u need to prompt in detail like writing a novel for llm to understand. At that price, we want real AI who can really have common sense, not just an autocompletion tool.
Stop adverting LLM as AI, instead sell it as a superior copy & paste engine.
What's worst about LLM, is the more you talk with it, the worse it became to the point of broken.
There's not much incentive to subsidize prices for OpenRouter providers for example, and the prices are much lower than the $6.37/M estimate from the article.
https://openrouter.ai/meta-llama/llama-3.3-70b-instruct
avg $0.37/M input tokens, $0.73/M output tokens (21 providers)
Llama is not even a good example, as the recent models are more optimized using Mixture Of Experts and KV cache compression.
Already vendors of legacy models like ChatGPT-4 have to subsidize inference to keep up with new entrants based on a better foundation. It's likely that inference costs can be brought down by another factor of ten or so so of course you have to 90% subsidize these to get where the industry will be in 2-3 years.
It's like the early days of the internet when everything was amazing and all the people who put money into this thing were "losing" their money.
It's going to be like this until monopolization and moat becomes defensible and then they will enshittify the crap of it and make their money back 10x, 100x etc.
Stop adverting LLM as AI, instead sell it as a superior copy & paste engine.
What's worst about LLM, is the more you talk with it, the worse it became to the point of broken.