The real news is that non-thinking output is now 4x more expensive, which they of course carefully avoid mentioning in the blog, only comparing the thinking prices.
How cute they are with their phrasing:
> $2.50 / 1M output tokens (*down from $3.50 output)
Which should be "up from $0.60 (non-thinking)/down from $3.50 (thinking)"
amazingamazing · 1h ago
Is it possible to get non-thinking only now, though? If not, why would that matter, since it's irrelevant?
jjani · 1h ago
Yes, by setting the thinking budget to 0. Which is very common when a task doesn't need thinking.
In addition, it's also relevant because for the last 3 months people have built things on top of this.
amazingamazing · 1h ago
interesting - why wouldn't you use dynamic thinking? and yeah, sucks when the price changes.
drift_code · 1h ago
They seem just rebrand the non-thinking model to flash-lite, so it’s less expensive than before
jjani · 1h ago
Not at all. Non-thinking flash is... flash with the thinking budget set to 0 (which you can still run that way, just at 2x input 4x output pricing). Flash-lite is far weaker, unusable for the overwhelming majority of usecases of flash. A quick glance at the benchmark reveals this.
rvnx · 1h ago
Yeah, so basically their announcement is "good news, we tripled the price, and will deprecate Gemini Flash 2.0 asap"
mcintyre1994 · 1h ago
The OP says Flash-Lite has thinking and non-thinking, so it’s not that simple.
irthomasthomas · 2h ago
"Soon, AI too cheap to meter"
"Meantime, price go up".
skybrian · 2h ago
There are a lot more price drops, though.
llm_nerd · 44m ago
Not too long ago Google was a bit of a joke in AI and their offerings were uncompetitive. For a while a lot of their preview/beta models had a price of 0.00. They were literally giving it away for free to try to get people to consider their offerings when building solutions.
As they've become legitimately competitive they have moved towards the pricing of their competitors.
nicce · 2h ago
We have likely seen the cheapest prices already. Once we can’t function without them anymore - go as high as you can!
nico · 2h ago
Hopefully we get more competition and someone willing to undercut the more expensive options
nicce · 51m ago
Entering the market and being competitive gets more difficult all the time. People want the best and fastest models - can you compete with trillion dollar datacenters?
tekno45 · 2h ago
"will be too cheap to meter" means we're definitely metering it now.
dangoodmanUT · 34m ago
Good catch, that's a pretty notable change considering this was about to be the GOAT of audio-to-audio
rudedogg · 2h ago
A cool 2x+ price increase.
And Gemini 2.0 Flash was $0.10/$0.40.
__jl__ · 1h ago
1.5 -> 2.0 was a price increase as well (double, I think, and something like 4x for image input)
Now 2.0 -> 2.5 is another hefty price increase.
jjani · 1h ago
4x price increase over preview output for non-thinking.
k8sToGo · 2h ago
You can also see this difference in open router.
But why is there only thinking flash now?
Tiberium · 2h ago
It might be a bit confusing, but there's no "only thinking flash" - it's a single model, and you can turn off thinking if you set thinking budget to 0 in the API request. Previously 2.5 Flash Preview was much cheaper with the thinking budget set to 0, now the price is the same. Of course, with thinking enabled the model will still use far more output tokens than the non-thinking mode.
hnuser123456 · 2h ago
Apparently, you can make a request to 2.5 flash to not use thinking, but it will still sometimes do it anyways, this has been an issue for months, and hasn't been fixed by model updates: https://github.com/google-gemini/cookbook/issues/722
varun_chopra · 3h ago
At one point, when they made Gemini Pro free on AI Studio, Gemini was the model of choice for many people, I believe.
Somehow it's gotten worse since then, and I'm back to using Claude for serious work.
Gemini is like that guy who keeps talking but has no idea what he's actually talking about.
I still use Gemini for brainstorming, though I take its suggestions with several grains of salt. It's also useful for generating prompts that I can then refine and use with Claude.
I use only the APIs directly with Aider (so no experience with AI Studio).
My feeling with Claude is that they still perform good with weak prompts, the "taste" is maybe a little better when the direction is kinda unknown by the prompter.
When the direction is known I see Gemini 2.5 Pro (with thinking) on top of Claude with code which does not break. And with o4-mini and o3 I see more "smart" thinking (as if there is a little bit of brain inside these models) at the expense of producing unstable code (Gemini produces more stable code).
I see problems with Claude when complexity increases and I would put it behind Gemini and o3 in my personal ranking.
So far I had no reason to go back to Claude since o3-mini was released.
macNchz · 2h ago
Using all of the popular coding models pretty extensively over the past year, I've been having great success with Gemini 2.5 Pro as far as getting working code the first time, instruction following around architectural decisions, and staying on-task. I use Aider and write mostly Python, JS, and shell scripts. I've spent hundreds of dollars on the Claude API over time but have switched almost entirely to Gemini. The API itself is also much more reliable.
My only complaint about 2.5 Pro is around the inane comments it leaves in the code (// Deleted varName here).
ZeWaka · 2h ago
If you use one of the AI static instructions methods (e.g., .github/copilot-instructions.md) and tell it to not leave the useless comments, that seems to solve the issue.
macNchz · 1h ago
I've been intending to try some side by side tests with and without a conventions file instructing it not to leave stupid comments—I'm curious to see if somehow they're providing value to the model, e.g. in multi-turn edits.
luckydata · 1h ago
it's easier to just make it do a code review with focus on removing unhelpful comments instead of asking it not to do it the first time. I do the cleanup after major rounds of work and that strategy seems to work best for me.
jjani · 1h ago
This was not my experience with the earlier preview (03), where its insistence on comment spam was too strong to overcome. Wonder if this adherence improved in the 05 or 06 updates.
stavros · 2h ago
I just spent $35 for Opus to solve a problem with a hardware side-project (I'm turning an old rotary phone into a meeting handset so I can quit meetings by hanging up, if you must know). It didn't solve the problem, it churned and churned and spent a ton of money.
I was much more satisfied with o3 and Aider, I haven't tried them on this specific problem but I did quite a bit of work on the same project with them last night. I think I'm being a bit unfair, because what Claude got stuck on seems to be a hard problem, but I don't like how they'll happily consume all my money trying the same things over and over, and never say "yeah I give up".
alecco · 2h ago
Give them feedback.
stavros · 1h ago
Feedback on what?
CamperBob2 · 1h ago
When I obtain results from one paid model that are significantly better than what I previously got from another paid model, I'll typically give a thumbs-down to the latter and point out in the comment that it was beaten by a competitor. Can't hurt.
stavros · 54m ago
Ah, this wasn't from the web interface, I was using Claude Code. I don't think it has a feedback mechanism.
willseth · 2h ago
Same experience here. I even built a Gem with am elaborate prompt instructing it how to be concise, but it still gives annoying long-winded responses and frequently expands the scope of its answer far beyond the prompt.
theturtletalks · 2h ago
I feel like this is part of the AI playbook now. Launch a really strong, capable model (expensive price inference) and once users think it’s SOTA, neuter it so the cost is cheaper and most users won’t notice.
The same happened with GPT-3.5. It was so good early on and got worse as OpenAI began to cut costs. I feel like when GPT-4.1 was cloaked as Optimus on Openrouter, it was really good, but once it launched, it also got worse.
carlos22 · 2h ago
That is the capitalism' playbook all along. Its just much faster because its just software. But they do it for everything all the time.
theturtletalks · 1h ago
I disagree with the comparison between LLM behavior and traditional software getting worse. When regular software declines in quality, it’s usually noticeable through UI changes, release notes, or other signals. Companies often don’t bother hiding it, since their users are typically locked into their ecosystem.
LLMs, on the other hand, operate under different incentives. It’s in a company’s best interest to initially release the strongest model, top the benchmarks, and then quietly degrade performance over time. Unlike traditional software, LLMs have low switching costs, users can easily jump to a better alternative. That makes it more tempting for companies to conceal model downgrades to prevent user churn.
jjani · 1h ago
> When regular software declines in quality, it’s usually noticeable through UI changes, release notes, or other signals.
Counterexample: 99% of average Joes have no idea how incredibly enshittified Google Maps has become, to just name one app. These companies intentionally boil the frog very slowly, and most people are incredibly bad at noticing gradual changes (see global warming).
Sure, they could know by comparing, but you could also know whether models are changing behind the scenes by having sets of evals.
theturtletalks · 1h ago
This is where switching costs matter. Take Google Maps, many people can’t switch to another app. In some areas, it’s the only app with accurate data, so Google can degrade the experience without losing users.
We can tell it’s getting worse because of UI changes, slower load times, and more ads. The signs are visible.
With LLMs, it’s different. There are no clear cues when quality drops. If responses seem off, users often blame their own prompts. That makes it easier for companies to quietly lower performance.
That said, many of us on HN use LLMs mainly for coding, so we can tell when things get worse.
Both cases involve the “boiling frog” effect, but with LLMs, users can easily jump to another pot. With traditional software, switching is much harder.
andybak · 1h ago
Do you mind explaining how you see this working as a nefarious plot? I don't see an upside in this case so I'm going with the old "never ascribe to malice" etc
noisy_boy · 21m ago
They should just roll back to the preview versions. Those were so much more even keeled and actually did some useful pushback instead of this cheerleader-on-steroids version they GA'd.
k8sToGo · 13m ago
But they claim it's the same model and version?
unshavedyak · 3h ago
Yea, i had similar experiences. At first it felt like it solved complex problems really well, but then i realized i was having trouble steering it for simple things. It was also very verbose.
Overall though my primary concern is the UX, and Claude Code is the UX of choice for me currently.
jasonjmcghee · 1h ago
I have no inside information but feels like they quantized it. I've seen patterns that I usually only see in quantized models like getting stuck repeating a single character indefinitely
huevosabio · 2h ago
They made it talk like buzzfeed articles for every single interaction. It's absolutely horrible
chrismustcode · 2h ago
When I ask it do to do something in cursor it goes full sherlock thinking about every possible outcome.
Just claude 4 sonnet with thinking just has a bit think then does it
UncleOxidant · 3h ago
Used to be able to use Gemini Pro free in cline. Now the API limits are so low that you immediately get messages about needing to top up your wallet and API queries just don't go through. Back to using DeepSeek R1 free in cline (though even that eventually stops after a few hours and you have to wait until the next day for it to work again). Starting to look like I need to setup a local LLM for coding - which means it's time to seriously upgrade my PC (well, it's been about 10 years so it was getting to be time anyway)
Workaccount2 · 2h ago
By the time you breakeven on whatever you spend on a decent LLM capable build, your hardware will be too far behind to run whatever is best locally then. It's something that feels cheaper but with the pace of things, unless you are churning an insane amount of tokens, probably doesn't make sense. Never mind that local models running on 24 or 48GB are maybe around flash-lite in ability while being slower than SOTA models.
Local models are mostly for hobby and privacy, not really efficiency.
FirmwareBurner · 3h ago
I found Gemini now terrible for coding. I gave it my code blocks and told it what to change and it added tonnes and tonnes of needles extra code plus endless comments. It turned a tight code into a Papyrus.
ChatGPT is better but tends to be too agreeable, never trying to disagree with what you say even if it's stupid so you end up shooting yourself in the foot.
Claude seems like the best compromise.
Just my two kopecks.
dr_kiszonka · 2h ago
They nerfed Pro 2.5 significantly in the last few months. Early this year, I had genuinely insightful conversations with Gemini 2.5 Pro. Now they are mostly frustrating.
I also have a personal conspiracy theory, i.e., that once a user exceeds a certain use threshold of 2.5 Pro in the Google Gemini app, they start serving a quantized version. Of course, I have no proof, but it certainly feels that way.
conradkay · 1h ago
Maybe they've been focusing so much on improving coding performance with RL for the new versions/previews that other areas degraded in performance
dr_kiszonka · 36m ago
I think you are right and this is probably the case.
Although, given that I rapidly went from +4 to 0 karma, a few other comments in this topic are grey, and at least one is missing, I am getting suspicious. (Or maybe it is just lunch time in MTV.)
esafak · 1h ago
I wonder how smart they are about quantizing. Do they look at feedback to decide which users won't mind?
lvl155 · 2h ago
I am very impressed with Gemini and stopped using OpenAI. Sometimes, I ping all three major models on OpenRouter but 90% is on Gemini now. Compare that to 90% ChatGPT last year.
codingwagie · 48m ago
I love to hate on google, but yeah their models are really good. The larger context window is huge
aatd86 · 56m ago
Same. For now I have canceled my claude subscription. Gemini has been catching up.
jbellis · 3h ago
Love to see it, this takes Flash Lite from "don't bother" territory for writing code to potentially useful. (Besides being inexpensive, Flash Lite is fast -- almost always sub-second, to as low as 200ms. Median around 400ms IME.)
Brokk (https://brokk.ai/) currently uses Flash 2.0 (non-Lite) for Quick Edits, we'll evaluate 2.5 Lite now.
ETA: I don't have a use case for a thinking model that is dumber than Flash 2.5, since thinking negates the big speed advantage of small models. Curious what other people use that for.
candiddevmike · 3h ago
Curious to hear what folks are doing with Gemini outside of the coding space and why you chose it. Are you building your app so you can swap the underlying GenAI easily? Do you "load balance" your usage across other providers for redundancy or cost savings? What would happen if there was ever some kind of spot market for LLMs?
thimabi · 3h ago
In my experience, Gemini 2.5 Pro really shines in some non-coding use cases such as translation and summarization via Canvas. The gigantic context window and large usage limits help in this regard.
I also believe Gemini is much better than ChatGPT in generating deep research reports. Google has an edge in web search and it shows. Gemini’s reports draw on a vast number of sources, thus tend to be more accurate. In general, I even prefer its writing style, and I like the possibility of exporting reports to Google Docs.
One thing that I don’t like about Gemini is its UI, which is miles behind the competition. Custom instructions, projects, temporary chats… these things either have no equivalent in Gemini or are underdeveloped.
hnuser123456 · 2h ago
If you're a power user, you should probably be using Gemini through AI studio rather than the "basic user" version. That allows you to set system instructions, temperature, structured output, etc. There's also NotebookLM. Google seems to be trying to make a bunch of side projects based on Gemini and seeing what sticks, and the generic gemini app/webchat is just one of those.
thimabi · 1h ago
My complaint is that any data within AI Studio can be kept by Google and used for training purposes — even if using the paid tier of the API, as far as I know. Because of that, I end up only using it rarely, when I don’t care about the fate of the data.
happyopossum · 26m ago
This is only true for the free tier. Paid Ai Studio users have strong privacy protections.
VeejayRampay · 1h ago
for translation you'll still be limited for longer texts by the 65K output limit though I suppose?
thimabi · 1h ago
Yes. I haven't had problems with the output limit so far, as I do translations iteratively, over each section of longer texts.
What I like the most about translating with Gemini is that its default performance is already good enough, and it can be improved via the one million tokens of the context window. I load to the context my private databases of idiomatic translations, separated by language pairs and subject areas. After doing that, the need for manually reviewing Gemini translations is greatly diminished.
ttul · 2h ago
I can throw a pile of NDAs at it and it neatly pulls out relevant stuff from them within a few seconds. The huge context window and excellent needle in a haystack performance is great for this kind of task.
spmurrayzzz · 2h ago
The NIAH performance is a misleading indicator for performance on the tasks people really want the long context for. It's great as a smoke/regression test. If you're bad on NIAH, you're not gonna do well on the more holistic evals.
But the long context eval they used (MRCR) is limited. It's multi-needle, so that's a start, but its not evaluating long range dependency resolution nor topic modeling, which are the things you actually care about beyond raw retrieval for downstream tasks. Better than nothing, but not great for just throwing a pile of text at it and hoping for the best. Particularly for out-of-distribution token sequences.
I do give google some credit though, they didn't try to hide how poorly they did on that eval. But there's a reason you don't see them adding RULER, HELMET, or LongProc to this. The performance is abysmal after ~32k.
EDIT: I still love using 2.5 Pro for a ton of different tasks. I just tend to have all my custom agents compress the context aggressively for any long context or long horizon tasks.
NitpickLawyer · 1h ago
> The performance is abysmal after ~32k.
Huh. We've not seen this in real-world use. 2.5 pro has been the only model where you can throw a bunch of docs into it, give it a "template" document (report, proposal, etc), even some other-project-example stuff, and tell it to gather all relevant context from each file and produce "template", and it does surprisingly well. Couldn't reproduce this with any other top tier model, at this level of quality.
spmurrayzzz · 59m ago
We're a G-suite shop so I set aside a ton of time trying to get 2.5 pro to work for us. I'm not entirely unhappy with it, its a highly capable model, but the long context implosion significantly limits it for the majority of task domains.
We have long context evals using internal data that are leveraged for this (modeled after longproc specifically) and the performance across the board is pretty bad. Task-wise for us, it's about as real world as it gets, using production data. Summarization, Q&A, coding, reasoning, etc.
But I think this is where the in-distribution vs out-of-distribution distinction really carries weight. If the model has seen more instances of your token sequences in training and thus has more stable semantic representations of them in latent space, it would make sense that it would perform better on average.
In my case, the public evals align very closely with performance on internal enterprise data. They both tank pretty hard. Notably, this is true for all models after a certain context cliff. The flagship frontier models predictably do the best.
sync · 1h ago
I use it extensively for https://lexikon.ai - in particular one part of what Lexikon does involves processing large amounts of images, and the way Google charges for vision is vastly cheaper compared to the big alternatives (OpenAI, Anthropic)
mrtesthah · 1h ago
Wow, if I knew that someone was using your product on my conversation with them I'd probably have to block them.
satvikpendem · 13m ago
I mean I've copy pasted conversations and emails into ChatGPT as well, it often gives good advice on tricky problems (essentially like your own personalized r/AmITheAsshole chat). This service seems to just automate that process.
extr · 1h ago
Gemini Flash 2.0 is an absolute workhorse of a model at extremely low cost. It's obviously not going to measure up to frontier models in terms of intelligence but the combination of low cost, extreme speed, and highly reliable structured output generation make it really pleasant to develop with. I'll probably test against 2.5 Lite for an upgrade here.
wg0 · 15m ago
I want to know what use cases you're using if for it it's not confidential.
k8sToGo · 2h ago
I use Gemini 2.5 Flash (non thinking) as a thought partner. It helps me organize my thoughts or maybe even give some new input I didn't think of before.
I really like to use it also for self reflection where I just input my thoughts and maybe concerns and just see what it has to say.
bradly · 1h ago
I've yet to run out of free image gen credits with Gemini, so I use it for any low-effort image gen like when my kids want to play with it or for testing prompts before committing my o4 tokens for better quality results.
androng · 2h ago
I use it for https://toolong.link Youtube summaries with images because only Gemini has easy access to YouTube and it has a gigantic context window
crowcroft · 3h ago
Simple unstructured to structured data transformation.
I find Flash and Flash Lite are more consistent than others as well as being really fast and cheap.
I could swap to other providers fairly easily, but don't intend to at this point. I don't operate at a large scale.
jasoncartwright · 2h ago
Web scraping - creating semi-structured data from a wide variety of horrific HTML soups.
Absolutely do swap out models sometimes, but Gemini 2.0 Flash is the right price/performance mix for me right now. Will test Gemini 2.5 Flash-Lite tomorrow though.
fastest963 · 2h ago
Yes, we implemented a separate service internally that interfaces with an LLM and so the callers can be agnostic as to what provider or model is being used. Haven't needed to load balance between models though.
willidiots · 3h ago
Low-latency LLM for my home automation. Anecdotally, Gemini was much quicker than OpenAI in responding to simple commands.
In general, when I need "cheap and fast" I choose Gemini.
Dnajsre · 2h ago
It basically made a university physics exam for me. It almost one-shot it as well. Just uploaded some exams from previous years together with a latex template and told it to make me a similar one. Worked great. Also made it do the solutions.
jeffbee · 2h ago
It's very good at automatically segmenting and recognizing handwritten and badly scanned text. I use it to make spreadsheets out of handwritten petitions.
HDThoreaun · 3h ago
I tried swapping for my project which involves having the LLM summarize and critique medical research and didn’t have great results. The prompt I found works best with the main LLM I use fucks up the intended format when fed to other LLMs. Thinking about refining prompts for each different llm but haven’t gotten there.
My favorite personal use of Gemini right now is basically as a book club. Of course it’s not as good as my real one but I often can’t them to read the books I want and Gemini is always ready when I want to explore themes. It’s often more profound than the book club too and seems a bit less likely to tunnel vision. Before LLMs I found exploring book themes pretty tedious, often I would have to wait a while to find someone who had read it but now I can get into it as soon as I’m done reading.
sethkim · 52m ago
I run a batch inference/LLM data processing service and we do a lot of work around cost and performance profiling of (open-weight) models.
One odd disconnect that still exists in LLM pricing is the fact that providers charge linearly with respect to token consumption, but costs are actually quadratic with an increase in sequence length.
At this point, since a lot of models have converged around the same model architecture, inference algorithms, and hardware - the chosen costs are likely due to a historical, statistical analysis of the shape of customer requests. In other words, I'm not surprised to see costs increase as providers gather more data about real-world user consumption patterns.
raybb · 12m ago
It's a bummer that 2.5 Pro is still removed from the free tier of the API.
dinesh2609 · 1h ago
6.33X increase in the price of Audio processing compared to 2.0 Flash-Lite
Gemini 2.5 Flash Lite (Audio Input) - $0.5/million tokens
Gemini 2.0 Flash Lite (Audio Input) - $0.075/million tokens
Wonder what led to such a high bump in Audio token processing
zurfer · 1h ago
for anyone, who was expecting more news: the GA models benchmark basically the same as the last preview models. It's really just Google telling us that we get less api errors and this model will have a checkpoint for a longer time.
zzleeper · 2h ago
Good luck using 2.5 for anything non-trivial.
I have about 500,000 news articles I am parsing. OpenAI models work well but found Gemini had fewer mistakes.
Problem is; they give me a terrible 10k RPD limit. To increase to the next tier, they then require a minimum amount of spending but I can't reach that amount even when maxing the RPD limit for multiple days in a row.
I emailed them twice and completed their forms but everyone knows how this works. So now I'm back at OpenAI, with a model with a bit more mistakes but that won't 403 me after half an hour of using it due to their limits.
be7a · 11m ago
The rate limits apply only to the Gemini API. There is also Vertex from GCP, which offers the same models (and even more, such as Claude) at the same pricing, but with much higher rate limits (basically none, as long as they don't need to cut anyone off with provisioned throughput iiuc) and with a process to get guaranteed throughput.
eldenring · 1h ago
I'm guessing now that it is GA this won't be a problem.
zzleeper · 27m ago
I wish! The tier-based limits are still the same!
At least it's more expensive now so I guess I will be able to hop to the next tier sooner? ¯\_(ツ)_/¯
serjester · 3h ago
I'm glad that they standardized pricing for the thinking vs non-thinking variant. A couple weeks ago I accidentally spent thousands of extra dollars by forgetting to set the thinking budget to zero. Forgetting a single config parameter should not automatically raise the model cost 5X.
[edit] I'm less excited about this because it looks like their solution was to dramatically raise the base price on the non-thinking variant.
zelias · 2h ago
Not sure where else to post this, but when attempting to use any of the Gemini 2.5 models via API, I receive an "empty content" response about 50% of the time. To be clear, the API responds successfully, but the `content` returned by the LLM is just an empty string.
Has anyone here had any luck working around this problem?
Tiberium · 2h ago
What finish reason are you getting? Perhaps your code sets a low max_tokens, so the generation stops while the model is still thinking, without giving any actual output.
zelias · 2h ago
The finish reason is `length`. I have tried setting minimal token budgets, really small prompts, and max lengths of various sizes from 100-4000 and nothing seems to make a consistent dent in the behavioral pattern.
heliophobicdude · 3h ago
Wishing they release the Gemini Diffusion model. It'll quickly replace the default model for Aider.
vessenes · 2h ago
It feels to me like properly instrumented, these diffusion models are going to be really powerful coding tools. Imagine a “smart” model carving out a certain number of tokens in a response for each category of response output, then diffusing the categories.
causal · 3h ago
Why do you think so? I've played with the Diffusion model a bit and it makes a lot of mistakes
b0a04gl · 2h ago
been testing gemini flash lite. latency is good, responses land under 400ms most times. useful for low-effort rewrites or boilerplate filler. quality isn’t stable though : context drifts after 4-5 turns, especially with anything recursive or structured. tried tagging it into prompt chains but fallback logic ends up too aggressive. good for assist, not for logic, wouldn't anchor anything serious on it yet
tiahura · 1h ago
Gemini 2.5 doesn’t get enough credit for the quality of its writing in non-code (eg law) topics. It’s definitely a notch below Claude 4, but well ahead of ChatGPT 4o, 4.5, o3.
jjani · 1h ago
Classic bait-and-switch to make developers build things on top off models for 2 months, and then raise input price by 2x and output by 4x. But hey, it's Google, wouldn't expect anything else from an advertising company.
hs86 · 3h ago
I am always disappointed when I compare the answers to the same queries on 2.5 Pro vs. o4-mini/o3. But trying out the same query in AI Studio gives much better results, closer to OpenAI's models.
What is wrong with 2.5 Pro in the Gemini app? I can't believe that the model in their consumer app would produce the same benchmark results as 2.5 Pro in the API or AI Studio.
thimabi · 2h ago
The models in the Gemini app are nerfed in comparison to those in AI Studio: they have less thinking budget, output less tokens, and have various safety filters. There’s certainly a trade-off between using AI Studio for its better performance and using the API or the Gemini app in a way that doesn’t involve Google keeping your data for training purposes.
mh- · 3h ago
I don't have any inside information, but I'm sure there are different system prompts used in the Gemini chat interface vs the API. On OpenAI/ChatGPT they're sometimes dramatically different.
remus · 3h ago
I mean the model names are always a bit odd, but flash-lite is particulary good!
sergiotapia · 3h ago
Considering moving from Groq Llama 3.3 70b to Gemini 2.5 Flash Lite for one of my use cases. Results are coming in great, and it's very fast (important for my real-time user perception needs).
What kind of rate limits do these new Gemini models have?
UncleOxidant · 3h ago
Are you using Groq Llama 3.3 70b from something like cline? Is it free and what are the API query limits?
sergiotapia · 3h ago
I'm using it from their HTTP API. Limits I can't remember what they were initially tbh, I had to reach out through backchannels to get it increased to 300,000 tokens per minute.
2Gkashmiri · 3h ago
I have a huge background.js file from a now removed browser extension that the Devs made into a single line. Around 800KB of a single line file I think....
I tried many free stuff to try to refactor it but they all loose context window quickly.
mh- · 2h ago
There are myriad non-LM tools that can deobfuscate and prettify JS. I used them with success long before LLMs were en vogue.
GaggiX · 3h ago
2.5 Flash Lite seems better at everything compare to 2.0 Flash Lite with the only exception being SimpleQA, so there is probably a small tradeoff on pop culture knowledge for coding, math, science, reasoning and multimodal tasks.
For 2.5 Flash Preview https://web.archive.org/web/20250616024644/https://ai.google...
$0.15/million input text / image / video
$1.00/million audio
Output: $0.60/million non-thinking, $3.50/million thinking
The new prices for Gemini 2.5 Flash ditch the difference between thinking and non-thinking and are now: https://ai.google.dev/gemini-api/docs/pricing
$0.30/million input text / image / video (2x more)
$1.00/million audio (same)
$2.50/million output - significantly more than the old non-thinking price, less than the old thinking price.
https://developers.googleblog.com/en/gemini-2-5-thinking-mod...
How cute they are with their phrasing:
> $2.50 / 1M output tokens (*down from $3.50 output)
Which should be "up from $0.60 (non-thinking)/down from $3.50 (thinking)"
In addition, it's also relevant because for the last 3 months people have built things on top of this.
As they've become legitimately competitive they have moved towards the pricing of their competitors.
And Gemini 2.0 Flash was $0.10/$0.40.
Now 2.0 -> 2.5 is another hefty price increase.
But why is there only thinking flash now?
Somehow it's gotten worse since then, and I'm back to using Claude for serious work.
Gemini is like that guy who keeps talking but has no idea what he's actually talking about.
I still use Gemini for brainstorming, though I take its suggestions with several grains of salt. It's also useful for generating prompts that I can then refine and use with Claude.
I use only the APIs directly with Aider (so no experience with AI Studio).
My feeling with Claude is that they still perform good with weak prompts, the "taste" is maybe a little better when the direction is kinda unknown by the prompter.
When the direction is known I see Gemini 2.5 Pro (with thinking) on top of Claude with code which does not break. And with o4-mini and o3 I see more "smart" thinking (as if there is a little bit of brain inside these models) at the expense of producing unstable code (Gemini produces more stable code).
I see problems with Claude when complexity increases and I would put it behind Gemini and o3 in my personal ranking.
So far I had no reason to go back to Claude since o3-mini was released.
My only complaint about 2.5 Pro is around the inane comments it leaves in the code (// Deleted varName here).
I was much more satisfied with o3 and Aider, I haven't tried them on this specific problem but I did quite a bit of work on the same project with them last night. I think I'm being a bit unfair, because what Claude got stuck on seems to be a hard problem, but I don't like how they'll happily consume all my money trying the same things over and over, and never say "yeah I give up".
The same happened with GPT-3.5. It was so good early on and got worse as OpenAI began to cut costs. I feel like when GPT-4.1 was cloaked as Optimus on Openrouter, it was really good, but once it launched, it also got worse.
LLMs, on the other hand, operate under different incentives. It’s in a company’s best interest to initially release the strongest model, top the benchmarks, and then quietly degrade performance over time. Unlike traditional software, LLMs have low switching costs, users can easily jump to a better alternative. That makes it more tempting for companies to conceal model downgrades to prevent user churn.
Counterexample: 99% of average Joes have no idea how incredibly enshittified Google Maps has become, to just name one app. These companies intentionally boil the frog very slowly, and most people are incredibly bad at noticing gradual changes (see global warming).
Sure, they could know by comparing, but you could also know whether models are changing behind the scenes by having sets of evals.
We can tell it’s getting worse because of UI changes, slower load times, and more ads. The signs are visible.
With LLMs, it’s different. There are no clear cues when quality drops. If responses seem off, users often blame their own prompts. That makes it easier for companies to quietly lower performance.
That said, many of us on HN use LLMs mainly for coding, so we can tell when things get worse.
Both cases involve the “boiling frog” effect, but with LLMs, users can easily jump to another pot. With traditional software, switching is much harder.
Overall though my primary concern is the UX, and Claude Code is the UX of choice for me currently.
Just claude 4 sonnet with thinking just has a bit think then does it
Local models are mostly for hobby and privacy, not really efficiency.
ChatGPT is better but tends to be too agreeable, never trying to disagree with what you say even if it's stupid so you end up shooting yourself in the foot.
Claude seems like the best compromise.
Just my two kopecks.
I also have a personal conspiracy theory, i.e., that once a user exceeds a certain use threshold of 2.5 Pro in the Google Gemini app, they start serving a quantized version. Of course, I have no proof, but it certainly feels that way.
Although, given that I rapidly went from +4 to 0 karma, a few other comments in this topic are grey, and at least one is missing, I am getting suspicious. (Or maybe it is just lunch time in MTV.)
Brokk (https://brokk.ai/) currently uses Flash 2.0 (non-Lite) for Quick Edits, we'll evaluate 2.5 Lite now.
ETA: I don't have a use case for a thinking model that is dumber than Flash 2.5, since thinking negates the big speed advantage of small models. Curious what other people use that for.
I also believe Gemini is much better than ChatGPT in generating deep research reports. Google has an edge in web search and it shows. Gemini’s reports draw on a vast number of sources, thus tend to be more accurate. In general, I even prefer its writing style, and I like the possibility of exporting reports to Google Docs.
One thing that I don’t like about Gemini is its UI, which is miles behind the competition. Custom instructions, projects, temporary chats… these things either have no equivalent in Gemini or are underdeveloped.
What I like the most about translating with Gemini is that its default performance is already good enough, and it can be improved via the one million tokens of the context window. I load to the context my private databases of idiomatic translations, separated by language pairs and subject areas. After doing that, the need for manually reviewing Gemini translations is greatly diminished.
But the long context eval they used (MRCR) is limited. It's multi-needle, so that's a start, but its not evaluating long range dependency resolution nor topic modeling, which are the things you actually care about beyond raw retrieval for downstream tasks. Better than nothing, but not great for just throwing a pile of text at it and hoping for the best. Particularly for out-of-distribution token sequences.
I do give google some credit though, they didn't try to hide how poorly they did on that eval. But there's a reason you don't see them adding RULER, HELMET, or LongProc to this. The performance is abysmal after ~32k.
EDIT: I still love using 2.5 Pro for a ton of different tasks. I just tend to have all my custom agents compress the context aggressively for any long context or long horizon tasks.
Huh. We've not seen this in real-world use. 2.5 pro has been the only model where you can throw a bunch of docs into it, give it a "template" document (report, proposal, etc), even some other-project-example stuff, and tell it to gather all relevant context from each file and produce "template", and it does surprisingly well. Couldn't reproduce this with any other top tier model, at this level of quality.
We have long context evals using internal data that are leveraged for this (modeled after longproc specifically) and the performance across the board is pretty bad. Task-wise for us, it's about as real world as it gets, using production data. Summarization, Q&A, coding, reasoning, etc.
But I think this is where the in-distribution vs out-of-distribution distinction really carries weight. If the model has seen more instances of your token sequences in training and thus has more stable semantic representations of them in latent space, it would make sense that it would perform better on average.
In my case, the public evals align very closely with performance on internal enterprise data. They both tank pretty hard. Notably, this is true for all models after a certain context cliff. The flagship frontier models predictably do the best.
I really like to use it also for self reflection where I just input my thoughts and maybe concerns and just see what it has to say.
I find Flash and Flash Lite are more consistent than others as well as being really fast and cheap.
I could swap to other providers fairly easily, but don't intend to at this point. I don't operate at a large scale.
Absolutely do swap out models sometimes, but Gemini 2.0 Flash is the right price/performance mix for me right now. Will test Gemini 2.5 Flash-Lite tomorrow though.
In general, when I need "cheap and fast" I choose Gemini.
My favorite personal use of Gemini right now is basically as a book club. Of course it’s not as good as my real one but I often can’t them to read the books I want and Gemini is always ready when I want to explore themes. It’s often more profound than the book club too and seems a bit less likely to tunnel vision. Before LLMs I found exploring book themes pretty tedious, often I would have to wait a while to find someone who had read it but now I can get into it as soon as I’m done reading.
One odd disconnect that still exists in LLM pricing is the fact that providers charge linearly with respect to token consumption, but costs are actually quadratic with an increase in sequence length.
At this point, since a lot of models have converged around the same model architecture, inference algorithms, and hardware - the chosen costs are likely due to a historical, statistical analysis of the shape of customer requests. In other words, I'm not surprised to see costs increase as providers gather more data about real-world user consumption patterns.
Gemini 2.5 Flash Lite (Audio Input) - $0.5/million tokens
Gemini 2.0 Flash Lite (Audio Input) - $0.075/million tokens
Wonder what led to such a high bump in Audio token processing
I have about 500,000 news articles I am parsing. OpenAI models work well but found Gemini had fewer mistakes.
Problem is; they give me a terrible 10k RPD limit. To increase to the next tier, they then require a minimum amount of spending but I can't reach that amount even when maxing the RPD limit for multiple days in a row.
I emailed them twice and completed their forms but everyone knows how this works. So now I'm back at OpenAI, with a model with a bit more mistakes but that won't 403 me after half an hour of using it due to their limits.
At least it's more expensive now so I guess I will be able to hop to the next tier sooner? ¯\_(ツ)_/¯
[edit] I'm less excited about this because it looks like their solution was to dramatically raise the base price on the non-thinking variant.
Has anyone here had any luck working around this problem?
What kind of rate limits do these new Gemini models have?
I tried many free stuff to try to refactor it but they all loose context window quickly.