Gemini 2.5 Pro Preview

> no amount of prompting will get current models to approach abstraction and architecture the way a person does

I find this sentiment increasingly worrisome. It's entirely clear that every last human will be beaten on code design in the upcoming years (I am not going to argue if it's 1 or 5 years away, who cares?)

I wished people would just stop holding on to what amounts to nothing, and think and talk more about what can be done in a new world. We need good ideas and I think this could be a place to advance them.

This is hilarious to read if you have actually seen the average (embedded systems) production code written by humans.

Either you have no idea how terrible real world commercial software (architecture) is or you're vastly underestimating newer LLMs or both.

I recently needed to recommend some IAM permissions for an assistant on a hobby project; not complete access but just enough to do what was required. Was rusty with the console and didn't have direct access to it at the time, but figured it was a solid use case for LLMs since AWS is so ubiquitous and well-documented. I actually queried 4o, 3.7 Sonnet, and Gemini 2.5 for recommendations, stripped the list of duplicates, then passed the result to Gemini to vet and format as JSON. The result was perfectly formatted... and still contained a bunch of non-existent permissions. My first time being burned by a hallucination IRL, but just goes to show that even the latest models working in concert on a very well-defined problem space can screw up.

This problem have been solved by LSP (language server protocol), all we need is a small server behind MCP that can communicate LSP information back to the LLM and get the LLM to use by adding to the prompt something like: "check your API usage with the LSP"

The unfortunate state of open source funding makes buildings such simple tool a loosing adventure unfortunately.

If they never get good at abstraction or architecture they will still provide a tremendous amount of value. I have them do the parts of my job that I don't like. I like doing abstraction and architecture.

I've found they do a decent job searching for bugs now as well. Just yesterday I had a bug report on a component/page I wasn't familiar with in our Angular app. I simply described the issue as well as I could to Claude and asked politely for help figuring out the cause. It found the exact issue correctly on the first try and came up with a few different suggestions for how to fix it. The solutions weren't quite what I needed but it still saved me a bunch of time just figuring out the error.

The opposite problem is also true. I was using it to edit code I had that was calling the new openai image API, which is slightly different from the dalle API. But Gemini was consistently "fixing" the OpenAI call even when I explained clearly not to do that since I'm using a new API design etc. Claude wasn't having that issue.

The models are very impressive. But issues like these still make me feel they are still more pattern matching (although there's also some magic, don't get me wrong) but not fully reasoning over everything correctly like you'd expect of a typical human reasoner.

I feel like there are two realities right now where half the people say LLM doesn't do anything well and there is another half that's just using LLM to the max. Can everybody preface what stack they are using or what exactly they are doing so we can better determine why it's not working for you? Maybe even include what your expectations are? Maybe even tell us what models you're using? How are you prompting the models exactly?

I find for 90% of the things I'm doing LLM removes 90% of the starting friction and let me get to the part that I'm actually interested in. Of course I also develop professionally in a python stack and LLMs are 1 shotting a ton of stuff. My work is standard data pipelines and web apps. I'm a tech lead at faang adjacent and the systems I work with are responsible for about half a billion dollars a year in transactions directly and growing. Everybody that I work with is getting valuable output from LLMs. We are using all the latest openAI models and have a business relationship with openAI. I don't think I'm even that good at prompting and mostly rely on "vibes". Half of the time I'm pointing the model to an example and telling it "in the style of X do X for me".

I feel like comments like these almost seem gaslight-y or maybe there's just a major expectation mismatch between people. Are you expecting LLMs to just do exactly what you say and your entire job is to sit back prompt the LLM?

I've had great success by asking it to do project design first, compose the design into an artifact, and then asking it to consult the design artifact as it writes code.

I’ve seen benchs on hallucinations and OpenAI has typically performed worse than Google and Anthropic models. Sometimes significantly so. But it doesn’t seem like they have cared much. I’ve suspected that LLM performance is correlated to risking hallucinations? That is, if they’re bolder, this can be beneficial? Which helps in other performance benchmarks. But of course at the risk of hallucinating more…

searching and ranking existing fragments and recombining them within well known paths is one thing, exploratively combining existing fragments to completely novel solutions quickly runs into combinatorial explosion.

so it's a great tool in the hands of a creative architect, but it is not one in and by itself and I don't see yet how it can be.

my pet theory is that the human brain can't understand and formalize its creativity because you need a higher order logic to fully capture some other logic. I've been contested that the second Gödel incompleteness theorem "can't be applied like this to the brain" but I stubbornly insist yes, the brain implements _some_ formal system and it can't understand how that system works. tongue in cheek, somewhat, maybe.

but back to earth I agree llms are a great tool for a creative human mind.

To date, LLMs can't replace the human element of:

- Determining what features to make for users

- Forecasting out a roadmap that are aligned to business goals

- Translating and prioritizing all of these to a developer (regardless of whether these developers are agentic or human)

Coincidentally these are the areas that frequently are the largest contributors to software businesses successes....not wether you use NextJs with a Go and Elixir backend against a multi-geo redundant multi sharded CockroachDB database, or that your code is clean/elegant.

Making LLMs know what they don't know is a hard problem. Many attempts at making them refuse to answer what they don't know caused them to refuse to answer things they did in fact know.

You should give it docs for each of your base dependencies in a mcp/tool whatever so it can just consult.

internet also helps.

Also having markdown files with the stack etc and any -rules-

I asked today both Claude and ChatGPT to fix a Grafana Loki query I was trying to build, both hallucinated functions that didn't exist, even when telling to use existing functions.

To my surprise, Gemini got it spot on first time.

I don't know about that, my own adventures with Gemini Pro 2.5 in Roo Code has it outputting code in a style that is very close to my own

While far from perfect for large projects, controlling the scope of individual requests (with orchestrator/boomerang mode, for example) seems to do wonders

Given the sheer, uh, variety of code I see day to day in an enterprise setting, maybe the problem isn't with Gemini?

I've noticed that models that can search internet do it a lot less because I guess they can look up documentation? My annoyance now is that it doesn't take version into consideration.

If you use Cursor, you can use @Docs to let it index the documentation for the libraries and languages you use, so no hallucination happens.

Use few-shot learning. Build a simple prompt with basic examples of how to use the API and it will do significantly better.

LLMs just guess, so you have to give it a cheatsheet to help it guess closer to what you want.

> hallucinate APIs

Tell me about it. Thankfully I have not experienced it as much with Claude as I did with GPT. It can get quite annoying. GPT kept telling me to use this and that and none of them were real projects.

I feel your pain. Cursor has docs features but many times when I pointed to check @docs and selected one recently indexed one it sometimes still didn't get it. I still have to try contex7 mcp which looks promising:

https://github.com/upstash/context7

Replacing stackoverflow is definitely helpful, but the best use case for me is how much it helps in high-level architecture and planning before starting a project.

I have been asking if AI without hallucination, coding or not is possible but so far with no real concrete answer.

Ask the models that can search to double check their API usage. This can just be part of a pre-prompt.

I definitely get more use out of Gemini Pro than other models I've tried, but it's still very prone to bullshitting.

I asked it a complicated question about the Scala ZIO framework that involved subtyping, type inference, etc. - something that would definitely be hard to figure out just from reading the docs. The first answer it gave me was very detailed, very convincing and very wrong. Thankfully I noticed it myself and was able to re-prompt it and I got an answer that is probably right. So it was useful in the end, but only because I realised that the first answer was nonsense.

same, I asked a simple question about javascript fetch api and it started talking about the workspace api. When I asked about that workspace api it replied it was the google workspace API ¯ \ _ (ツ) _ / ¯

Huh? Have you ever just told it, that API doesn’t exist, find another solution?

Never seen it fumble that around

Swear people act like humans themselves don’t ever need to be asked for clarification

Bwoah it's almost as if react and tailwind is the bees knees ind frontend atm

I usually remove the comments by hand. It's actually pretty helpful, it ensures I've reviewed every piece of code carefully, especially since most of the comments are literally just restating the next line, and "does this comment add any information?" is a really helpful question to make sure I understand the code.

I've found that heavily commented code can be better for the LLM to read later, so it pulls in explanatory comments into context at the same time as reading code, similar to pulling in @docs, so maybe it's doing that on purpose?

  --- SECTION ONE OF THE CODE ---

  ...

  --- SECTION TWO OF THE CODE ---

  try:
    [some "dangerous" code]
  except Exception as e:
     logging.error(f"Failed to save files to {output_path}: {e}")
     # Decide whether to raise the error or just warn
     # raise IOError(f"Failed to save files to {output_path}: {e}")

I have the same issue plus unnecessary refactorings (that break functionality). it doesn't matter if I write a whole paragraph in the chat or the prompt explaining I don't want it to change anything else apart from what is required to fulfill my very specific request. It will just go rogue and massacre the entirety of the file.

I have a strong sense that the comments are for the model more than the user. It's effectively more thinking in context.

Same experience. Especially the "step" comments about the performed changes are super annoying. Here is my prompt-rule to prevent them:

"5. You must never output any comments about the progress or type of changes of your refactoring or generation. Example: you must NOT add comments like: 'Added dependency' or 'Changed to new style' or worst of all 'Keeping existing implementation'."

Tell it not to write so many comments then. You have a great deal of flexibility in dictating the coding style and can even include that style in your system prompt or upload a coding style document and have Gemini use it.

2.5 was the most impressive model I use, but I agree about the comments. And when refactoring some code it wrote before, it just adds more comments, it becomes like archaeological history (disclaimer: I don’t use it for work, but to see what it can do, so I try to intervene as little as possible, and get it to refactor what it thinks it should)

It also does super defensive coding. Not that it's a bad thing in general, but I write a lot of prototype code.

I really liked the Gemini 2.5 pro model when it was first released - the upload code folder was very nice (but they removed it). The annoying things I find with the model is it does a really bad job of formatting the code it generates... I know I can use a code formatting tool and I do when i use gemini output but otherwise I find grok much easier to work with and yields better results.

My custom default Claude prompt asks it to never explain code unless specifically asked to. Also to produce modern and compact code. It's a beauty to see. You ask for code and you get code, nothing else.

That’s been my experience as well. It’s especially jarring when asking for a refactor as it will leave a bunch of WIP-style comments highlighting the difference with the previous approach.

So many comments, more verbose code and will refactor stuff on its own. Still better than chatgpt, but I just want a small amount of code that does what I asked for so I can read through it quickly.

It probably increases scores in the RL training since it's a kind of locally specific reasoning that would reduce bugs.

Which means if you try to force it to stop, the code quality will drop.

you can ask it to remove, it does p good job at it

What kind of problems are you putting in where that is the solution? Just curious.

You can ask it to remove the comments afterwards, and it'll do a decent job of it, but yeah, it's a pain.

you need to do a 2nd step as a post-process to erase the comments.

Models use comments to think, asking to remove will affect code quality.

My favourites are comments such as: from openai import DefaultHttpxClient # Import the httpx client

> all the gang

What does that mean?

It’s annoying, but I’ve done extensive work with this model and leaving the comments in for the first few iterations produced better outcomes. I expect this is baked into the RL they’re doing, but because of the context size, it’s not really an issue. You can just ask it to strip out in the final pass.

Just ask it for fewer comments, it’s not rocket science.

You can ask to not use comments or use less comments, you can put this in the system prompt too.

Ask it to do less of it, problem solved, no? With tools like Cursor it's become really easy to fit the models to the shoe, or the shoe to the foot.

If you don't want so many comments, have you tried asking the AI for fewer comments. Seems like something a little prompt engineering could solve.

And comments are bad? I mean you could tell it to not comment the code or to self-document with naming instead of inline comments, its a LLM it does what you tell it to

According to the article, "[t]he previous iteration (03-25) now points to the most recent version (05-06)." I assume this applies to both the free tier gemini-2.5-pro-exp-03-25 in the API (which will be used for training) and the paid tier gemini-2.5-pro-preview-03-25.

Fair enough, one could say, as these were all labeled as preview or experimental. Still, considering that the new model is slightly worse across the board in benchmarks (except for LiveCodeBench), it would have been nice to have the option to stick with the older version. Not everyone is using these models for coding.

This should be the top comment. Cherry-picking is hurting this industry.

I bet they kept training on coding tasks, made everything worse on the way, and tried to hide it under the rug because of the sunk costs.

Livebench.ai actually suggests the new version is better on most things.

https://livebench.ai/#/

Sounds like they were losing so much money on 2.5-Pro they came up with a forced update that made it cheaper to run. They can't come out with "we've made it worse across the board", nor do they want to be the first to actually raise prices, so instead they made a bit of a distill that's slightly better at coding so they can still spin it positively.

Update 2: I've been using this model in both aider and cline and I've haven't gotten a diff matching error yet, even with some pretty difficult substitutions across different places in multiple files. The overall feel of this model is nice.

I don't have a formal benchmark but there's a notable improvement in code generation due to this alone.

I've had gemini chug away on plans that have taken ~1 hour to implement. (~80mln tokens spent) A good portion of that energy was spent fixing mistakes made by cline/aider/roo due to search/replace mistakes. If this model gets anywhere close to 100% on diffs then this is a BFD. I estimate this will translate to a 50-75% productivity boost on long context coding tasks. I hope the initial results i'm seeing hold up!

I'm surprised by the reaction in the rest of the thread. A lot unproductive complaining, a lot of off topic stuff, nothing talking about the model itself.

Any thoughts from anyone else using the updated model?

What do you mean by agentic work in this context?

The text in the model card says the results are from March (including the Gemini 2.5 Pro results), and o3 wasn't released yet.

Is this maybe not the updated card, even though the blog post claims there is one? Sure, the timestamp is in late April, but I seem to remember that the first model card for 2.5 Pro was only released in the last couple of weeks.

o3 is $40/M output tokens and 2.5 Pro is $10-15/M output tokens so o3 being slightly ahead is not really worth 4 times more than gemini.

Loading the UI on mobile while on low bandwidth is also a non-starter. It simply doesn't work.

Noticed this too. There's something funny about billion dollar models being handicapped by stuck buttons.

It seems that trying to build llms is the definition of accepting sunk cost.

The one with exp in the name is free (you may have to add it yourself) but they train on you. And after a certain limit it becomes paid).

At the bottom of the article it says no action is required and the Gemini-2.5-pro-preview-03-25 now points to the new model

Cursor UI sucks, it tells me to use -auto mode- to be faster, but gemini 2.5 is way faster than any of the other free models, so just selecting that one is faster even if the UI says otherwise

I take preview to mean the model may be retired on an accelerated timescale and replaced with a "real" model so it's dangerous to put into prod unless you are paying attention.

Are you saying you find model names like o4-mini-high-pro-experimental-version5 confusing and stupid?

Give Gemini a shot. It is genuinely very good.

I'm wondering when Claude 4 will drop. It's long overdue.

I agree, from my experience Grok gives superior coding results, especially when modifying large sections of the codebase at once such as in refactoring.

Although it’s not for coding, I have noticed Gemini 2.5 pro Deep Research has surpassed Grok’s DeepSearch in thoroughness and research quality however.

I haven't tested this release yet, but I found Gemini to be overrated before.

My choice of LLMs was

Coding in cursor: Claude

General questions: Grok, if it fails then Gemini

Deep Research: Gemini (I don't have GPT plus, I heard it's better)

Anecdotally grok has been the worst of the bunch for me.

Anyone know of any coding agents that support video inputs?

Web chat interfaces are great, but copy/paste gets old fast.

I’ve found the same thing. I don’t use it for code any more because it produces highly verbose and inefficient code that may work but is ugly and subtly brittle.

The Gemini app is absolute dog doo... use it through AI studio. Google ought to shut down the entire Gemini app.

> The amount of hoops I would have to jump through to get either of the smaller companies I have worked for since the AI boom to let me use a tool like this would make it absolutely not worth the effort.

Define "smaller"? In small companies, say 10 people, there are no hoops. That is the whole point of small companies!

If you can get them to approve GitHub Copilot Business then Gemini Pro 2.5 and many others are available there. They have guarantees that they don't share/store prompts or code and the parent company is Microsoft. If you can argue that they will save money (on saved developer time), what would be their argument against?

I work for a large company and everything other than MS Copilot is blocked aggressively at the DNS/cert level. Tried Deepseek when it came out and they already had it blocked. All .ai TLDs are blocked as well. If you're not in tech, there is a lot of "security" fear around AI.

Not every coding task is something you want to check into your repo. I have mostly used Gemini to generate random crud. For example I had a huge JSON representation of a graph, and I wanted the graph modified in a given way, and I wanted it printed out on my terminal in color. None of which I was remotely interested in writing, so I let a robot do it and it was fine.

It's very good at Go, which makes sense because I'm assuming it's trained on a lot of Google's code

Think that was still the experimental model incorrectly labeled by many platforms as “Pro”

Haven't tried it yet, but I've heard good things about Plandex.

https://github.com/plandex-ai/plandex

I'm literally working on this particular problem. Locally-run server; browser-based interface instead of TUI/CLI; connects to all the major model APIs; many, many quality of life and feature improvements over other tools that hook into your browser.

Drop me a line (see profile) if you're interested in beta testing it when it's out.

OpenAi has a version called Codex that has support. It's lacking in a few features like MCP right now and the TUI isn't there yet, but interestingly they are building a Rust version (it's all open source) that seems to include MCP support and looks significantly higher quality. I'd bet within the next few weeks there will be a high quality claude code alternative.

Goose by Block (Square/CashApp) is like an open-source Claude Code that works with any remote or local LLM.

https://github.com/block/goose

Aider

Check the dropdown in the top left (on my screen,at least).

"The previous iteration (03-25) now points to the most recent version (05-06), so no action is required to use the improved model"

Bard sucked, Gemini sucked, Gemini 2 was alright, 2.5 is awesome and my main driver for coding these days.

The Gemini deep research is a revelation. I obsessively research most things I buy, from home appliances to gym equipment. It has literally saved untold hours of comparisons. You get detailed reports generated from every website including youtube reviews. I've bought a bunch of stuff on it's recommendation.

Probably the best one right now, their deep research is also very good.

The benchmarks (1) seem to suggest that o3 is in 3rd place after Gemini 2.5 pro preview and Gemini 2.5 pro exp (for text reasoning, o3 4th for webdev). o3 doesn't even appear on the openrouter leaderboards (2) suggesting is hardly used (if at all) by anyone using LLMs do actually do anything (such as coding) which makes one question if it is actually any good at all (otherwise if it was so great I'd expect to see heavy usage)

Not sure where your data is coming from but everything else is pointing to Google supremacy in AI right now. I look forward to some new models from Anthropic, xAi, Meta et al (remains to be seen if OpenAI has anything left apart from bluster). Exciting times.

1 - https://beta.lmarena.ai/leaderboard

2 - https://openrouter.ai/rankings

8x the cost for maybe 5% improvement?

o3 is expensive in the API and intentionally crippled in the web app.

source?

Relevance?

LMArena is a joke, though

How are you using it?

I find that I get the best results from 2.5 Pro via Google AI Studio with a low temperature (0.2-0.3).

You can do this with https://openrouter.ai/

You can do this with LLM proxies like LiteLLM. e.g. Cursor -> LiteLLM -> LLM provider API.

I have LiteLLM server running locally with Langfuse to view traces. You configure LiteLLM to connect directly to providers' APIs. This has the added benefit of being able to create LiteLLM API keys per project that proxies to different sets of provider API keys to monitor or cap billing usage.

I use https://github.com/LLemonStack/llemonstack/ to spin up local instances of LiteLLM and Langfuse.

You can do that by using deepinfra to manage your billing. It's pay-as-you-go and they have a pass-through virtual target for Google Gemini.

Deepinfra token usage updates every time you switch to the tab if it is opened to the usage page so it is possible to see updates even every second

Is this on Google AI Studio or Google Vertex or both?

In in the meantime, I'm using openrouter.

You need LLM Ops. YC happens to have invested in Langfuse, which is if you're serious about tracking metrics, you'll appreciate the rest, too.

And before you ask: yes, for cached content and batch completion discounts you can accommodate both—just needs a bit of logic in your completion-layer code.

openrouter, i dont think anyone should use google direct till they fix their shit billing

If it's zero effort, then why do devs need to adapt fast? And wouldn't adapting be incredibly easy?

The only disadvantage to not using these tools would be that your current output is slower. As soon as your employer asks for more or you're looking for a new job, you can just turn on AI and be as fast as everyone who already uses it.

i'm surprised by no line of css html in months. maybe it's an exageration and that's okay.

However, just today i was building a website for fun with gemini and had to manually fix some issues with css that he struggled with. as it often happens, trying to let it repair the damage only made it go into a pit of despair (for me). i fixed the issues in about a glance and 5 minutes. This is not to say it's bad, but sometimes it still makes absurd mistakes and can't find a way to solve them

> are now done in hours to a higher level of quality

However, I feel that there is a big difference between the models. In my tests, using Cursor, Clause 3.7 Sonnet has a much more refined "aesthetic sense" than other models. Many times I ask "make it more beautiful" and it manages to improve, where other models just can't understand it.

Elaborate, because I have serious doubts about this.

If we're talking about just slapping on tailwind+component-library(e.g. shadcn-ui, material), then that's just one step-above using no-code solutions. Which, yes, that works well. But if someone didn't need customized logic, then it was always possible to just hop on fiverr or use some very simple template-based tools to accomplish this.

If we're talking more advanced logic, understanding aesthetics, etc. Then I'd say it's much worse than other coding areas like backend, because they work on a visual and ux level beyond just code which is just text manipulation (and what llms excel at). In other words, I think the results are still very shallow beyond first impressions.

What does your tool and model stack look like for this?

I find they achieve acceptable, but not polished levels of work.

I'm not even a designer, but I care about the consistency of UI design and whether the overall experience is well-organized, aligned properly, things are placed in a logical flow for the user, and so on.

While I'm pro-AI tooling and use it heavily, and these models usually provide a good starting point, I can't imagine shipping the slop without writing/editing a line of HTML for anything that's interaction-heavy.

What tools do you use?