GPTs and Feeling Left Behind

I'm completely equally lost the other way.

I've went through multiple phases of LLM usage for development.

GPT3.5 era: wow this is amazing, oh. everything is hallucinated. not actually as useful as I first thought

GPT4 era: very helpful as stackoverflow on steroids.

Claude 3.5 Sonnet: have it open pretty much all the time, constantly asking questions and getting it to generate simple code (in the web UI) when it goes down actually feels very old school googling stuff. Tried a lot of in IDE AI "chat" stuff but hugely underwhelmed.

Now: rarely open IDE as I can do (nearly) absolutely everything in Claude Code. I do have to refactor stuff every so often "manually", but this is more for my sanity and understanding of the codebase..

To give an example of a task I got Claude code to do today in a few minutes which would take me hours. Had a janky looking old admin panel in bootstrap styles that I wanted to make look nice. Told Claude code to fetch the marketing site for the project. Got it to pull CSS, logos, fonts from there using curl and apply similar styling to the admin panel project. Within 10 mins it was looking far, far better than I would have ever got it looking (at least without a designers help). Then got it to go through the entire project (dozens of screens) and update "explanation" copy - most of which was TODO placeholders to explain what everything did properly. I then got it to add an e2e test suite to the core flows.

This took less than an hour while I was watching TV. I would have almost certainly _never_ got around to this before. I'd been meaning to do all this and I always sigh when I go into this panel at how clunky it all is and hard to explain to people.

You're (they're?) not alone. This mirrors every experience I've had trying to give them a chance. I worry that I'm just speaking another language at this point.

EDIT: Just to add context seeing other comments, I almost exclusively work in C++ on GPU drivers.

One blogpost I found on HN completely leveled up how I use LLMs for coding: https://harper.blog/2025/02/16/my-llm-codegen-workflow-atm/

Having the AI ask me questions and think about the PRD/spec ultimately made me a better system designer.

I kept hearing about Claude Code for a while and never really tried it until a week ago. I used it to prototype some Mac app ideas and I quickly realized how useful it was at getting prototypes up and running very, very quickly, like within minutes. It saves so much time with boilerplate code that I would've had to type out by hand and have done hundreds of times before.

With my experience, I wonder what the author of this blog post has tried to do to complete a task as that might make a difference on why they couldn't get much use out of it. Maybe other posters can chime in on how big of a difference programming language and size of project can make. I did find that it was able to glean how I had architected an app and it was able to give feedback on potential refactors, although I didn't ask it to go that far.

Prior to trying out Claude Code, I had only used ChatGPT and DeepSeek to post general questions on how to use APIs and frameworks and asking for short snippets of code like functions to do text parsing with regexes, so to be honest I was very surprised at what the state of the art could actually do, at least for my projects.

I have a degree in CS from MIT and did professional software engineering from 2004 - 2020.

I recently started a company in another field and haven’t done any real development for about 4 years.

Earlier this summer I took a vacation and decided to start a small software hobby project specific to my industry. I decided to try out Cursor for the first time.

I found it incredibly helpful at saving time implementing all the bullshit involved in starting a new code base - setting up a build system, looking up libraries and APIs, implementing a framework for configuration and I/O, etc.

Yes, I still had to do some of the hard parts myself, and (probably most relevant) I still had to understand the code it was writing and correct it when it went down the wrong direction. I literally just told Cursor “No, why do it that way when you could do it much simpler by X”, and usually it fixed it.

A few times, after writing a bunch of code myself, I compiled the project for the first time in a while and (as one does) ran into a forest of inscrutable C++ template errors. Rather than spend my time scrolling through all of them I just told cursor “fix the compile errors”, and sure enough, it did it.

Another example - you can tell it things like “implement comparison operators for this class”, and it’s done in 5 seconds.

As the project got more complicated, I found it super useful to write tests for behaviors I wanted, and just tell it “make this test pass”. It really does a decent job of understanding the codebase and adding onto it like a junior developer would.

Using an IDE that gives it access to your whole codebase (including build system and tests) is key. Using ChatGPT standalone and pasting stuff in is not where the value is.

It’s nowhere near able to do the entire project from scratch, but it saved me from a bunch of tedious work that I don’t enjoy anyway.

Seems valuable enough to me!

I think what people are missing is that they work sometimes and sometimes they don't work.

People think "Oh, it works better when somebody else does it" or "There must be some model that does better than the one I am using" or "If I knew how to prompt better I'd get better results" or "There must be some other agentic IDE which is better than the one I am using."

All those things might be true but they just change the odds, they don't change the fact that it works sometimes and fails other times.

For instance I asked an agent to write me a screen to display some well-typed data. It came up with something great right away that was missing some fields and had some inconsistent formatting but it fixed all those problems when I mentioned them -- all speaking the language of product managers and end users. The code quality was just great, as good as if I wrote it, maybe better.

Plenty of times it doesn't work out like that.

I was working on some code where I didn't really understand the typescript types and fed it the crazy error messages I was getting and it made a try to understand them and didn't really, I used it as a "rubber duck" over the course of a day or two and working with it I eventually came to understand what was wrong and how to fix and I got into a place that I like and when there is an error I can understand it and it can understand it too.

Sometimes it writes something that doesn't typecheck and I tell it to run tsc and fix the errors and sometimes it does a job I am proud of and other times it adds lame typeguards like

   if (x && typeof x === "object") x.someMethod()

Give it essentially the same problem, say writing tests in Java, and it might take very different approaches. One time it will use the same dependency injection framework used in other tests to inject mocks into private fields, other times it will write some a helper method to inject the mocks into private fields with introspection directly.

You might be able to somewhat tame this randomness with better techniques but sometimes it works and sometimes it doesn't and if I just told you about the good times or just told you about the bad times it would be a very different story.

The worst thing is when LLMs introduce subtle bugs into code and one just can't spot them quickly. I was recently doing some Langfuse integration and used Cursor to generate skeleton code for pushing some traces/scores quickly. The generated code included one parameter "score_id" that was undocumented in Langfuse but somehow was accepted and messed the whole tracking up. Even after multiple passes of debugging I couldn't figure out what the issue with tracking was, until I asked another LLM to find any possible issues with the code, that promptly marked those score_id lines.

Haven’t even really tried them. The sand is shifting way too fast. Once things stabilize and other people figure out how to really use them I’ll probably start but for now it just feels like effort that will have been wasted.

Well, it takes a while to learn Vim and then get value from it.

It also takes a while to learn using an LLM and get value from it.

The keys are how to build prompts, ways of working, and guidelines that help the AI stay focused.

You end up spending much more time guiding and coaching rather than coding, that can take a while to get used to.

Eventually though, you will master it and be able to write secure, fast code far beyond what you could have done by yourself.

Note: Also, prep yourself for incoming hate every time you make claims like that! If you write bad code, it's your fault. If your LLM writes bad code, you're a moron! hah

I think back to the first version of ChatGPT and I would pick it up once in a while, ask it something or chat with it, and then be like... this is cool but I don't know wtf I would use it for, now I use a GPT at least a couple of times a day. Granted, the LLMs have obviously become considerably more capable, but I do believe part of it is I've also learned how to use them and what to use them for, I'm at the point now where I can generally predict the output of what I'm asking for - I don't know if that's the norm (it mostly gives me exactly what I want) I do know how I use them today and how I used them when they first came out is quite different. I guess all that is to say, imo how you prompt them really matters, and that takes time to learn.

I really wish posts like this included the parameters that they were using. What model? What was the question? How many shots? Etc etc

You’re going to get vastly different responses if you’re using Opus versus 4o.

I have a feeling this person is using far-from-frontier models, totally disconnected from the development environment.

Using, like, gpt-4o is extremely not useful for programming. But using Claude Code in your actual repo is insanely useful.

Gotta use the right tool + model.

In the end, the greatest use I get from coding agents and stuff is hijacking the Stack Overflow principle - it's much easier to trick myself into correcting the poor code Claude generates than it is to start writing code from a blank slate.

This exactly mirrors my experience. I can't see the whole LLM/GPT thing as anything but another blockchain level scam. It isn't zero value it is actually a negative value as the time it takes is an opportunity cost.

I find myself on both sides actually.

I did have some great luck producing quite useful and impactful code. But also lost time chasing tiny changes.

At what point does clinging to driving a manual transmission round the track go from a sign of practical skill to a sign of stubborn arrogance?

I spend a fair amount of time on open source and one thing I noticed is that in real pieces of software it doesn't look like all these 10x and 100x AI engineers are anywhere to be found.

VLC has like 4000 open issues. Why aren't the AI geniuses fixing these? Nobody has ever any actual code to show, and if they do it's "here's an LED that blinks every time my dog farts, I could've never done it on my own!". I'm feeling like Charlie in that episode of It's Always Sunny with his conspiracy dashboard. All these productivity gurus don't actually exist in the real world.

Can anybody show me their coding agent workflow on a 50k LOC C codebase instead of throwaway gimmick examples. As far as I'm concerned these things can't even understand pointers

A lot of this is classic gaslighting. Gaslighting is the manipulation of someone into questioning their perception of reality (per Wikipedia).

Basically, a lot of people who are experts are being told this story and they think they are the only one who doesn't get it.

There are plenty of gains to be had with AI/LLMs but just not in the way it's typically marketed.