Prompt engineering playbook for programmers

Context is king.

Start out with Typescript and have it answer data science questions - won't know its way around.

Start out with Python and ask the same question - great answers.

LLMs can't (yet) really transfer knowledge between domains, you have to prime them in the right way.

Even role prompting is totally useless imo. Maybe it was a thing with GPT3, but most of the LLMs already know they're "expert programmers". I think a lot of people are just deluding themselves with "prompt engineering".

Be clear with your requirements. Add examples, if necessary. Check the outputs (or reasoning trace if using a reasoning model). If they aren't what you want, adjust and iterate. If you still haven't got what you want after a few attempts, abandon AI and use the reasoning model in your head.

As a clarification, we used fine tuning more than prompt engineering because low or few-shot prompt engineering did not work for our use case.

Regarding point 3, my colleagues and i studied this for a use case in science: https://doi.org/10.1038/s41467-024-45563-x

Did you find it hard to create structured output while also trying to make it reason in the same prompt?

That's exactly how I started using them as well. 1. Give it just enough context, the assumptions that hold and the goal. 2. Review answer and iterate on the initial prompt. It is also the economical way to use them. I've been burned one too many times by using agents (they just spin and spin, burn 30 dollars for one prompt and either mess the code base or converge on the previous code written ).

I also feel the need to caution others that by letting the AI write lots of code in your project it makes it harder to advance it, evolve it and just move on with confidence (code you didn't think about and write it doesn't stick as well into your memory).

That has been my conclusion too but how do you explain the long ass prompt by AI labs: https://docs.anthropic.com/en/release-notes/system-prompts#m...

For another kind of task, a colleague had written a very verbose prompt. Since I had to integrate it, I added some CRUD ops for prompts. For a test, I made a very short one, something like "analyze this as a <profession>". The output was pretty much comparable, except that the output on the longer prompt contained (quite a few) references to literal parts of that prompt. It wasn't incoherent, but it was as if that model (gemini 2.5, btw) has a basic response for the task it extracts from the prompt, and merges the superfluous bits in. It would seem that, at least for this particular task, the model cannot (easily) be made to "think" differently.

I’d have to hunt, but there is evidence that using the vocabulary of an expert versus a layman will produce better results. Which makes sense since places where people talk “normally” in spaces are more likely to be incorrect. Whereas in places where people speak in the in the professional vernacular they are more likely to be correct. And the training will associate them together in their spaces.

Yeah I had this experience today where I had been running code review with a big detailed prompt in CLAUDE.md but then I ran it in a branch that did not have that file yet and got better results.

At what point does it become programming in legalese?

Same here: it starts with a relatively precise need, keeping a roadmap in mind rather than forcing one upfront. When it involves a technology I'm unfamiliar with, I also ask questions to understand what certain things mean before "copying and pasting".

I've found that with more advanced prompts, the generated code sometimes fails to compile, and tracing the issues backward can be more time consuming than starting clean.

I think part of the skill in using LLMs is getting a sense for how to effectively break problems down, and also getting a sense of when and when not to do it. The article also mentions this.

I think we'll also see ways of restructuring, organizing, and commenting code to improve interaction with LLMs. And also expect LLMs to get better at doing this, and maybe suggesting ways for programmers to break problems down that it is struggling with.

I think the intent of prompt engineering is to get better solutions quicker, in formats you want. But yeah, ideally the model just "knows" and you don't have to engineer your question

It reminds me that we had the same hype and FOMO when Google became popular. Books were being written on the subject and you had to buy those or you would become a caveman in a near future. What happened is that anyone could learn the whole thing in a day and that was it, no need to debate about whether you would miss anything if you didn't knew all those tools.

I think there are people for whom reading a prompt guide (or watching an experienced user) will be very valuable.

Many people just won't put any conscious thought into trying to get better on their own, though some of them will read or watch one thing on the topic. I will readily admit to picking up several useful tips from watching other people use these tools and from discussing them with peers. That's improvement that I don't think I achieve by solely using the tools on my own.

Many years ago there were guides on how to write user stories: “As a [role], I want to be able to do [task] so I can achieve [objective]”, because it was useful to teach high-level thinkers how to communicate requirements with less ambiguity.

It may seem simple, but in my experience even brilliant developers can miss or misinterpret unstructured requirements, through no fault of their own.

It's at least useful for seeing how other people are being productive with these tools. I also sometimes find a clever idea that improves that I'm already doing.

And documenting the current state of this space as well. It's easy to have tried doing something a year ago and think they're still bad.

I also usually prefer researching some area before reinventing the wheel by trial/failure myself. I appreciate when people share what they've discovered with their own their time, as I don't always have all the time in the world to explore it as I would if I were still a teen.

> Since when the ability to write proper and meaningful sentences became engineering?

Since what's proper and meaningful depends on a lot of variables. Testing these, keeping track of them, logging and versioning take it from "vibe prompting" to "prompt engineering" IMO.

There are plenty of papers detailing this work. Some things work better than others (do this and this works better than don't do this - pink elephants thing). Structuring is important. Style is important. Order of information is important. Re-stating the problem is important.

Then there's quirks with family models. If you're running an API-served model you need internal checks to make sure the new version still behaves well on your prompts. These checks and tests are "prompt engineering".

I feel a lot of people take the knee-jerk reaction to the hype and miss critical aspects because they want to dunk on the hype.

God, do you get off of word “engineer”? Is it cultural?

AI sloperators are desperate to make it look like they are actually doing something.

I would caution against thinking it's impossible even if it's not something you've personally experienced. Prompt engineering is necessary (but not sufficient) to creating high leverage outcomes from LLMs when solving complex problems.

Without it, the chances of getting to a solution are slim. With it, the chances of getting to 90% of a solution and needing to fine tune the last mile are a lot higher but still not guaranteed. Maybe the phrase "prompt engineering" is bad and it really should be called "prompt crafting" because there is more an element of craft, taste, and judgment than there is durable, repeatable principles which are universally applicable.

I came across a pretty amusing analogy back when prompt "engineering" was all the rage a few years ago.

> Calling someone a prompt engineer is like calling the guy who works at Subway an artist because his shirt says ‘Sandwich Artist.’

All jokes aside I wouldn't get to hung up on the title, the term engineer has long since been diluted to the point of meaninglessness.

https://jobs.mysubwaycareer.eu/careers/sandwich-artist.htm

Because your imagination stopped at chat interface asking for funny cat pictures.

There are prompts to be used with API an inside automated workflows and more to it.

Isn't this basically the same argument that comes up all the time about software engineering in general?

I understand your point, but don't we already have e.g. AWS engineers? Or I believe SAP/Tableau/.. engineers?

IT is where words and their meanings come to die. I wonder if words ever needed to mean something :p

For real. Editing prompts bares no resemblance to engineering at all, there is no accuracy or precision. Say you have a benchmark to test against and you're trying to make an improvement. Will your change to the prompt make the benchmark go up? Down? Why? Can you predict? No, it is not a science at all. It's just throwing shit and examples at the wall in hopes and prayers.

"Common sense" doesn't exist. It is a term people use when they can't explain what they actually mean.

The less you use “crutches” the better you get, right? Judging by your comment, you don’t use Google, Stack Overflow, public forums (for programming assistance), books, courses, correct?

Markets shift people to where they are needed with salaries as a price signal.

There aren't enough software engineers to create the software the world needs.

Lots of those „prompt engineering” things would be nice to teach to business people as they seem to lack common sense.

Like writing out clear requirements.

Yeah I don’t get it. By the time I’m done writing all these prompts to get what I want, refining over and over not to mention the waiting time for the characters to appear on the screen, I could have written what I want myself. I find LLMs more useful as a quick documentation search and for the most basic refactoring and template building.

Your boss (or CEO) probably wouldn't.

Writing the code would also save you time over whatever I just read.

Your native language is now a programming language.

What shall we teach them instead? Creating bazillion CRUDs? Serializing/Deserializing JSON?

Including a coding style guide can help the code looks like what you want. Also including an explanation of the project structure, and overall design of the code base. Always specify what libraries it should make use of (or it'll bring in anything or implement stuff a library has already).

You can also make the AI review itself. Have it modify code, than ask to review the code, than ask to address review comments, and iterate until it has no more comments.

Use an agentic tool like Claude Code or Amazon Q CLI. Then ask it to run tests after code changes and to address all issues until test pass. Make sure to tell it not to change the test code.

I found that presenting your situation and asking for a plan/ideas + « do not give me code. Make sure you understand the requirements and ask questions if needed.» works much better for me.

It also allows me to more easily control what the llm will do and not end up reviewing and throwing 200 lines of code.

In a nextjs + vitest context, I try to really outline which tests I want and give it proper data examples so that it does not cheat around mocking fake objects.

I do not buy into the whole you’re a senior dev etc. Most people use Claude for coding so I guess it’s engrained by default.

And to add to it, here's my experience: sometimes you spend a lot of time on this upfront prompt engineering and get bad results and sometimes you just YOLO it and get good results. It's hard to advocate for a determined strategy for prompt engineering when the tool you're prompting itself is non-deterministic.

Edit: I also love that the examples come with "AI’s response to the poor prompt (simulated)"

It's really just technical writing. Majority of tricks from the GTP-4 era are obsolete with reasoning models.

Seriously, I seem to get good results by just "being a good communicator". Not only that, but as the tools get better, prompt engineering should get less important.

In most HN LLM Programming discussion there's a vocal majority saying LLMs are useless. Now we have this commenter saying all they need to do is vibe and it all works out.

WHICH IS IT?