> What this interaction shows is how much knowledge you need to bring when you interact with an LLM. The “one big flaw” Claude produced in the middle would probably not have been spotted by someone less experienced with crypto code than this engineer obviously is. And likewise, many people would probably not have questioned the weird choice to move to PBKDF2 as a response
For me this is the key takeaway. You gain proper efficiency using LLMs when you are a competent reviewer, and for lack of a better word, leader. If you don't know the subject matter as well as the LLM, you better be doing something non-critical, or have the time to not trust it and verify everything.
donatj · 5h ago
My question is kind of in this brave new world, where do the domain experts come from? Whose going to know this stuff?
svara · 5h ago
LLMs make learning new material easier than ever. I use them a lot and I am learning new things at an insane pace in different domains.
The maximalists and skeptics both are confusing the debate by setting up this straw man that people will be delegating to LLMs blindly.
The idea that someone clueless about OAuth should develop an OAuth lib with LLM support without learning a lot about the topic is... Just wrong. Don't do that.
But if you're willing to learn, this is rocket fuel.
junon · 4h ago
On the flip side, I wanted to see what common 8 layer PCB stackups were yesterday. ChatGPT wasn't giving me an answer that really made sense. After googling a bit, I realized almost all of the top results were AI generated, and also had very little in the way of real experience or advice.
It was extremely frustrating.
roxolotl · 4h ago
This is my big fear. We’re going to end up in a world where information that isn’t common is significantly more difficult to find than it is today.
andersa · 4h ago
It's going to be like the pre-internet dark ages, but worse. Back then you only didn't find the information. Now, you find unlimited information, but it is all wrong.
koolba · 1h ago
Content from before the AI Cambrian explosion is going to be treated like low-background steel.
I don't know, this sounds a lot like in the late 90s when we heard a lot about how anyone could put information on the internet and that you shouldn't trust what you read online.
Well it turns out you can manage just fine.
You shouldn't blindly trust anything. Not what you read, not what people say.
Using LLMs effectively is a skill too, and that does involve deciding when and how to verify information.
andersa · 3h ago
The difference is in scale. Back then, only humans were sometimes putting up false information, and other humans had a chance to correct it. Now, machines are writing infinitely more garbage than humans can ever read. Search engines like Google are already effectively unusable.
vohk · 22m ago
I think there will be solutions, although I don't think getting there will be pretty.
Google's case (and Meta and spam calls and others) is at least in part an incentives problem. Google hasn't been about delivering excellent search to users for a very long time. They're an ad company and their search engine is a tool to better deliver ads. Once they had an effective monopoly, they just had to stay good enough not to lose it.
I've been using Kagi for a few years now and while SEO spam and AI garbage is still an issue, it is far less of one than with Google or Bing. My conclusion is these problems are at least somewhat addressable if doing so is what gets the business paid.
But I think a real long term solution will have to involved a federated trust model. It won't be viable to index everything dumped on the web; there will need to be a component prioritizing trust in the author or publisher. If that follows the same patterns as email (ex: owned by Google and Microsoft), then we're really screwed.
skeeter2020 · 3h ago
>> Well it turns out you can manage just fine.
You missed the full context: you would never be able to trust a bunch of amateur randos self-policing their content. Turns out it's not perfect but better than a very small set of professionals; usually there's enough expertise out there, it's just widely distributed. The challenge this time is 1. the scale, 2. the rate of growth, 3. the decline in expertise.
>> Using LLMs effectively is a skill too, and that does involve deciding when and how to verify information.
How do you verify when ALL the sources are share the same AI-generated root, and ALL of the independent (i.e. human) experts have aged-out and no longer exist?
svara · 49m ago
> How do you verify when ALL the sources are share the same AI-generated root,
Why would that happen? There's demand for high quality, trustworthy information and that's not going away.
When asking an LLM coding questions, for example, you can ask for sources and it'll point you to documentation. It won't always be the correct link, but you can prod it more and usually get it, or fall back to searching the docs the old fashioned way.
m11a · 2h ago
The solution is kagi.com imo.
Before AI generated results, the first page of Google was SEO-optimised crap blogs. The internet has been hard to search for a while.
No comments yet
endofreach · 2h ago
It will dawn on non-tech people soon enough. Hopefully the "AI" (LLM) hypetrain riders will follow.
catlifeonmars · 49m ago
I think what’s missing here is you should start by reading the RFCs. RFCs tend to be pretty succinct so I’m not really sure what a summarization is buying you there except leaving out important details.
(One thing that might be useful is use the LLM as a search engine to find the relevant RFCs since sometimes it’s hard to find all of the applicable ones if you don’t know the names of them already.)
I really can’t stress this enough: read the RFCs from end to end. Then read through the code of some reference implementations. Draw a sequence diagram. Don’t have the LLM generate one for you, the point is to internalize the design you’re trying to implement against.
By this time you should start spotting bugs or discrepancies between the specs and implementations in the wild. That’s a good sign. It means you’re learning
skeeter2020 · 3h ago
>> LLMs make learning new material easier than ever.
feels like there's a logical flaw here, when the issue is that LLMs are presenting the wrong information or missing it all together. The person trying to learn from it will experience Donald Rumsfield's "unknown unknowns".
I would not be surprised if we experience an even more dramatic "Cobol Moment" a generation from now, but unlike that one thankfully I won't be around to experience it.
paradox242 · 1h ago
The value of LLMs is that they do things for you, so yeah the incentive is to have them take over more and more of the process. I can also see a future not far into the horizon where those who grew up with nothing but AI are much less discerning and capable and so the AI becomes more and more a crutch, as human capability withers from extended disuse.
a13n · 1h ago
If the hypothesis is that we still need knowledgeable people to run LLMs, but the way you become knowledgeable is by talking to LLMs, then I don’t think the hypothesis will be correct for long..
svara · 47m ago
You inserted a hidden "only" there to make it into a logical sounding dismissive quip.
You don't get knowledge by ONLY talking to LLMs, but they're a great tool.
mwigdahl · 1h ago
We need knowledgeable people to run computers, but you can become knowledgeable about computers by using computers to access learning material. Seems like that generalizes well to LLMs.
elvis10ten · 4h ago
> LLMs make learning new material easier than ever. I use them a lot and I am learning new things at an insane pace in different domains.
With learning, aren’t you exposed to the same risks? Such that if there was a typical blind spot for the LLM, it would show up in the learning assistance and in the development assistance, thus canceling out (i.e unknown unknowns)?
Or am I thinking about it wrongly?
sulam · 4h ago
If you trust everything the LLM tells you, and you learn from code, then yes the same exact risks apply. But this is not how you use (or should use) LLMs when you’re learning a topic. Instead you should use high quality sources, then ask the LLM to summarize them for you to start with (NotebookLM does this very well for instance, but so can others). Then you ask it to build you a study plan, with quizzes and exercises covering what you’ve learnt. Then you ask it to setup a spaced repetition worksheet that covers the topic thoroughly. At the end of this you will know the topic as well as if you’d taken a semester-long course.
One big technique it sounds like the authors of the OAuth library missed is that LLMs are very good at generating tests. A good development process for today’s coding agents is to 1) prompt with or create a PRD, 2) break this down into relatively simple tasks, 3) build a plan for how to tackle each task, with listed out conditions that should be tested, 3) write the tests, so that things are broken, TDD style and finally 4) write the implementation. The LLM can do all of this, but you can’t one-shot it these days, you have to be a human in the loop at every step, correcting when things go off track. It’s faster, but it’s not a 10x speed up like you might imagine if you think the LLM is just asynchronously taking a PRD some PM wrote and building it all. We still have jobs for a reason.
kentonv · 4m ago
I did actually use the LLM to write tests, and was pleased to see the results, which I thought were pretty good and thorough, though clearly the author of this blog post has a different opinion.
But TDD is not the way I think. I've never been able to work that way (LLM-assisted or otherwise). I find it very hard to write tests for software that isn't implemented yet, because I always find that a lot of the details about how it should work are discovered as part of the implementation process. This both means that any API I come up with before implementing is likely to change, and also it's not clear exactly what details need to be tested until I've fully explored how the thing works.
This is just me, other people may approach things totally differently and I can certainly understand how TDD works well for some people.
evnu · 3h ago
> Instead you should use high quality sources, then ask the LLM to summarize them for you to start with (NotebookLM does this very well for instance, but so can others).
How do you determine if the LLM accurately reflects what the high-quality source contains, if you haven't read the source? When learning from humans, we put trust on them to teach us based on a web-of-trust. How do you determine the level of trust with an LLM?
perching_aix · 2h ago
> When learning from humans, we put trust on them to teach us based on a web-of-trust.
But this is only part of the story. When learning from another human, you'll also actively try and devise whether they're trustworthy based on general linguistic markers, and will try to find and poke holes in what they're saying so that you can question intelligently.
This is not much different from what you'd do with an LLM, which is why it's such a problem that they're more convincing than correct pretty often. But it's not an insurmountable issue. The other issue is that their trustworthiness will wary in a different way than a human's, so you need experience to know when they're possibly just making things up. But just based on feel, I think this experience is definitely possible to gain.
ativzzz · 3h ago
Because summarizing is one of the few things LLMs are generally pretty good at. Plus you should use the summary to determine if you want to read the full source, kind of like reading an abstract for a research paper before deciding if you want to read the whole thing.
Bonus: the high quality source is going to be mostly AI written anyway
sroussey · 2h ago
Actually, LLMs aren’t that great for summarizing. It would be a boon for RAG workflows if they were.
I’m still on the lookout for a great model for this.
perching_aix · 4h ago
When I'm exploring a topic, I make sure to ask for links to references, and will do a quick keyword search in there or ask for an excerpt to confirm key facts.
This does mean that there's a reliance on me being able to determine what are key facts and when I should be asking for a source though. I have not experienced any significant drawbacks when compared to a classic research workflow though, so in my view it's a net speed boost.
However, this does mean that a huge variety of things remain out of reach for me to accomplish, even with LLM "assistance". So there's a decent chance even the speed boost is only perceptual. If nothing else, it does take a significant amount of drudgery out of it all though.
motorest · 4h ago
> With learning, aren’t you exposed to the same risks? Such that if there was a typical blind spot for the LLM, it would show up in the learning assistance and in the development assistance, thus canceling out (i.e unknown unknowns)?
I don't think that's how things work. In learning tasks, LLMs are sparring partners. You present them with scenarios, and they output a response. Sometimes they hallucinate completely, but they can also update their context to reflect new information. Their output matches what you input.
conradev · 1h ago
Just wrong. Don’t do that
I’d personally qualify this: don’t ship that code, but absolutely do it personally to grow if you’re interested.
I’ve grown the most when I start with things I sort of know and I work to expand my understanding.
threeseed · 3h ago
Learning from LLMs is akin to learning from Joe Rogan.
You are getting a stylised view of a topic from an entity who lacks the deep understanding needed to be able to fully distill the information. But it is enough to gain enough knowledge for you to feel confident which is still valuable but also dangerous.
And I assure you that many, many people are delegating to LLMs blindly e.g. it's a huge problem in the UK legal system right now because of all the invented case law references.
diogocp · 1h ago
> You are getting a stylised view of a topic from an entity who lacks the deep understanding
Isn't this how every child learns?
Unless his father happens to be king of Macedonia, of course.
slashdev · 3h ago
It depends very much on the quality of the questions. I get deep technical insight into questions I can't find anything on with Google.
blibble · 4h ago
how do you gain anything useful from a sycophantic tutor that agrees with everything you say, having being trained to behave as if the sun shines out of your rear end?
making mistakes is how we learn, and if they are never pointed out...
svara · 3h ago
It's a bit of a skill. Gaining an incorrect understanding of some topic is a risk anyway you learn, and I don't feel it's greater with LLMs than many of the alternatives.
Sure, having access to legit experts who can tutor you privately on a range of topics would be better, but that's not realistic.
What I find is that if I need to explore some new domain within a field I'm broadly familiar with, just thinking through what the LLM is saying is sufficient for verification, since I can look for internal consistency and check against things I know already.
When exploring a new topic, often times my questions are superficial enough for me to be confident that the answers are very common in the training data.
When exploring a new topic that's also somewhat niche or goes into a lot of detail, I use the LLM first to get a broad overview and then drill down by asking for specific sources and using the LLM as an assistant to consume authoritative material.
blibble · 3h ago
this "logic" applied across society will lead to our ruin
svara · 1h ago
Say more?
perching_aix · 2h ago
> from a sycophantic tutor that agrees with everything you say
You know that it's possible to ask models for dissenting opinions, right? Nothing's stopping you.
> and if they are never pointed out...
They do point out mistakes though?
wslh · 22m ago
Another limitation of LLMs lies in their inability to stay in sync with novel topics or recently introduced methods, especially when these are not yet part of their training data or can't be inferred from existing patterns.
It's important to remember that these models depend not only on ML breakthroughs but also on the breadth and freshness of the data used to train them.
That said, the "next-door" model could very well incorporate lessons from the recent Cloudflare OAuth Library issues, thanks to the ongoing discussions and community problem-solving efforts.
therealpygon · 2h ago
And yet, human coders may do that exact type of thing daily, producing far worse code. I find it humorous at how much higher of a standard is applied to LLMs in every discussion when I can guarantee those exact some coders likely produce their own bug-riddled software.
We’ve gone from skeptics saying LLMs can’t code, to they can’t code well, to they can’t produce human-level code, to they are riddled with hallucinations, to now “but they can’t one-shot code a library without any bugs or flaws” and “but they can only one-shot code, they can’t edit well” even tho recents coding utilities have been proving that wrong as well. And still they say they are useless.
Some people just don’t hear themselves or see how AI is constantly moving their bar.
brookst · 1h ago
And now the complaint is that the bugs are too subtle. Soon it will be that the overall quality is too high, leading to a false sense of security.
belter · 4h ago
> But if you're willing to learn, this is rocket fuel.
LLMs will tell you 1 or 2 lies for each 20 facts. Its a hard way to learn. They cant even get their urls right...
diggan · 4h ago
> LLMs will tell you 1 or 2 lies for each 20 facts. Its a hard way to learn.
That was my experience when growing up with school also, except you got punished one way or another for speaking up/trying to correct the teacher. If I speak up with the LLM they either explain why what they said is true, or corrects themselves, 0 emotions involved.
> They cant even get their urls right...
Famously never happens with humans.
belter · 3h ago
You are ignoring the fact that the types of mistakes or lies are of a different nature.
If you are in class, and you incorrectly argue, there is a mistake in an explanation of Derivatives or Physics, but you are the one in error, your Teacher hopefully, will not say: "Oh, I am sorry you are absolutely correct. Thank you for your advice.."
diggan · 3h ago
Yeah, no of course if I'm wrong I don't expect the teacher to agree with me, what kind of argument is that? I thought it was clear, but the base premise of my previous comment is that the teacher is incorrect and refuse corrections...
belter · 2h ago
My point is a teacher will not do something like this:
- Confident synthesis of incompatible sources:
LLM: “Einstein won the 1921 Nobel Prize for his theory of relativity, which he presented at the 1915 Solvay Conference.”
Or
- Fabricated but plausible citations:
LLM: “According to Smith et al., 2022, Nature Neuroscience, dolphins recognise themselves in mirrors.”
There is no such paper...model invents both authors and journal reference
And this is the danger of coding with LLMs....
diggan · 2h ago
I don't know what reality you live in, but it happens that teachers are incorrect, no matter what your own personal experience have been. I'm not sure how this is even up for debate.
What matters is how X reacts when you point out it wasn't correct, at least in my opinion, and was the difference I was trying to highlight.
belter · 50m ago
A human tutor typically misquotes a real source or says “I’m not sure”
An LLM, by contrast, will invent a flawless looking but nonexistent citation.
Even a below average teacher doesn’t churn out fresh fabrications every tenth sentence.
Because a teacher usually cites recognizable material, you can check the textbook
and recover quickly. With an LLM you first have to discover the source never existed. That verification cost is higher, the more complex task you are trying to achieve.
A LLM will give you a perfect paragraph about the AWS Database Migration service,
the list of supported databases, and then include in there a data flow
like on-prem to on-prem data that is not supported...Relying on an LLM is like flying with a friendly copilot but who has multiple personality disorder. You dont know which day he will forget to take his meds :-)
Stressful and mentally exhausting in a different kind of way....
signatoremo · 3h ago
And you are saying human teachers or online materials won't lie to you once or twice for every 20 facts? no matter how small. Did you do any comparison?
skeeter2020 · 3h ago
it's not jsut the lies, but how it lies and the fact that LLMs are very hesitant to call out humans on their BS
brookst · 1h ago
Is this the newest meme?
Me: “explain why radioactive half-life changes with temperature”
ChatGPT 4o: “ Short answer: It doesn’t—at least not significantly. Radioactive Half-Life is (Almost Always) Temperature-Independent”
…and then it goes on to give a few edge cases where there’s a tiny effect.
belter · 3h ago
You are missing the point. See my comment to @diggan in this thread. LLMs lie in a different way.
maegul · 5h ago
This, for me, has been the question since the beginning. I’m yet to see anyone talk/think about the issue head on too. And whenever I’ve asked someone about it, they’ve not had any substantial thoughts.
PUSH_AX · 4h ago
Engineers will still exist and people will vibe code all kinds of things into existence. Some will break in spectacular ways, some of those projects will die, some will hire a real engineer to fix things.
I cannot see us living in a world of ignorance where there are literally zero engineers and no one on the planet understands what's been generated. Weirdly we could end up in a place where engineering skills are niche and extremely lucrative.
No comments yet
paradox242 · 1h ago
The implication is that they are hoping to bridge the gap between current AI capabilities and something more like AGI in the time it takes the senior engineers to leave the industry. At least, that's the best I can come up with, because they are kicking out all of the bottom rings of the ladder here in what otherwise seems like a very shortsighted move.
shswkna · 5h ago
Most important question on this entire topic.
Fast forward 30
years and modern civilisation is entirely dependent on our AI’s.
Will deep insight and innovation from a human perspective perhaps come to a stop?
brookst · 1h ago
Did musical creativity end with synths and sequencers?
Tools will only amplify human skills. Sure, not everyone will choose to use tools for anything meaningful, but those people are driving human insight and innovation today anyway.
Earw0rm · 4h ago
No. Even with power tools, construction and joinery are physical work and require strength and skill.
What is new is that you'll need the wisdom to figure out when the tool can do the whole job, and where you need to intervene and supervise it closely.
So humans won't be doing any less thinking, rather they'll be putting their thinking to work in better ways.
skeeter2020 · 3h ago
to use your own example though, many of these core skills are declining, mechanized or viewed through a historical lens vs. application. I don't know if this is net good or bad, but it is very different. Maybe humans will think as you say, but it feels like there will be significantly less diverse areas of thought. If you look at the front page of HN as a snapshot of "where's tech these days" it is very homgenous compared to the past. Same goes for the general internet and the AI-content continues to grow. IMO published works are a precursor to future human discovery, forming the basis of knowledge, community and growth.
qzw · 4h ago
No, but it'll become a hobby or artistic pursuit, just like running, playing chess, or blacksmithing. But I personally think it's going to take longer than 30 years.
risyachka · 4h ago
Use it or lose it.
Experts will become those who use llm to learn and not to write code for them or solve tasks for them so they can build that skill.
kypro · 4h ago
In a few years hopefully the AI reviewers will be far more reliable than even the best human experts. This is generally how competency progresses in AI...
For example, at one point a human + computer would have been the strongest combo in chess, now you'd be insane to allow a human to critic a chess bot because they're so unlikely to add value, and statistically a human in the loop would be far more likely to introduce error. Similar things can be said in fields like machine vision, etc.
Software is about to become much higher quality and be written at much, much lower cost.
sarchertech · 4h ago
My prediction is that for that to happen we’ll need to figure out a way to measure software quality in the way we can measure a chess game, so that we can use synthetic data to continue improving the models.
I don’t think we are anywhere close to doing that.
kypro · 3h ago
Not really... If you're an average company you're not concerned about producing perfect software, but optimising for some balance between cost and quality. At some point companies via capitalist forces will naturally realise that it's more productive to not have humans in the loop.
A good analogy might be how machines gradually replaced textile workers in the 19th century. Were the machines better? Or was there a was to quantitatively measure the quality of their output? No. But at the end of the day companies which embraced the technology were more productive than those who didn't, and the quality didn't decrease enough (if it did at all) that customers would no longer do business with them – so these companies won out.
The same will naturally happen in software over the next few years. You'd be an moron to hire a human expert for $200,000 to critic a cybersecurity optimised model which costs maybe a 100th of the cost of employing a human... And this would likely be true even if we assume the human will catch the odd thing the model wouldn't because there's no such thing as perfect security – it's always a trade off between cost and acceptable risk.
Bookmark this and come back in a few years. I made similar predictions when ChatGPT first came out that within a few years agents would be picking up tickets and raising PRs. Everyone said LLMs were just stochastic parrots and this would not happen, well now it has and increasingly companies are writing more and more code with AI. At my company it's a little over 50% at the mo, but this is increasing every month.
marcusb · 5h ago
I’m puzzled when I hear people say ‘oh, I only use LLMs for things I don’t understand well. If I’m an expert, I’d rather do it myself.’
In addition to the ability to review output effectively, I find the more closely I’m able to describe what I want in the way another expert in that domain would, the better the LLM output. Which isn’t really that surprising for a statistical text generation engine.
diggan · 4h ago
I guess it depends. In some cases, you don't have to understand the black box code it gives you, just that it works within your requirements.
For example, I'm horrible at math, always been, so writing math-heavy code is difficult for me, I'll confess to not understanding math well enough. If I'm coding with an LLM and making it write math-heavy code, I write a bunch of unit tests to describe what I expect the function to return, write a short description and give it to the LLM. Once the function is written, run the tests and if it passes, great.
I might not 100% understand what the function does internally, and it's not used for any life-preserving stuff either (typically end up having to deal with math for games), but I do understand what it outputs, and what I need to input, and in many cases that's good enough. Working in a company/with people smarter than you tends to make you end up in this situation anyways, LLMs or not.
Though if in the future I end up needing to change the math-heavy stuff in the function, I'm kind of locked into using LLMs for understanding and changing it, which obviously feels less good. But the alternative is not doing it at all, so another tradeoff I suppose.
I still wouldn't use this approach for essential/"important" stuff, but more like utility functions.
ipaddr · 3h ago
Would you rather it be done incorrectly when others are expecting correctness or not at all? I would choose not at all.
diggan · 3h ago
Well, given the context is math in video games, I guess I'd chose "not at all", if there was no way for me to verify it's correct or not. But since I can validate, I guess I'd chose to do it, although without fully understanding the internals.
_heimdall · 4h ago
That's why outsource most other things in our life though, why would it be different with LLMs?
People don't learn how a car works before buying one, they just take it to a mechanic when it breaks. Most people don't know how to build a house, they have someone else build it and assume it was done well.
I fully expect people to similarly have LLMs do what the person doesn't know how and assume the machine knew what to do.
marcusb · 1h ago
> why would it be different with LLMs?
Because LLMs are not competent professionals to whom you might outsource tasks in your life. LLMs are statistical engines that make up answers all the time, even when the LLM “knows” the correct answer (i.e., has the correct answer hidden away in its weights.)
I don’t know about you, but I’m able to validate something is true much more quickly and efficiently if it is a subject I know well.
bradly · 5h ago
I've found llms are very quick to add defaults, fallbacks, rescues–which all makes it very easy for code to look like it is working when it is not or will not. I call this out three different places in my CLAUDE.md trying to adjust for this, and still occasionally get.
loandbehold · 1h ago
Over time AI coding tools will be able to research domain knowledge. Current "AI Research" tools are already very good at it but they are not integrated with coding tools yet. The research could look at both public Internet as well as company documents that contain internal domain knowledge. Some of the domain knowledge is only in people's heads. That would need to be provided by the user.
wslh · 1h ago
I'd like to add a practical observation, even assuming much more capable AI in the future: not all failures are due to model limitations, sometimes it's about external [world] changes.
For instance, I used Next.js to build a simple login page with Google auth. It worked great, even though I only had basic knowledge of Node.js and a bit of React.
Then I tried adding a database layer using Prisma to persist users. That's where things broke. The integration didn't work, seemingly due to recent versions in Prisma or subtle breaking updates. I found similar issues discussed on GitHub and Reddit, but solving them required shifting into full manual debugging mode.
My takeaway: even with improved models, fast-moving frameworks and toolchains can break workflows in ways that LLMs/ML (at least today) can't reason through or fix reliably. It's not always about missing domain knowledge, it's that the moving parts aren't in sync with the model yet.
ajmurmann · 5h ago
I've been using an llm to do much of a k8s deployment for me. It's quick to get something working but I've had to constantly remind it to use secrets instead of committing credentials in clear text. A dangerous way to fail. I wonder if in my case this is caused by the training data having lots of examples from online tutorials that omit security concerns to focus on the basics.
diggan · 4h ago
> my case this is caused by the training data having
I think it's caused by you not having a strong enough system prompt. Once you've built up a slightly reusable system prompt for coding or for infra work, where you bit by bit build it up while using a specific model (since different models respond differently to prompts), you end up getting better and better responses.
So if you notice it putting plaintext credentials in the code, add to the system prompt to not do that. With LLMs you really get what you ask for, and if you miss to specify anything, the LLM will do whatever the probabilities tells it to, but you can steer this by being more specific.
Imagine you're talking to a very literal and pedantic engineer who argues a lot on HN and having to be very precise with your words, and you're like 80% of the way there :)
ajmurmann · 4h ago
Yes, you are definitely right on that. I still find it a concerning failure mode. That said, maybe it's no worse than a junior copying from online examples without reading all the text some the code which of course has been very common also.
ants_everywhere · 5h ago
> It's quick to get something working but I've had to constantly remind it to use secrets instead of committing credentials in clear text.
This is going to be a powerful feedback loop which you might call regression to the intellectual mean.
On any task, most training data is going to represent the middle (or beginning) of knowledge about a topic. Most k8s examples will skip best practices, most react apps will be from people just learning react, etc.
If you want the LLM to do best practices in every knowledge domain (assuming best practices can be consistently well defined), then you have to push it away from the mean of every knowledge domain simultaneously (or else work with specialized fine tuned models).
As you continue to add training data it will tend to regress toward the middle because that's where most people are on most topics.
jstummbillig · 4h ago
You will always trust domain experts at some junction; you can't build a company otherwise. The question is: Can LLMs provide that domain expertise? I would argue, yes, clearly, given the development of the past 2 years, but obviously not on a straight line.
I just finished writing a Kafka consumer to migrate data with heavy AI help. This was basically best case a scenario for AI. It’s throw away greenfield code in a language I know pretty well (go) but haven’t used daily in a decade.
For complicated reasons the whole database is coming through on 1 topic, so I’m doing some fairly complicated parallelization to squeeze out enough performance.
I’d say overall the AI was close to a 2x speed up. It mostly saved me time when I forgot the go syntax for something vs looking it up.
However, there were at least 4 subtle bugs (and many more unsubtle ones) that I think anyone who wasn’t very familiar with Kafka or multithreaded programming would have pushed to prod. As it is, they took me a while to uncover.
On larger longer lived codebases, I’ve seen something closer to a 10-20% improvement.
All of this is using the latest models.
Overall this is at best the kind of productivity boost we got from moving to memory managed languages. Definitely not something that is going to replace engineers with PMs vibe coding anytime soon (based on rate of change I’ve seen over the last 3 years).
My real worry is that this is going to make mid level technical tornadoes, who in my experience are the most damaging kind of programmer, 10x as productive because they won’t know how to spot or care about stopping subtle bugs.
I don’t see how senior and staff engineers are going to be able to keep up with the inevitable flood of reviews.
I also worry about the junior to senior pipeline in a world where it’s even easier to get something
up that mostly works—we already have this problem today with copy paste programmers, but we’ve just make copy paste programming even easier.
I think the market will eventually sort this all out, but I worry that it could take decades.
awfulneutral · 3h ago
Yeah, the AI-generated bugs are really insidious. I also pushed a couple subtle bugs in some multi-threaded code I had AI write, because I didn't think it through enough. Reviews and tests don't replace the level of scrutiny something gets when it's hand-written. For now, you have to be really careful with what you let AI write, and make sure any bugs will be low impact since there will probably be more than usual.
skeeter2020 · 3h ago
> I’ve seen something closer to a 10-20% improvement.
The seems to match my experience in "important" work too; a real increase but not essentially changing the essence of software development. Brook's "No Silver Bullet" strikes again...
murukesh_s · 2h ago
What about generating testable code? I mean you mentioned detecting subtle bugs in generated code - I too have seen similar - but what if that was found via generated test cases than found by a human reviewers? Of course the test code could have bugs, but I can see a scenario in the future where all we do is review the tests output instead of scrutinising the generated code...
LgWoodenBadger · 3h ago
Complicated parallelization? That’s what partitions and consumers/consumer-groups are for!
ape4 · 5h ago
The article says there aren't too many useless comments but the code has:
// Get the Origin header from the request
const origin = request.headers.get('Origin');
spenczar5 · 1h ago
Of course, these are awful for a human. But I wonder if they're actually helpful for the LLM when it's reading code. It means each line of behavior is written in two ways: human language and code. Maybe that rosetta stone helps it confidently proceed in understanding, at the cost of tokens.
All speculation, but I'd be curious to see it evaluated - does the LLM do better edits on egregiously commented code?
slashdev · 3h ago
Those kinds of comments are a big LLM giveaway, I always remove them, not to hide that an LLM was used, but because they add nothing.
lucas_codes · 2h ago
Plus you just know in a few months they are going to be stale and reference code that has changed. I have even seen this happen with colleagues using llms between commits on a single pr.
kissgyorgy · 2h ago
I also noticed Claude likes writing useless redundant comments like this A LOT.
aiono · 5h ago
I agree with the last paragraph about doing this yourself. Humans have tendency to take shortcuts while thinking. If you see something resembling what you expect for the end product you will be much less critical of it. The looks/aesthetics matter a lot on finding problems with in a piece of code you are reading. You can verify this by injecting bugs in your code changes and see if reviewers can find them.
On the other hand, when you have to write something yourself you drop down to slow and thinking state where you will pay attention to details a lot more. This means that you will catch bugs you wouldn't otherwise think of. That's why people recommend writing toy versions of the tools you are using because writing yourself teaches a lot better than just reading materials about it. This is related to know our cognition works.
kentonv · 1h ago
I agree that most code reviewers are pretty bad at spotting subtle bugs in code that looks good superficially.
I have a lot of experience reviewing code -- more than I ever really wanted. It has... turned me cynical and bitter, to the point that I never believe anything is right, no matter who wrote it or how nice it looks, because I've seen so many ways things can go wrong. So I tend to review every line, simulate it in my head, and catch things. I kind of hate it, because it takes so long for me to be comfortable approving anything, and my reviewees hate it too, so they tend to avoid sending things to me.
I think I agree that if I'd written the code by hand, it would be less likely to have bugs. Maybe. I'm not sure, because I've been known to author some pretty dumb bugs of my own. But yes, total Kenton brain cycles spent on each line would be higher, certainly.
On the other hand, though, I probably would not have been the one to write this library. I just have too much on my plate (including all those reviews). So it probably would have been passed off to a more junior engineer, and I would have reviewed their work. Would I have been more or less critical? Hard to say.
But one thing I definitely disagree with is the idea that humans would have produced bug-free code. I've seen way too many bugs in my time to take that seriously. Hate to say it but most of the bugs I saw Claude produce are mistakes I'd totally expect an average human engineer could make.
Aside, since I know some people are thinking it: At this time, I do not believe LLM use will "replace" any human engineers at Cloudflare. Our hiring of humans is not determined by how much stuff we have to do, because we basically have infinite stuff we want to do. The limiting factor is what we have budget for. If each human becomes more productive due to LLM use, and this leads to faster revenue growth, this likely allows us to hire more people, not fewer. (Disclaimer: As with all of my comments, this is my own opinion / observation, not an official company position.)
HocusLocus · 6h ago
I suggest they freeze a branch of it, then spawn some AIs to introduce and attempt to hide vulnerabilities, and another to spot and fix them. Every commit is a move, and try to model the human evolution of chess.
throwawaybob420 · 1h ago
I’ve never seen such “walking off the cliff” behavior than from people who whole heartedly champion LLMs and the like.
Leaning on and heavily relying on a black box that hallucinates gibberish to “learn”, perform your work, and review your work.
All the while it literally consumes ungodly amounts of energy and is used as pretext to get rid of people.
Really cool stuff! I’m sure it’s 10x’ing your life!
kentonv · 2h ago
Hi, I'm the author of the library. (Or at least, the author of the prompts that generated it.)
> I’m also an expert in OAuth
I'll admin I think Neil is significantly more of an expert than me, so I'm delighted he took a pass at reviewing the code! :)
I'd like to respond to a couple of the points though.
> The first thing that stuck out for me was what I like to call “YOLO CORS”, and is not that unusual to see: setting CORS headers that effectively disable the same origin policy almost entirely for all origins:
I am aware that "YOLO CORS" is a common novice mistake, but that is not what is happening here. These CORS settings were carefully considered.
We disable the CORS headers specifically for the OAuth API (token exchange, client registration) endpoints and for the API endpoints that are protected by OAuth bearer tokens.
This is valid because none of these endpoints are authorized by browser credentials (e.g. cookies). The purpose of CORS is to make sure that a malicious website cannot exercise your credentials against some other website by sending a request to it and expecting the browser to add your cookies to that request. These endpoints, however, do not use browser credentials for authentication.
Or to put in another way, the endpoints which have open CORS headers are either control endpoints which are intentionally open to the world, or they are API endpoints which are protected by an OAuth bearer token. Bearer tokens must be added explicitly by the client; the browser never adds one automatically. So, in order to receive a bearer token, the client must have been explicitly authorized by the user to access the service. CORS isn't protecting anything in this case; it's just getting in the way.
(Another purpose of CORS is to protect confidentiality of resources which are not available on the public internet. For example, you might have web servers on your local network which lack any authorization, or you might unwisely use a server which authorizes you based on IP address. Again, this is not a concern here since the endpoints in question don't provide anything interesting unless the user has explicitly authorized the client.)
Aside: Long ago I was actually involved in an argument with the CORS spec authors, arguing that the whole spec should be thrown away and replaced with something that explicitly recognizes bearer tokens as the right way to do any cross-origin communications. It is almost never safe to open CORS on endpoints that use browser credentials for auth, but it is almost always safe to open it on endpoints that use bearer tokens. If we'd just recognized and embraced that all along I think it would have saved a lot of confusion and frustration. Oh well.
> A more serious bug is that the code that generates token IDs is not sound: it generates biased output.
I disagree that this is a "serious" bug. The tokens clearly have enough entropy in them to be secure (and the author admits this). Yes, they could pack more entry per byte. I noticed this when reviewing the code, but at the time decided:
1. It's secure as-is, just not maximally efficient.
2. We can change the algorithm freely in the future. There is not backwards-compatibility concern.
So, I punted.
Though if I'd known this code was going to get 100x more review than anything I've ever written before, I probably would have fixed it... :)
> according to the commit history, there were 21 commits directly to main on the first day from one developer, no sign of any code review at all
Please note that the timestamps at the beginning of the commit history as shown on GitHub are misleading because of a history rewrite that I performed later on to remove some files that didn't really belong in the repo. GitHub appears to show the date of the rebase whereas `git log` shows the date of actual authorship (where these commits are spread over several days starting Feb 27).
> I had a brief look at the encryption implementation for the token store. I mostly like the design! It’s quite smart.
Thank you! I'm quite proud of this design. (Of course, the AI would never have come up with it itself, but it was pretty decent and filling in the details based on my explicit instructions.)
lapcat · 6m ago
Does Cloudflare intend to put this library into production?
epolanski · 4h ago
Part of me this "written by LLM" has been a way to get attention on the codebase and plenty of free reviews by domain expert skeptics, among the other goals (pushing AI efficiency to investors, experimenting, etc).
kentonv · 2h ago
Free reviews by domain experts are great.
I didn't think of that, though. I didn't have an agenda here, I just put the note in the readme about it being LLM-generated only because I thought it was interesting.
max2he · 2h ago
Interesting to have people submit their promts to git. Do you think it'll be generally an accepted thing or was this just a showcase of how they promt?
kentonv · 1h ago
I included the prompts because I personally found it extremely illuminating to see what the LLM was able to produce based on those prompts, and I figured other people would be interested to. Seems I was right.
But to be clear, I had no idea how to write good prompts. I basically just wrote like I would write to a human. That seemed to work.
dweekly · 4h ago
An approach I don't see discussed here is having different agents using different models critique architecture and test coverage and author tests to vet the other model's work, including reviewing commits. Certainly no replacement for human in the loop but it will catch a lot of goofy "you said to only check in when all the tests pass so I disabled testing because I couldn't figure out how to fix the tests".
djoldman · 6h ago
> At ForgeRock, we had hundreds of security bugs in our OAuth implementation, and that was despite having 100s of thousands of automated tests run on every commit, threat modelling, top-flight SAST/DAST, and extremely careful security review by experts.
Wow. Anecdotally it's my understanding that OAuth is ... tricky ... but wow.
Some would say it's a dumpster fire. I've never read the spec or implemented it.
stuaxo · 5h ago
The times I've been involved with implementations it's been really horrible.
jofzar · 4h ago
Oauth is so annoying, there is so much niche to it.
bandoti · 5h ago
Honestly, new code always has bugs though. That’s pretty much a guarantee—especially if it’s somewhat complex.
That’s why companies go for things that are “battle tested” like vibe coding. ;)
Joke aside—I like how Anthropic is using their own product in a pragmatic fashion. I’m wondering if they’ll use it for their MCP authentication API.
jajko · 5h ago
Hundreds of thousands of tests? That sounds like quantity > quality or outright llm-generated ones, who even maintains them?
nmadden · 4h ago
This was before LLMs. It was a combination of unit and end-to-end tests and tests written to comprehensively test every combination of parameters (eg test this security property holds for every single JWT algorithm we support etc). Also bear in mind that the product did a lot more than just OAuth.
roxolotl · 4h ago
> Many of these same mistakes can be found in popular Stack Overflow answers, which is probably where Claude learnt them from too.
This is what keeps me up at night. Not that security holes will inevitably be introduced, or that the models will make mistakes, but that the knowledge and information we have as a society is basically going to get frozen in time to what was popular on the internet before LLMs.
tuxone · 4h ago
> This is what keeps me up at night.
Same here. For some of the services I pay, say the e-mail provider, the fact that they openly deny using LLMs for coding would be a plus for me.
ChrisArchitect · 25m ago
Related:
I read all of Cloudflare's Claude-generated commits
LLMs are like power tools. You still need to understand the architecture, do the right measurements, and apply the right screw to the right spot.
kcatskcolbdi · 6h ago
Really interesting breakdown. What jumped out to me wasn’t just the bugs (CORS wide open, incorrect Basic auth, weak token randomness), but how much the human devs seemed to lean on Claude’s output even when it was clearly offbase. That “implicit grant for public clients” bit is wild; it’s deprecated in OAuth 2.1, and Claude just tossed it in like it was fine, and then it stuck.
kentonv · 2h ago
I put in the implicit grant because someone requested it. I had it flagged off by default because it's deprecated.
keybored · 4h ago
Oh another one,[1] cautious somewhat-skeptic edition.
"...A more serious bug is that the code that generates token IDs is not sound: it generates biased output. This is a classic bug when people naively try to generate random strings, and the LLM spat it out in the very first commit as far as I can see. I don’t think it’s exploitable: it reduces the entropy of the tokens, but not far enough to be brute-forceable. But it somewhat gives the lie to the idea that experienced security professionals reviewed every line of AI-generated code...."
In the Github repo Cloudflare says:
"...Claude's output was thoroughly reviewed by Cloudflare engineers with careful attention paid to security and compliance with standards..."
Admittedly I have done some cryptographic string generation based on different alphabet sizes and characteristics a few years ago, which is pretty specifically relevant, and I’m competent at cryptographic and security concerns for a layman, but I certainly hope security reviewers will be more skilled at these things than me.
I’m very confident I would have noticed this bias in a first pass of reviewing the code. The very first thing you do in a security review is look at where you use `crypto`, what its inputs are, and what you do with its outputs, very carefully. On seeing that %, I would have checked characters.length and found it to be 62, not a factor of 256; so you need to mess around with base conversion, or change the alphabet, or some other such trick.
This bothers me and makes me lose confidence in the review performed.
thegeomaster · 6h ago
But... is it a real problem? As the author says, the entropy reduction is tiny.
yusina · 5h ago
It shows carelessness or incompetence or a combination thereof which extend to the entire code base.
OutOfHere · 3h ago
This is why I have multiple LLMS review and critique my specifications document, iteratively and repeatedly so, before I have my preferred LLM code it for me. I address all important points of feedback in the specifications document. To do this iteratively and repeatedly until there are no interesting points is crucial. This really fixes 80% of the expertise issues.
Moreover, after developing the code, I have multiple LLMs critique the code, file by file, or even method by method.
When I say multiple, I mean a non-reasoning one, a reasoning large one, and a next-gen reasoning small one, preferably by multiple vendors.
user9999999999 · 1h ago
why on earth would you code oauth in ai at this stage?
sdan · 6h ago
> Another hint that this is not written by people familiar with OAuth is that they have implemented Basic auth support incorrectly.
so tldr most of the issue the author has is against the person who made the library is the design not the implementation?
CuriouslyC · 7h ago
Mostly a good writeup, but I think there's some serious shifting the goalposts of what "vibe coded" means in a disingenuous way towards the end:
'Yes, this does come across as a bit “vibe-coded”, despite what the README says, but so does a lot of code I see written by humans. LLM or not, we have to give a shit.'
If what most people do is "vibe coding" in general, the current definition of vibe coding is essentially meaningless. Instead, the author is making the distinction between "interim workable" and "stainless/battle tested" which is another dimension of code entirely. To describe that as vibe coding causes me to view the author's intent with suspicion.
simonw · 5h ago
How do you define vibe coding?
techpression · 6h ago
I find ”vibe coding” to be one of the, if not the, concepts in this business to lose its meaning the fastest. Similar to how everything all of a sudden was ”cloud” now everything is ”vibe coded”, even though reading the original tweet really narrows it down thoroughly.
dimitri-vs · 5h ago
IMO it's pretty clear what vibe coding is: you don't look at the code, only the results. If you're making judgement on the code, it's not vibe coding.
keybored · 4h ago
Viral marketing campaign term losing its meaning makes sense.
croes · 6h ago
Isn’t vibe coding just C&P from AI instead of Stack Overflow?
I read it as: done by AI but not checked by humans.
ranguna · 6h ago
Yep I see it like that as well, code with 0 or very close to 0 interactions from humans. Anyone who wants to change that meaning is not serious.
SiempreViernes · 6h ago
A very good piece that clearly illustrates one of the dangers with LLS's: responsibility for code quality is blindly offloaded on the automatic system
> There are some tests, and they are OK, but they are woefully inadequate for what I would expect of a critical auth service. Testing every MUST and MUST NOT in the spec is a bare minimum, not to mention as many abuse cases as you can think of, but none of that is here from what I can see: just basic functionality tests.
and
> There are some odd choices in the code, and things that lead me to believe that the people involved are not actually familiar with the OAuth specs at all. For example, this commit adds support for public clients, but does so by implementing the deprecated “implicit” grant (removed in OAuth 2.1).
As Madden concludes "LLM or not, we have to give a shit."
JimDabell · 6h ago
> A very good piece that clearly illustrates one of the dangers with LLS's: responsibility for code quality is blindly offloaded on the automatic system
It does not illustrate that at all.
> Claude's output was thoroughly reviewed by Cloudflare engineers with careful attention paid to security and compliance with standards.
> To emphasize, *this is not "vibe coded"*. Every line was thoroughly reviewed and cross-referenced with relevant RFCs, by security experts with previous experience with those RFCs.
The humans who worked on it very, very clearly took responsibility for code quality. That they didn’t get it 100% right does not mean that they “blindly offloaded responsibility”.
Perhaps you can level that accusation at other people doing different things, but Cloudflare explicitly placed the responsibility for this on the humans.
No comments yet
jstummbillig · 5h ago
Note that this has very little to do with AI assisted coding; the authors of the library explicitly approved/vetted the code. So this comes down to different coders having different thoughts about what constitutes good and bad code, with some flaunting of credentials to support POVs, and nothing about that is particularly new.
add-sub-mul-div · 4h ago
The whole point of this is that people will generally put the least effort into work as they think they can get away with, and LLMs will accelerate that force. This is the future of how code will be "vetted".
It's not important whose responsbility led to mistakes, it's important to understand we're creating a responsbility gap.
For me this is the key takeaway. You gain proper efficiency using LLMs when you are a competent reviewer, and for lack of a better word, leader. If you don't know the subject matter as well as the LLM, you better be doing something non-critical, or have the time to not trust it and verify everything.
The maximalists and skeptics both are confusing the debate by setting up this straw man that people will be delegating to LLMs blindly.
The idea that someone clueless about OAuth should develop an OAuth lib with LLM support without learning a lot about the topic is... Just wrong. Don't do that.
But if you're willing to learn, this is rocket fuel.
It was extremely frustrating.
https://en.wikipedia.org/wiki/Low-background_steel
Well it turns out you can manage just fine.
You shouldn't blindly trust anything. Not what you read, not what people say.
Using LLMs effectively is a skill too, and that does involve deciding when and how to verify information.
Google's case (and Meta and spam calls and others) is at least in part an incentives problem. Google hasn't been about delivering excellent search to users for a very long time. They're an ad company and their search engine is a tool to better deliver ads. Once they had an effective monopoly, they just had to stay good enough not to lose it.
I've been using Kagi for a few years now and while SEO spam and AI garbage is still an issue, it is far less of one than with Google or Bing. My conclusion is these problems are at least somewhat addressable if doing so is what gets the business paid.
But I think a real long term solution will have to involved a federated trust model. It won't be viable to index everything dumped on the web; there will need to be a component prioritizing trust in the author or publisher. If that follows the same patterns as email (ex: owned by Google and Microsoft), then we're really screwed.
You missed the full context: you would never be able to trust a bunch of amateur randos self-policing their content. Turns out it's not perfect but better than a very small set of professionals; usually there's enough expertise out there, it's just widely distributed. The challenge this time is 1. the scale, 2. the rate of growth, 3. the decline in expertise.
>> Using LLMs effectively is a skill too, and that does involve deciding when and how to verify information.
How do you verify when ALL the sources are share the same AI-generated root, and ALL of the independent (i.e. human) experts have aged-out and no longer exist?
Why would that happen? There's demand for high quality, trustworthy information and that's not going away.
When asking an LLM coding questions, for example, you can ask for sources and it'll point you to documentation. It won't always be the correct link, but you can prod it more and usually get it, or fall back to searching the docs the old fashioned way.
Before AI generated results, the first page of Google was SEO-optimised crap blogs. The internet has been hard to search for a while.
No comments yet
(One thing that might be useful is use the LLM as a search engine to find the relevant RFCs since sometimes it’s hard to find all of the applicable ones if you don’t know the names of them already.)
I really can’t stress this enough: read the RFCs from end to end. Then read through the code of some reference implementations. Draw a sequence diagram. Don’t have the LLM generate one for you, the point is to internalize the design you’re trying to implement against.
By this time you should start spotting bugs or discrepancies between the specs and implementations in the wild. That’s a good sign. It means you’re learning
feels like there's a logical flaw here, when the issue is that LLMs are presenting the wrong information or missing it all together. The person trying to learn from it will experience Donald Rumsfield's "unknown unknowns".
I would not be surprised if we experience an even more dramatic "Cobol Moment" a generation from now, but unlike that one thankfully I won't be around to experience it.
You don't get knowledge by ONLY talking to LLMs, but they're a great tool.
With learning, aren’t you exposed to the same risks? Such that if there was a typical blind spot for the LLM, it would show up in the learning assistance and in the development assistance, thus canceling out (i.e unknown unknowns)?
Or am I thinking about it wrongly?
One big technique it sounds like the authors of the OAuth library missed is that LLMs are very good at generating tests. A good development process for today’s coding agents is to 1) prompt with or create a PRD, 2) break this down into relatively simple tasks, 3) build a plan for how to tackle each task, with listed out conditions that should be tested, 3) write the tests, so that things are broken, TDD style and finally 4) write the implementation. The LLM can do all of this, but you can’t one-shot it these days, you have to be a human in the loop at every step, correcting when things go off track. It’s faster, but it’s not a 10x speed up like you might imagine if you think the LLM is just asynchronously taking a PRD some PM wrote and building it all. We still have jobs for a reason.
But TDD is not the way I think. I've never been able to work that way (LLM-assisted or otherwise). I find it very hard to write tests for software that isn't implemented yet, because I always find that a lot of the details about how it should work are discovered as part of the implementation process. This both means that any API I come up with before implementing is likely to change, and also it's not clear exactly what details need to be tested until I've fully explored how the thing works.
This is just me, other people may approach things totally differently and I can certainly understand how TDD works well for some people.
How do you determine if the LLM accurately reflects what the high-quality source contains, if you haven't read the source? When learning from humans, we put trust on them to teach us based on a web-of-trust. How do you determine the level of trust with an LLM?
But this is only part of the story. When learning from another human, you'll also actively try and devise whether they're trustworthy based on general linguistic markers, and will try to find and poke holes in what they're saying so that you can question intelligently.
This is not much different from what you'd do with an LLM, which is why it's such a problem that they're more convincing than correct pretty often. But it's not an insurmountable issue. The other issue is that their trustworthiness will wary in a different way than a human's, so you need experience to know when they're possibly just making things up. But just based on feel, I think this experience is definitely possible to gain.
Bonus: the high quality source is going to be mostly AI written anyway
I’m still on the lookout for a great model for this.
This does mean that there's a reliance on me being able to determine what are key facts and when I should be asking for a source though. I have not experienced any significant drawbacks when compared to a classic research workflow though, so in my view it's a net speed boost.
However, this does mean that a huge variety of things remain out of reach for me to accomplish, even with LLM "assistance". So there's a decent chance even the speed boost is only perceptual. If nothing else, it does take a significant amount of drudgery out of it all though.
I don't think that's how things work. In learning tasks, LLMs are sparring partners. You present them with scenarios, and they output a response. Sometimes they hallucinate completely, but they can also update their context to reflect new information. Their output matches what you input.
I’ve grown the most when I start with things I sort of know and I work to expand my understanding.
You are getting a stylised view of a topic from an entity who lacks the deep understanding needed to be able to fully distill the information. But it is enough to gain enough knowledge for you to feel confident which is still valuable but also dangerous.
And I assure you that many, many people are delegating to LLMs blindly e.g. it's a huge problem in the UK legal system right now because of all the invented case law references.
Isn't this how every child learns?
Unless his father happens to be king of Macedonia, of course.
making mistakes is how we learn, and if they are never pointed out...
Sure, having access to legit experts who can tutor you privately on a range of topics would be better, but that's not realistic.
What I find is that if I need to explore some new domain within a field I'm broadly familiar with, just thinking through what the LLM is saying is sufficient for verification, since I can look for internal consistency and check against things I know already.
When exploring a new topic, often times my questions are superficial enough for me to be confident that the answers are very common in the training data.
When exploring a new topic that's also somewhat niche or goes into a lot of detail, I use the LLM first to get a broad overview and then drill down by asking for specific sources and using the LLM as an assistant to consume authoritative material.
You know that it's possible to ask models for dissenting opinions, right? Nothing's stopping you.
> and if they are never pointed out...
They do point out mistakes though?
It's important to remember that these models depend not only on ML breakthroughs but also on the breadth and freshness of the data used to train them.
That said, the "next-door" model could very well incorporate lessons from the recent Cloudflare OAuth Library issues, thanks to the ongoing discussions and community problem-solving efforts.
We’ve gone from skeptics saying LLMs can’t code, to they can’t code well, to they can’t produce human-level code, to they are riddled with hallucinations, to now “but they can’t one-shot code a library without any bugs or flaws” and “but they can only one-shot code, they can’t edit well” even tho recents coding utilities have been proving that wrong as well. And still they say they are useless.
Some people just don’t hear themselves or see how AI is constantly moving their bar.
LLMs will tell you 1 or 2 lies for each 20 facts. Its a hard way to learn. They cant even get their urls right...
That was my experience when growing up with school also, except you got punished one way or another for speaking up/trying to correct the teacher. If I speak up with the LLM they either explain why what they said is true, or corrects themselves, 0 emotions involved.
> They cant even get their urls right...
Famously never happens with humans.
If you are in class, and you incorrectly argue, there is a mistake in an explanation of Derivatives or Physics, but you are the one in error, your Teacher hopefully, will not say: "Oh, I am sorry you are absolutely correct. Thank you for your advice.."
- Confident synthesis of incompatible sources: LLM: “Einstein won the 1921 Nobel Prize for his theory of relativity, which he presented at the 1915 Solvay Conference.”
Or
- Fabricated but plausible citations: LLM: “According to Smith et al., 2022, Nature Neuroscience, dolphins recognise themselves in mirrors.” There is no such paper...model invents both authors and journal reference
And this is the danger of coding with LLMs....
What matters is how X reacts when you point out it wasn't correct, at least in my opinion, and was the difference I was trying to highlight.
An LLM, by contrast, will invent a flawless looking but nonexistent citation. Even a below average teacher doesn’t churn out fresh fabrications every tenth sentence.
Because a teacher usually cites recognizable material, you can check the textbook and recover quickly. With an LLM you first have to discover the source never existed. That verification cost is higher, the more complex task you are trying to achieve.
A LLM will give you a perfect paragraph about the AWS Database Migration service, the list of supported databases, and then include in there a data flow like on-prem to on-prem data that is not supported...Relying on an LLM is like flying with a friendly copilot but who has multiple personality disorder. You dont know which day he will forget to take his meds :-)
Stressful and mentally exhausting in a different kind of way....
Me: “explain why radioactive half-life changes with temperature”
ChatGPT 4o: “ Short answer: It doesn’t—at least not significantly. Radioactive Half-Life is (Almost Always) Temperature-Independent”
…and then it goes on to give a few edge cases where there’s a tiny effect.
I cannot see us living in a world of ignorance where there are literally zero engineers and no one on the planet understands what's been generated. Weirdly we could end up in a place where engineering skills are niche and extremely lucrative.
No comments yet
Fast forward 30 years and modern civilisation is entirely dependent on our AI’s.
Will deep insight and innovation from a human perspective perhaps come to a stop?
Tools will only amplify human skills. Sure, not everyone will choose to use tools for anything meaningful, but those people are driving human insight and innovation today anyway.
What is new is that you'll need the wisdom to figure out when the tool can do the whole job, and where you need to intervene and supervise it closely.
So humans won't be doing any less thinking, rather they'll be putting their thinking to work in better ways.
Experts will become those who use llm to learn and not to write code for them or solve tasks for them so they can build that skill.
For example, at one point a human + computer would have been the strongest combo in chess, now you'd be insane to allow a human to critic a chess bot because they're so unlikely to add value, and statistically a human in the loop would be far more likely to introduce error. Similar things can be said in fields like machine vision, etc.
Software is about to become much higher quality and be written at much, much lower cost.
I don’t think we are anywhere close to doing that.
A good analogy might be how machines gradually replaced textile workers in the 19th century. Were the machines better? Or was there a was to quantitatively measure the quality of their output? No. But at the end of the day companies which embraced the technology were more productive than those who didn't, and the quality didn't decrease enough (if it did at all) that customers would no longer do business with them – so these companies won out.
The same will naturally happen in software over the next few years. You'd be an moron to hire a human expert for $200,000 to critic a cybersecurity optimised model which costs maybe a 100th of the cost of employing a human... And this would likely be true even if we assume the human will catch the odd thing the model wouldn't because there's no such thing as perfect security – it's always a trade off between cost and acceptable risk.
Bookmark this and come back in a few years. I made similar predictions when ChatGPT first came out that within a few years agents would be picking up tickets and raising PRs. Everyone said LLMs were just stochastic parrots and this would not happen, well now it has and increasingly companies are writing more and more code with AI. At my company it's a little over 50% at the mo, but this is increasing every month.
In addition to the ability to review output effectively, I find the more closely I’m able to describe what I want in the way another expert in that domain would, the better the LLM output. Which isn’t really that surprising for a statistical text generation engine.
For example, I'm horrible at math, always been, so writing math-heavy code is difficult for me, I'll confess to not understanding math well enough. If I'm coding with an LLM and making it write math-heavy code, I write a bunch of unit tests to describe what I expect the function to return, write a short description and give it to the LLM. Once the function is written, run the tests and if it passes, great.
I might not 100% understand what the function does internally, and it's not used for any life-preserving stuff either (typically end up having to deal with math for games), but I do understand what it outputs, and what I need to input, and in many cases that's good enough. Working in a company/with people smarter than you tends to make you end up in this situation anyways, LLMs or not.
Though if in the future I end up needing to change the math-heavy stuff in the function, I'm kind of locked into using LLMs for understanding and changing it, which obviously feels less good. But the alternative is not doing it at all, so another tradeoff I suppose.
I still wouldn't use this approach for essential/"important" stuff, but more like utility functions.
People don't learn how a car works before buying one, they just take it to a mechanic when it breaks. Most people don't know how to build a house, they have someone else build it and assume it was done well.
I fully expect people to similarly have LLMs do what the person doesn't know how and assume the machine knew what to do.
Because LLMs are not competent professionals to whom you might outsource tasks in your life. LLMs are statistical engines that make up answers all the time, even when the LLM “knows” the correct answer (i.e., has the correct answer hidden away in its weights.)
I don’t know about you, but I’m able to validate something is true much more quickly and efficiently if it is a subject I know well.
For instance, I used Next.js to build a simple login page with Google auth. It worked great, even though I only had basic knowledge of Node.js and a bit of React.
Then I tried adding a database layer using Prisma to persist users. That's where things broke. The integration didn't work, seemingly due to recent versions in Prisma or subtle breaking updates. I found similar issues discussed on GitHub and Reddit, but solving them required shifting into full manual debugging mode.
My takeaway: even with improved models, fast-moving frameworks and toolchains can break workflows in ways that LLMs/ML (at least today) can't reason through or fix reliably. It's not always about missing domain knowledge, it's that the moving parts aren't in sync with the model yet.
I think it's caused by you not having a strong enough system prompt. Once you've built up a slightly reusable system prompt for coding or for infra work, where you bit by bit build it up while using a specific model (since different models respond differently to prompts), you end up getting better and better responses.
So if you notice it putting plaintext credentials in the code, add to the system prompt to not do that. With LLMs you really get what you ask for, and if you miss to specify anything, the LLM will do whatever the probabilities tells it to, but you can steer this by being more specific.
Imagine you're talking to a very literal and pedantic engineer who argues a lot on HN and having to be very precise with your words, and you're like 80% of the way there :)
This is going to be a powerful feedback loop which you might call regression to the intellectual mean.
On any task, most training data is going to represent the middle (or beginning) of knowledge about a topic. Most k8s examples will skip best practices, most react apps will be from people just learning react, etc.
If you want the LLM to do best practices in every knowledge domain (assuming best practices can be consistently well defined), then you have to push it away from the mean of every knowledge domain simultaneously (or else work with specialized fine tuned models).
As you continue to add training data it will tend to regress toward the middle because that's where most people are on most topics.
For complicated reasons the whole database is coming through on 1 topic, so I’m doing some fairly complicated parallelization to squeeze out enough performance.
I’d say overall the AI was close to a 2x speed up. It mostly saved me time when I forgot the go syntax for something vs looking it up.
However, there were at least 4 subtle bugs (and many more unsubtle ones) that I think anyone who wasn’t very familiar with Kafka or multithreaded programming would have pushed to prod. As it is, they took me a while to uncover.
On larger longer lived codebases, I’ve seen something closer to a 10-20% improvement.
All of this is using the latest models.
Overall this is at best the kind of productivity boost we got from moving to memory managed languages. Definitely not something that is going to replace engineers with PMs vibe coding anytime soon (based on rate of change I’ve seen over the last 3 years).
My real worry is that this is going to make mid level technical tornadoes, who in my experience are the most damaging kind of programmer, 10x as productive because they won’t know how to spot or care about stopping subtle bugs.
I don’t see how senior and staff engineers are going to be able to keep up with the inevitable flood of reviews.
I also worry about the junior to senior pipeline in a world where it’s even easier to get something up that mostly works—we already have this problem today with copy paste programmers, but we’ve just make copy paste programming even easier.
I think the market will eventually sort this all out, but I worry that it could take decades.
The seems to match my experience in "important" work too; a real increase but not essentially changing the essence of software development. Brook's "No Silver Bullet" strikes again...
All speculation, but I'd be curious to see it evaluated - does the LLM do better edits on egregiously commented code?
On the other hand, when you have to write something yourself you drop down to slow and thinking state where you will pay attention to details a lot more. This means that you will catch bugs you wouldn't otherwise think of. That's why people recommend writing toy versions of the tools you are using because writing yourself teaches a lot better than just reading materials about it. This is related to know our cognition works.
I have a lot of experience reviewing code -- more than I ever really wanted. It has... turned me cynical and bitter, to the point that I never believe anything is right, no matter who wrote it or how nice it looks, because I've seen so many ways things can go wrong. So I tend to review every line, simulate it in my head, and catch things. I kind of hate it, because it takes so long for me to be comfortable approving anything, and my reviewees hate it too, so they tend to avoid sending things to me.
I think I agree that if I'd written the code by hand, it would be less likely to have bugs. Maybe. I'm not sure, because I've been known to author some pretty dumb bugs of my own. But yes, total Kenton brain cycles spent on each line would be higher, certainly.
On the other hand, though, I probably would not have been the one to write this library. I just have too much on my plate (including all those reviews). So it probably would have been passed off to a more junior engineer, and I would have reviewed their work. Would I have been more or less critical? Hard to say.
But one thing I definitely disagree with is the idea that humans would have produced bug-free code. I've seen way too many bugs in my time to take that seriously. Hate to say it but most of the bugs I saw Claude produce are mistakes I'd totally expect an average human engineer could make.
Aside, since I know some people are thinking it: At this time, I do not believe LLM use will "replace" any human engineers at Cloudflare. Our hiring of humans is not determined by how much stuff we have to do, because we basically have infinite stuff we want to do. The limiting factor is what we have budget for. If each human becomes more productive due to LLM use, and this leads to faster revenue growth, this likely allows us to hire more people, not fewer. (Disclaimer: As with all of my comments, this is my own opinion / observation, not an official company position.)
Leaning on and heavily relying on a black box that hallucinates gibberish to “learn”, perform your work, and review your work.
All the while it literally consumes ungodly amounts of energy and is used as pretext to get rid of people.
Really cool stuff! I’m sure it’s 10x’ing your life!
> I’m also an expert in OAuth
I'll admin I think Neil is significantly more of an expert than me, so I'm delighted he took a pass at reviewing the code! :)
I'd like to respond to a couple of the points though.
> The first thing that stuck out for me was what I like to call “YOLO CORS”, and is not that unusual to see: setting CORS headers that effectively disable the same origin policy almost entirely for all origins:
I am aware that "YOLO CORS" is a common novice mistake, but that is not what is happening here. These CORS settings were carefully considered.
We disable the CORS headers specifically for the OAuth API (token exchange, client registration) endpoints and for the API endpoints that are protected by OAuth bearer tokens.
This is valid because none of these endpoints are authorized by browser credentials (e.g. cookies). The purpose of CORS is to make sure that a malicious website cannot exercise your credentials against some other website by sending a request to it and expecting the browser to add your cookies to that request. These endpoints, however, do not use browser credentials for authentication.
Or to put in another way, the endpoints which have open CORS headers are either control endpoints which are intentionally open to the world, or they are API endpoints which are protected by an OAuth bearer token. Bearer tokens must be added explicitly by the client; the browser never adds one automatically. So, in order to receive a bearer token, the client must have been explicitly authorized by the user to access the service. CORS isn't protecting anything in this case; it's just getting in the way.
(Another purpose of CORS is to protect confidentiality of resources which are not available on the public internet. For example, you might have web servers on your local network which lack any authorization, or you might unwisely use a server which authorizes you based on IP address. Again, this is not a concern here since the endpoints in question don't provide anything interesting unless the user has explicitly authorized the client.)
Aside: Long ago I was actually involved in an argument with the CORS spec authors, arguing that the whole spec should be thrown away and replaced with something that explicitly recognizes bearer tokens as the right way to do any cross-origin communications. It is almost never safe to open CORS on endpoints that use browser credentials for auth, but it is almost always safe to open it on endpoints that use bearer tokens. If we'd just recognized and embraced that all along I think it would have saved a lot of confusion and frustration. Oh well.
> A more serious bug is that the code that generates token IDs is not sound: it generates biased output.
I disagree that this is a "serious" bug. The tokens clearly have enough entropy in them to be secure (and the author admits this). Yes, they could pack more entry per byte. I noticed this when reviewing the code, but at the time decided:
1. It's secure as-is, just not maximally efficient. 2. We can change the algorithm freely in the future. There is not backwards-compatibility concern.
So, I punted.
Though if I'd known this code was going to get 100x more review than anything I've ever written before, I probably would have fixed it... :)
> according to the commit history, there were 21 commits directly to main on the first day from one developer, no sign of any code review at all
Please note that the timestamps at the beginning of the commit history as shown on GitHub are misleading because of a history rewrite that I performed later on to remove some files that didn't really belong in the repo. GitHub appears to show the date of the rebase whereas `git log` shows the date of actual authorship (where these commits are spread over several days starting Feb 27).
> I had a brief look at the encryption implementation for the token store. I mostly like the design! It’s quite smart.
Thank you! I'm quite proud of this design. (Of course, the AI would never have come up with it itself, but it was pretty decent and filling in the details based on my explicit instructions.)
I didn't think of that, though. I didn't have an agenda here, I just put the note in the readme about it being LLM-generated only because I thought it was interesting.
But to be clear, I had no idea how to write good prompts. I basically just wrote like I would write to a human. That seemed to work.
Wow. Anecdotally it's my understanding that OAuth is ... tricky ... but wow.
Some would say it's a dumpster fire. I've never read the spec or implemented it.
That’s why companies go for things that are “battle tested” like vibe coding. ;)
Joke aside—I like how Anthropic is using their own product in a pragmatic fashion. I’m wondering if they’ll use it for their MCP authentication API.
This is what keeps me up at night. Not that security holes will inevitably be introduced, or that the models will make mistakes, but that the knowledge and information we have as a society is basically going to get frozen in time to what was popular on the internet before LLMs.
Same here. For some of the services I pay, say the e-mail provider, the fact that they openly deny using LLMs for coding would be a plus for me.
I read all of Cloudflare's Claude-generated commits
https://news.ycombinator.com/item?id=44205697
[1] https://news.ycombinator.com/item?id=44205697
In the Github repo Cloudflare says:
"...Claude's output was thoroughly reviewed by Cloudflare engineers with careful attention paid to security and compliance with standards..."
My conclusion is that as a development team, they learned little since 2017: https://news.ycombinator.com/item?id=13718752
I’m very confident I would have noticed this bias in a first pass of reviewing the code. The very first thing you do in a security review is look at where you use `crypto`, what its inputs are, and what you do with its outputs, very carefully. On seeing that %, I would have checked characters.length and found it to be 62, not a factor of 256; so you need to mess around with base conversion, or change the alphabet, or some other such trick.
This bothers me and makes me lose confidence in the review performed.
Moreover, after developing the code, I have multiple LLMs critique the code, file by file, or even method by method.
When I say multiple, I mean a non-reasoning one, a reasoning large one, and a next-gen reasoning small one, preferably by multiple vendors.
so tldr most of the issue the author has is against the person who made the library is the design not the implementation?
'Yes, this does come across as a bit “vibe-coded”, despite what the README says, but so does a lot of code I see written by humans. LLM or not, we have to give a shit.'
If what most people do is "vibe coding" in general, the current definition of vibe coding is essentially meaningless. Instead, the author is making the distinction between "interim workable" and "stainless/battle tested" which is another dimension of code entirely. To describe that as vibe coding causes me to view the author's intent with suspicion.
I read it as: done by AI but not checked by humans.
> There are some tests, and they are OK, but they are woefully inadequate for what I would expect of a critical auth service. Testing every MUST and MUST NOT in the spec is a bare minimum, not to mention as many abuse cases as you can think of, but none of that is here from what I can see: just basic functionality tests.
and
> There are some odd choices in the code, and things that lead me to believe that the people involved are not actually familiar with the OAuth specs at all. For example, this commit adds support for public clients, but does so by implementing the deprecated “implicit” grant (removed in OAuth 2.1).
As Madden concludes "LLM or not, we have to give a shit."
It does not illustrate that at all.
> Claude's output was thoroughly reviewed by Cloudflare engineers with careful attention paid to security and compliance with standards.
> To emphasize, *this is not "vibe coded"*. Every line was thoroughly reviewed and cross-referenced with relevant RFCs, by security experts with previous experience with those RFCs.
— https://github.com/cloudflare/workers-oauth-provider
The humans who worked on it very, very clearly took responsibility for code quality. That they didn’t get it 100% right does not mean that they “blindly offloaded responsibility”.
Perhaps you can level that accusation at other people doing different things, but Cloudflare explicitly placed the responsibility for this on the humans.
No comments yet
It's not important whose responsbility led to mistakes, it's important to understand we're creating a responsbility gap.