When I first heard about the government's push to mass-filter contracts under a tight deadline with a small team, I figured they must be using AI. Hopefully not the “retail version” most of us tinker with.
But then I paused. I thought, surely with federal urgency and resources, they had access to advanced internal tooling or some supercharged AI product. This is sensitive work, impacting essential services. It would require deep domain knowledge, careful oversight, and rigorous review. I imagined a team of 10x engineers and policy analysts coordinating a finely tuned system.
After reading this piece, I realized my original suspicion was correct, but the reality was worse. No secret AI. Just a general-purpose model, with vague prompts, limited contract context, and questionable assumptions. That they didn’t anticipate hallucinated figures or misclassified services is frankly baffling.
Even with my limited AI experience, I would have known these risks. And that’s the real concern: not just flawed execution, but a fundamental misunderstanding of how and when these tools should be used.
It makes me wonder how many other organizations are deploying LLMs in the same way, unproven or experimental systems driven by deadlines, with little grasp of the underlying complexity or consequences.
mediumdeviation · 2h ago
The other part that's disconcerting is the person who did all this has decided to put his name on the public record in an interview with journalists. After reading through the article I had the feeling this would, in a few years, show up in software / AI ethics courses in university. Is this a form of atonement, whistleblower, or extremely misplaced pride?
belter · 2h ago
The person is the CEO of Gumroad and I sincerely hope it happens to Gumroad the same happening to Tesla. Life too short to give money and tolerate unethical self absorbed aholes
exe34 · 1h ago
You imagined all that? With Elon Musk in charge? I don't understand why so many people don't realise that his companies do well in spite of his meddling, not because.
But then I paused. I thought, surely with federal urgency and resources, they had access to advanced internal tooling or some supercharged AI product. This is sensitive work, impacting essential services. It would require deep domain knowledge, careful oversight, and rigorous review. I imagined a team of 10x engineers and policy analysts coordinating a finely tuned system.
After reading this piece, I realized my original suspicion was correct, but the reality was worse. No secret AI. Just a general-purpose model, with vague prompts, limited contract context, and questionable assumptions. That they didn’t anticipate hallucinated figures or misclassified services is frankly baffling.
Even with my limited AI experience, I would have known these risks. And that’s the real concern: not just flawed execution, but a fundamental misunderstanding of how and when these tools should be used.
It makes me wonder how many other organizations are deploying LLMs in the same way, unproven or experimental systems driven by deadlines, with little grasp of the underlying complexity or consequences.