IMO the step most of these projects fail on is UTILITY. I have yet to hear an AI project start with, "We need to do X and I think AI might help." Every single project meeting I've sat in started with an non-technical exec asking, "what can we use AI for?" They've got a solution and go looking for a problem so they can say they're using AI.
At my last company thankfully I was able to limit the losses to just under $450k which sounds like a lot, and it is, but it could have been about $5m/yr. I didn't sabotage anything, either. It was mostly through simply asking a lot of questions early, repeatedly suggesting we write plans down and make sure everyone on the project is up to day, and suggesting we test drive components before buying the whole car.
Spent a bunch of money, rolled out the pilot project to 70 folks, and began 30 day checkins. After 30 days we had a retention rate over 80%. By the 60 day mark, we were down to 40%. By day 90 we were at 22% and landed at 11% after 4 months and never went higher. By month 8 we cancelled the project and the 8 people still using the tool were ok letting it go, feeling it didn't help that much.
All because someone from the board said, "we need to use AI, anyone not using AI in a year will be out of business." The CEO asked what we should use it for, and the board member said, "I have no idea, that's your job. But we need AI somewhere. Look here, it can buy a plane ticket for me now!" and then wasted the next 40 minutes of the meeting talking about how amazing AI agents were. We're not a tech company, he was definitely NOT a tech person.
geoduck14 · 1h ago
>Every single project meeting I've sat in started with an non-technical exec asking, "what can we use AI for?"
To be fair, if the goal is to learn and discover how a new tool works and when to use it, then a legitimate strategy is to solve a bunch of problems with this new tool
burnte · 23m ago
I totally agree but AI is such a broad category it's like asking "what can we do with electricity?" Well, that depends, what problems are you looking to solve?
You always start with the problem that needs to be solved, not the solution. I'm concerned we'll have another bubble pop like the dotcom crash. A lot of companies are spending money like mad to have any kind of AI thing they can market about without regard to "Does it work right? Does is solve a problem?"
It took my former company a year and half a million dollars to learn that what we actually needed was a completely different tool that promised to do a lot less, but did what we needed it to do quite well. Mgmt got stars in their eyes with all the things the first tool might be able to do someday, and didn't pay attention to what it could do today.
We switched to the other tool at $70/month/user and saw a retention rate over 90% at 6 months. Exact same test group, exact same type of tool, but this tool was more focused and tangible. We went from spending $450k per year at with the first company to $59k/yr with the second company.
AI isn't magic, You can't skip all the project planning steps you'd normally do just because this is AI. Don't fast track it, don't ignore testing data or feedback, etc. It's hard to push back on excitement but it's important that we do. When our emotions carry us away that's when we make worse decisions.
bluefirebrand · 1h ago
Realistically that isn't the goal of almost any company these days
The goal is just "make all the money at all costs"
MrDarcy · 1h ago
It’s interesting. On the one hand you’re clearly right.
On the other hand though, with hindsight, imagine the board member saying the same thing about the PC, the internet, mobile computing, cloud computing…
I think both of you are right and it’s only a matter of timing. Time will tell.
burnte · 19m ago
Oh, I agree, I'm not saying "AI will always be garbage" but I WOULD say, "yep, mobile computing just isn't ready yet" in 1996. I also said every single day for the past 25 years that "cloud computing is a tool, not a solution." With how many gigs I've had moving workloads back on prem to reduce costs because someone else did a bad lift-and-shift, I think I'm right in tempering excitement. It's just hard to do. Everyone wants new and shiny.
jes5199 · 1h ago
probably more than 95% of early internet projects failed, it took like a decade before people really started to have intuitions about what it was good for.
It’s easy to imagine that we’re going to make the AI analog of paypal or amazon, but the vast majority of people flailed at doing anything useful with the web until patterns had been established
jcranmer · 59m ago
> imagine the board member saying the same thing about the PC, the internet, mobile computing, cloud computing…
For the deployment of the PC, I think the vast majority of companies start from an environment where there were already terminal systems or shared workstations available, and the adoption of PCs happened as they became powerful enough to accomplish the tasks that were already being done on the larger computers. It wasn't really a "thou must use PCs, though I know not how they are useful" mandate.
A better example of such a mandate can be found in education, and most of my personal experience with such thou-must-use-technology mandates has been that they've been similarly ineffective as the AI mandate.
OJFord · 1h ago
'What can we use x for' is a dumb approach for all of those xs today too.
burnte · 19m ago
Agreed, you start with a problem and explore possible solutions, not start with a solution and go find a problem.
greymalik · 1h ago
Because they’re known quantities. We know how to use them well. Right now we’re experimenting with LLMs in order to turn them into known quantities.
Aurornis · 1h ago
This article's structure and wording feel like it was barely tweaked after coming out of ChatGPT.
Anyway, it misses one of the current biggest problems with all things AI in the workplace: It has attracted every resume-builder and ladder-climber who don't know what they're doing. They just want an AI initiative for their resume.
As long as it's not too public (like the Taco Bell AI drive through disaster) it doesn't really matter if it's successful or not. They can spin it as a success and the next company they apply to won't be able to check.
The same thing happened a few years ago when every PM and rising manager was looking for a way to put blockchain into products. Before that it was "big data", and so on.
saaaaaam · 30m ago
It’s a piece of marketing content for an “AI Ops” company that will, presumably, sell you a solution to stop your AI pilot failing.
Hendrikto · 1h ago
100% agree.
> Before that it was "big data", and so on.
Cloud, micro services, NoSQL, serverless, …
We never learn :(
realz · 58m ago
Hey now, some of those things are actually useful.
ambicapter · 1h ago
> This article's structure and wording feel like it was barely tweaked after coming out of ChatGPT.
eugh, yeah that's definitely the case. I stopped reading once I got to the first repeated paragraph about data quality.
th0ma5 · 1h ago
It's wild to see would-be influencers continue to build upon a pile of sand, each trying to out pace the lingo on one hand and clinging to anecdotes and conjecture on the other. I wish like other technologies there were first principals that everyone could agree on but you still see it here with people offering what worked for them as some kind of objective fact. Someone should study how these influencer tactics are short circuited by this stuff?
citizenpaul · 1h ago
Step one. Don't absentmindedly twirl your mustache while mumbling how you can finally get rid of all these troublesome employees.
realz · 1h ago
But I had a two hour conversation with ChatGPT and we couldn’t come up any scenario where I didn’t end up winning.
spogbiper · 1h ago
this is an advertisement for a company promising to make your pilot one of the 5% that succeed
creesch · 1h ago
I mean sure, but it is based on an actual MIT study. Which they are clearly trying to spin in their favor, but real data nonetheless.
diggan · 1h ago
> based on an actual MIT study
Speaking of which, anyone have a link to the study itself? Since the linked marketing post doesn't even provide a link to it, just a bunch of links to their own platform and social media accounts.
Probably better for the submission to link directly to the study, without the extra call-to-action fluff at the end?
of course it wasn't as big as this new AI push, but since the 80s AI over promised and under devlivered, it's not the fault of AI, but the greed of the business people.
guluarte · 1h ago
remember Cortana?
srameshc · 1h ago
I found this company today https://www.usemotion.com/ai-employees/ai-sdr and they are selling AI employees. I understand the need to sell. But if you are selling tools as employees, then there is something that will break eventually. I agree with most of the sentiment, that don't try to invest thinking people will be replaced. Make your employees efficinet with tools.
kelseyfrog · 59m ago
There's a huge opportunity to pair this with gig workers to embody AI intelligence with human presence. Kind of like the LinkedIn version of Surroo[1].
Isn't this accomplished via an almanac of published ILS Category III airports and their runways?
I thought this system just picked the closest runway and broadcasts "get out of my way, I'm making an emergency landing!"
DeepYogurt · 1h ago
What I wouldn't give to live in a world where people had long term goals.
LtWorf · 35m ago
For that you need the emperor to fuse himself with sandworms so that he can live thousands of years.
softwaredoug · 1h ago
To me this is an indictment of the leadership style of the day of charging in, laying everyone off, then putting you on an AI death march project. That just destroys institutional knowledge, your credibility as a leader, while inevitably the project fails (and few on the team believed in it anyway).
Sure this type of leadership can work in limited short term circumstances where you have a very clear and meaningful objective. But if you can’t quickly get the team itself to believe in the project - and make it THEIR project not YOUR project - you’re going to fail.
malthaus · 1h ago
95% of any pilots at corporates fail, probably even more than that, you don't need ai for that
jimbokun · 1h ago
Source?
elif · 1h ago
That may be true but what happens when 5% grow exponentially.
markasoftware · 1h ago
if 5% grows exponentially, it'll become 0.025%, then 0.00125%, ...
flembat · 1h ago
This is why I will never get in an AI plane.
anonymars · 1h ago
A one-word difference in the title (Pilot Projects) would have saved a lot of confusion
Just like what often happens in software documentation: a few seconds in the right spot can save hours or weeks of Chesterton Fence work
vcryan · 1h ago
The main reason pilots fail is that people don't see the value.
If they did, they might have addressed many of the issues described in the report as the cause of the failures.
mrweasel · 1h ago
Many of the same issues would equally trip up none AI project. One reason these project might fail is because, like so many other things in business, AI is thrown at a project to cover up organisations short coming, terrible procedures, poor data management and many other issues.
I'm not the biggest fan of current AI technology, but claiming that it's a failure because it doesn't magically solves your problems, which was always more organisations/managerial in nature rather than technical, is a little disingenuous.
EcommerceFlow · 1h ago
The main problem is that Ai "intelligence" has vastly outpaced the associated tooling.
At my last company thankfully I was able to limit the losses to just under $450k which sounds like a lot, and it is, but it could have been about $5m/yr. I didn't sabotage anything, either. It was mostly through simply asking a lot of questions early, repeatedly suggesting we write plans down and make sure everyone on the project is up to day, and suggesting we test drive components before buying the whole car.
Spent a bunch of money, rolled out the pilot project to 70 folks, and began 30 day checkins. After 30 days we had a retention rate over 80%. By the 60 day mark, we were down to 40%. By day 90 we were at 22% and landed at 11% after 4 months and never went higher. By month 8 we cancelled the project and the 8 people still using the tool were ok letting it go, feeling it didn't help that much.
All because someone from the board said, "we need to use AI, anyone not using AI in a year will be out of business." The CEO asked what we should use it for, and the board member said, "I have no idea, that's your job. But we need AI somewhere. Look here, it can buy a plane ticket for me now!" and then wasted the next 40 minutes of the meeting talking about how amazing AI agents were. We're not a tech company, he was definitely NOT a tech person.
To be fair, if the goal is to learn and discover how a new tool works and when to use it, then a legitimate strategy is to solve a bunch of problems with this new tool
You always start with the problem that needs to be solved, not the solution. I'm concerned we'll have another bubble pop like the dotcom crash. A lot of companies are spending money like mad to have any kind of AI thing they can market about without regard to "Does it work right? Does is solve a problem?"
It took my former company a year and half a million dollars to learn that what we actually needed was a completely different tool that promised to do a lot less, but did what we needed it to do quite well. Mgmt got stars in their eyes with all the things the first tool might be able to do someday, and didn't pay attention to what it could do today.
We switched to the other tool at $70/month/user and saw a retention rate over 90% at 6 months. Exact same test group, exact same type of tool, but this tool was more focused and tangible. We went from spending $450k per year at with the first company to $59k/yr with the second company.
AI isn't magic, You can't skip all the project planning steps you'd normally do just because this is AI. Don't fast track it, don't ignore testing data or feedback, etc. It's hard to push back on excitement but it's important that we do. When our emotions carry us away that's when we make worse decisions.
The goal is just "make all the money at all costs"
On the other hand though, with hindsight, imagine the board member saying the same thing about the PC, the internet, mobile computing, cloud computing…
I think both of you are right and it’s only a matter of timing. Time will tell.
It’s easy to imagine that we’re going to make the AI analog of paypal or amazon, but the vast majority of people flailed at doing anything useful with the web until patterns had been established
For the deployment of the PC, I think the vast majority of companies start from an environment where there were already terminal systems or shared workstations available, and the adoption of PCs happened as they became powerful enough to accomplish the tasks that were already being done on the larger computers. It wasn't really a "thou must use PCs, though I know not how they are useful" mandate.
A better example of such a mandate can be found in education, and most of my personal experience with such thou-must-use-technology mandates has been that they've been similarly ineffective as the AI mandate.
Anyway, it misses one of the current biggest problems with all things AI in the workplace: It has attracted every resume-builder and ladder-climber who don't know what they're doing. They just want an AI initiative for their resume.
As long as it's not too public (like the Taco Bell AI drive through disaster) it doesn't really matter if it's successful or not. They can spin it as a success and the next company they apply to won't be able to check.
The same thing happened a few years ago when every PM and rising manager was looking for a way to put blockchain into products. Before that it was "big data", and so on.
> Before that it was "big data", and so on.
Cloud, micro services, NoSQL, serverless, …
We never learn :(
eugh, yeah that's definitely the case. I stopped reading once I got to the first repeated paragraph about data quality.
Speaking of which, anyone have a link to the study itself? Since the linked marketing post doesn't even provide a link to it, just a bunch of links to their own platform and social media accounts.
Probably better for the submission to link directly to the study, without the extra call-to-action fluff at the end?
https://nanda.media.mit.edu/
they seem to be pushing their own take on how to implement agentic AI
frankly this study and the headlines surrounding it all seem like a bunch of marketing spin
https://www.geekwire.com/2015/amazon-makes-echos-alexa-avail...
of course it wasn't as big as this new AI push, but since the 80s AI over promised and under devlivered, it's not the fault of AI, but the greed of the business people.
1. https://surroo.me/
https://www.garmin.com/en-US/autonomi/autoland/
I thought this system just picked the closest runway and broadcasts "get out of my way, I'm making an emergency landing!"
Sure this type of leadership can work in limited short term circumstances where you have a very clear and meaningful objective. But if you can’t quickly get the team itself to believe in the project - and make it THEIR project not YOUR project - you’re going to fail.
Just like what often happens in software documentation: a few seconds in the right spot can save hours or weeks of Chesterton Fence work
If they did, they might have addressed many of the issues described in the report as the cause of the failures.
I'm not the biggest fan of current AI technology, but claiming that it's a failure because it doesn't magically solves your problems, which was always more organisations/managerial in nature rather than technical, is a little disingenuous.
95% of Companies See 'Zero Return' on $30B Generative AI Spend - https://news.ycombinator.com/item?id=44974104 - Aug 2025 (415 comments)
95% of generative AI pilots at companies are failing – MIT report - https://news.ycombinator.com/item?id=44941118 - Aug 2025 (167 comments)