Someone I know works for a municipality in digital transformation. They have a public facing website where local residents can report things to the municipality, such as potholes, water system issues, noise complaints, etc.
The UI has this huge taxonomy with like 200 categories with three levels of nesting that route to different departments. There are multiple "other" categories.
When a resident chooses other, it becomes an employee's job to choose a different category. But 30% of the categories employees choose are wrong (about the same for residents).
It's a UX problem, a taxonomy problem, a training problem, a change management problem, a work routing problem, and all these contribute to longer resolution times.
My friend started tinkering with a small quantized local LLM, testing whether it could classify the reported issues more accurately than the public/staff. Of course it could. They're preparing to integrate it into production. Installing it will dramatically improve the UX for the residents, save staff time (at least hundreds of hours a year), improve resolution time, etc.
My friends boss mentioned it at a conference and apparently "no one else is doing this".
So yeah, we are just barely scratching the surface of the productivity opportunities LLMs offer. It's early, not because the technology isn't developed enough to help (it is), but because most people are still figuring out how it can help.
caminante · 37m ago
Wait for the next admin to unplug the decision tool so they can re-add the jobs and backfill with their private staffing service.
codyklimdev · 2h ago
I think a lot of the reasons for this is because AI helps provide a productivity boost to non-profitable sectors of most businesses, i.e. software development, finance, HR, etc. Since these departments do not directly drive profits, there's no visible bottom line to make meaningful observations on.
I do software for a retail company now and we've been having a similar debate: AI helps me and other departments do work more efficiently, but me getting a feature out the door faster is better for the business but doesn't get more products off the shelves. So, to the shareholders and the C-suite, is AI doing anything for the company?
comte7092 · 2h ago
> Since these departments do not directly drive profits, there's no visible bottom line to make meaningful observations on.
“Bottom line” is a reference to costs, it doesn’t matter whether a department is a profit center. If AI is making these departments more efficient, it should show up in the bottom line.
credit_guy · 2h ago
AI does show up in the bottom line, but it’s not attributable to AI. How can you tell if the last month’s 20 new features vs 10 new features the same period one year ago are due to AI or simply to your developers being smarter/working harder/returning to office, etc?
comte7092 · 2h ago
I would pose that question back to you, if it isn’t measurable, how are you certain that it is affecting the bottom line?
JumpinJack_Cash · 58m ago
> > I think a lot of the reasons for this is because AI helps provide a productivity boost to non-profitable sectors of most businesses, i.e. software development, finance, HR, etc. Since these departments do not directly drive profits, there's no visible bottom line to make meaningful observations on.
So the right place to look should be looking at free time of employees or cognitive load/stress they are under.
How is it possible to measure that?
Anyways Goldman might not be the right firm to measure it because they are not interested in anything that isn't money.
codyklimdev · 9m ago
Very much agree on your last point.
I think finding out whether or not AI is actually boosting productivity is a problem of measuring productivity period, which at the very least my current company is pretty bad at. For a developer, is their productivity their lines of code produced, hours worked, project tasks completed per unit of time, agile points completed per unit of time, PRs reviewed, PRs submitted? In more human metrics, is it what their coworkers say about them, what leadership says about them, what customers say about them, testers, QA? The amount of bugs they fix, the amount of bugs they don't ship?
Sorry for the ramble, but apply this productivity measurement conundrum to entire corporations and it's no wonder that no productivity boost is being recorded. I'd be surprised if semi-accurate productivity measurements were taken in the first place.
electric_muse · 2h ago
It’s still too early to see real benefits.
Wrapping business processes around these LLMs is the same kind of hard organizational problem plaguing most internal IT projects. People are still the bottleneck.
You also run into the issue of accuracy compounding. Running multi step flows with AI compounds the success rate and dramatically increases the chances of a full-job failure. E.g. even at 99% success rate for any single step, a 30-step process is only likely to succeed 75% of the time without errors. If you go down to 95% success for each, you only have a 75% likelihood of flawless execution at about 6 steps.
So it’s also about getting those per step success rates way up.
lenzm · 2h ago
It has been years, when will it not be too early?
bluefirebrand · 2h ago
Something I'm noticing is that people in my network have lost some kind of concept of time passing
I had a conversation with an acquaintance a few weeks ago who was adamant that ChatGPT only really showed up less than a year ago. They were absolutely mind blown when I pointed out that ChatGPT got crazy popular literally years ago, late 2022, early 2023. They were convinced this was still very new stuff, like 8 months ago or so
I don't really blame people either. Personally I feel like the years since COVID have been a weird blur. People don't realize that lockdowns were half a decade ago already
blitzar · 2h ago
If it isnt showing up the they are probably depreciating the massive hardware spending over a long time, assuming that the announced billions are actually occuring.
ksec · 1h ago
Lots of companies are using AI already, which means there is no competitive edges against another company.
It mainly helps with mundane tasks. I think mostly employees have better life within the company do those stupid task or another email or meeting notes.
xpe · 1h ago
Bad logic on at least two counts. Try again.
ksec · 18m ago
Thank You for your input. May be next time dont reply.
xpe · 5m ago
Passive aggression is not a good way to live. Fix your logical errors. Are you saying you really don’t know better? Keep in mind that a low quality comment gets read more times that it is written. You may not understand this now, but you really should appreciate feedback of all kinds. If you look at my comment history, you’ll see I engaged substantively. With my earlier comment, I just flagged it because it wasn’t worth my time to spell it out to you.
ericdotlee · 2h ago
My guess is they don't really know how to price it in yet - but also they seem to massively reduce headcount in areas where you can loftily make assumptions that "ai is boosting productivity"
Seems like hand waving and layoffs will have to stop before we get real data
I remember identical assertions about the pc.
If you want to understand business trends, ask an accountant.
In 7-10 years.
quickthrowman · 1h ago
The Altair 8800 came out in 1974 with no video output. The Commodore PET and Apple II were released in 1977 as non-kit computers with video output. VisiCalc came out in late 1979. We are approximately at the VisiCalc release date now with LLMs.
Someone I know works for a municipality in digital transformation. They have a public facing website where local residents can report things to the municipality, such as potholes, water system issues, noise complaints, etc.
The UI has this huge taxonomy with like 200 categories with three levels of nesting that route to different departments. There are multiple "other" categories.
When a resident chooses other, it becomes an employee's job to choose a different category. But 30% of the categories employees choose are wrong (about the same for residents).
It's a UX problem, a taxonomy problem, a training problem, a change management problem, a work routing problem, and all these contribute to longer resolution times.
My friend started tinkering with a small quantized local LLM, testing whether it could classify the reported issues more accurately than the public/staff. Of course it could. They're preparing to integrate it into production. Installing it will dramatically improve the UX for the residents, save staff time (at least hundreds of hours a year), improve resolution time, etc.
My friends boss mentioned it at a conference and apparently "no one else is doing this".
So yeah, we are just barely scratching the surface of the productivity opportunities LLMs offer. It's early, not because the technology isn't developed enough to help (it is), but because most people are still figuring out how it can help.
I do software for a retail company now and we've been having a similar debate: AI helps me and other departments do work more efficiently, but me getting a feature out the door faster is better for the business but doesn't get more products off the shelves. So, to the shareholders and the C-suite, is AI doing anything for the company?
“Bottom line” is a reference to costs, it doesn’t matter whether a department is a profit center. If AI is making these departments more efficient, it should show up in the bottom line.
So the right place to look should be looking at free time of employees or cognitive load/stress they are under.
How is it possible to measure that?
Anyways Goldman might not be the right firm to measure it because they are not interested in anything that isn't money.
I think finding out whether or not AI is actually boosting productivity is a problem of measuring productivity period, which at the very least my current company is pretty bad at. For a developer, is their productivity their lines of code produced, hours worked, project tasks completed per unit of time, agile points completed per unit of time, PRs reviewed, PRs submitted? In more human metrics, is it what their coworkers say about them, what leadership says about them, what customers say about them, testers, QA? The amount of bugs they fix, the amount of bugs they don't ship?
Sorry for the ramble, but apply this productivity measurement conundrum to entire corporations and it's no wonder that no productivity boost is being recorded. I'd be surprised if semi-accurate productivity measurements were taken in the first place.
Wrapping business processes around these LLMs is the same kind of hard organizational problem plaguing most internal IT projects. People are still the bottleneck.
You also run into the issue of accuracy compounding. Running multi step flows with AI compounds the success rate and dramatically increases the chances of a full-job failure. E.g. even at 99% success rate for any single step, a 30-step process is only likely to succeed 75% of the time without errors. If you go down to 95% success for each, you only have a 75% likelihood of flawless execution at about 6 steps.
So it’s also about getting those per step success rates way up.
I had a conversation with an acquaintance a few weeks ago who was adamant that ChatGPT only really showed up less than a year ago. They were absolutely mind blown when I pointed out that ChatGPT got crazy popular literally years ago, late 2022, early 2023. They were convinced this was still very new stuff, like 8 months ago or so
I don't really blame people either. Personally I feel like the years since COVID have been a weird blur. People don't realize that lockdowns were half a decade ago already
It mainly helps with mundane tasks. I think mostly employees have better life within the company do those stupid task or another email or meeting notes.
Seems like hand waving and layoffs will have to stop before we get real data