Bank forced to rehire workers after lying about chatbot productivity, union says

The few times I've let a company sucker me into engaging with a chatbot, it was nothing but a worse interface to searching their support website. It was capable of nothing but directing me to pages which could not help me, because what I needed was not more information about the problem I already knew I had, but someone to fix the damn problem.

I'm currently working on adding a bot to our support chat at TalkJS. And it's great, it has probably a 90% success rate at handling complex queries. But that's because we're throwing money at it. That chat is normally staffed by senior devs, meaning it's not unusual for a single response to cost $10 of labour.

If you approach it as a cost cutting exercise, you end up with crap. If you approach it as a way to make a better experience while you sleep, it's achievable.

My last experience with a chatbot wasn't terrible. It asked me all the relevant information about the problem, even follow up questions, and all required reference numbers, and then passed me off to a human support person.

A human would have done the same thing but it would be a waste of time for them.

It was actually nice that I wasn't stuck in chatbot hell, I didn't have to ask for human, it just did it at the right moment.

The only time a chatbot worked for me is Amazon's, of all things. It auto approved my return after I answered a few questions.

I haven't had any chatbot outside that be useful to me. I always end up getting to the end of all the prompts only to be told I need to speak to a human or the chatbot going in a circle, in which I have to reach out to a different layer of support.

Success rate depends on many factors (risk of failure, your value to the business, complexity of the ask), but it's definitely on average much higher than 5% (I sell this technology and look at the results many times a day).

> Companies need to stop looking at customer support as an expense, but rather as an opportunity to build trust and strengthen your business relationship.

This is bang on. But unfortunately many companies have top down mandates to drive costs down (without backstops for LTV retention) and they look at top line growth as separate from OpEx. It's weird and broken, but it's a side effect of the common organizational structure of most enterprises. There are companies that do not look at themselves divisionally as CX, Sales, Product, Marketing etc. and the ones I can think of do have very high NPS (apple comes to mind).

Preach. Shout out to the be/st bank in Germany, Commerzbank. They advertise being reachable 24h a day[1] but all day you gotta go through a chatbot first and beyond core hours there isn't a human at the end of the line. Worse, they don't tell you what their actual core hours are so if you are calling at, say, 9:00, you could be suckered into waiting on the line for the next hour for an actual human agent. You could call 16:01 on a Friday not realizing that the next opportunity to talk to a human agent is at 10:00 the next Monday.

Okay, the last one is a stretch but people worried about their money are desperate. Not to go all radical all of a sudden here but fact of the matter is banks hold a lot of power over your average guy. The least that they could do is show some respect for your time.

[1] See: https://web.archive.org/web/20250617083138/https://www.comme... and even the actual page as of this writing.

I had a joint account with an ex who now lives abroad and I no longer have contact with.

I talked to the bank and there was no way to close the account without both of us present.

Recently they released a chat bot on their app and so I asked it to close the account and the bot did it for me! That's the best success I've had with a CS bot.

Possibly the policy changed in the mean-time or the lack of activity in the account for several years allowed it to happen (though the humans never told me after x years of inactivity I'd be able to close it)

Most of the companies where I've had first hand experience test customer experience in the aggregate, ie NPS and similar. Part of this is scale, but a big part is also CS is viewed as a cost center and it's a rare individual who does what you're suggesting. There's also seldom any sorry if pay off for making that one customer incredibly happy, CS people are not usually tested very well by anyone

I've had bad luck. Most of it very frustrating where the bot obviously doesn't understand anything.

My best luck with a chat bot was ironically only because of HN.

I was to complaining about amazon's chat bot (it would send me in an infinite loop of directions) and someone who worked at Amazon on HN told me that there were multiple chat bots, and they told me the right one use (I had to click a different link on the amazon webpage than I was clicking).

That one worked ... it took some engineer on HN to make me understand how to make it work.

Look at this from today: https://s.h4x.club/WnunYn98 > https://s.h4x.club/NQuXwXmQ > https://s.h4x.club/L1uwdwwo

I'm literally trying to give them tens of thousands of dollars...I dunno why I bothered even engaging with it, I hoped it would end up taking a report or something, but it doesn't, it just wastes my time.

I had a chatbot attempt to answer an issue I had with setting up a Shopify site yesterday. All of the information was correct, but I had already done it. A customer support rep was able to retrigger the TLS query. The chatbot knew that needed to be done, but couldn't actually do it. It suggested ways of doing it which didn't exist in the UI, which was even more frustrating

Maybe 75%?

I've gotten pointed to documentation I never would have found and I doubt a human would have found. I've had returns immediately processed rather than waiting 2 days for a RMA to show up in my email. I had a subscription rate lowered (my desired outcome) when I tried to cancel a service. And I've had a software bug escalated to the appropriate team within a couple of minutes. And all these interactions were probably 10x faster, at least, than they would have been with a human.

I love chatbots for customer service. Not because they save the company money, but because they seem to save me a ton of time (no more 20 minutes of hold, followed by being put on hold for 10 minutes multiple times), and they seem to follow policies more "objectively", and they escalate easily whenever they can't handle something. It just seems like more reliable and faster outcomes for "normal" support, and then you still get a real agent for more complicated situations.

It is consistently worse than a google search with a site: set to their website. My best interactions have been with ones that were slightly less annoying than a phone-tree to get to a CS rep.

It has one major use case: Converting natural language into a logged and understood FAQ issue.

Yeah definitely less than 5%. Probably more like 1%.

"Success rate" is a tricky metric.

I had to use an Amazon chatbot a few weeks ago. It introduced itself as "Deepshikha". After that facepalm, I started down the path with its various preferred responses to get a refund on an item I bought that never arrived, had no tracking information, and was almost certainly a scam by a third-party seller. I eventually, after a few tries, selected the right combination of things to get the refund processed. But the chatbot wasn't helpful, it didn't make any decisions, and it simply served as a filter for scam refund requests.

I guarantee you that some middle-management PM and some VP at Amazon counted that interaction as a success. I'm sure that's how it ended up on their quarterly graphs and charts. After all, the customer (me) got what they wanted and the right decision was made. And, !bonus!, it used "AI", reduces cost, and had low latency. Raises and promotions are almost certainly incoming!

But the experience was abysmal and insulting, contibuting to the ongoing ensh*ttification of the Amazon experience.

>Companies need to stop looking at customer support as an expense, but rather as an opportunity to build trust and strengthen your business relationship.

As someone that's worked in basically a service industry my entire life, good luck with this. I don't disagree, I'm just old enough to understand the world that humans build, and this type of long-term approach is dead in the current "Profits over all" culture of the US.

85%? I suppose most of the time I’m just seeking very specific information, so an RAG approach of it retrieving some document works fine for me.

The Amazon returns chatbot has always been successful for me at redirecting me to an actual human after several tries.

I bet chatbots are very successful when measuring how much the interaction costs, which seems to be what most companies are measuring when it comes to customer support. The problem is that it's very easy to measure cost (how many person-minutes did it take and what's your hourly cost for support agents, or how much API usage did it take for the bot?) and very hard to measure any outcome the customer actually cares about. Fix this misaligned incentive and the rest will follow naturally, but that requires treating support as a facilitator for the rest of the business rather than as a pure cost center that needs to be minimized.

I remember the pre-AI Geico chat bot that I liked. I could call it once every six months and pay my entire balance with a few words. But then the company started leaning harder on monthly payments and the "pay entire balance" option was removed and I now must either laboriously speak-out the entire dollars and cents due or talk to a person.

What is to say that a lot of the functions that a customer service person does is getting people things they need and that the company resists giving to them. Which is to say that companies mostly need customer service agents because the company's raw impulses are so shitty they need someone with the slight independence of a customer service agent just to provide the services their customers need.

It's like why I never go to company websites despite being very web-savy. These websites only serve the company's idea of what I get and if I'm calling at all, it's because I need more than that.

Naturally, the point is an AI chat can't do customer service because it can't override policy, tell people tricks and similar things.

I had an experience recently where the chatbot gathered details about my problem, but then referred me to a knowledgebase article. I just replied "human" and it connected me with a human, but the AI must have given them a detailed summary, because they joined the chat, said "I understand the issue, let me see what I can do," and then two minutes later, said "I went ahead and fixed that for you on the backend."

One way to look at that anecdote is "the AI failed." Another way is "the AI made the human agent about 100% more efficient." I'm pretty sure CS agents don't love gathering basic info.

This is mentioned a lot, but it's still true - people on HN are not representative of the majority of users for customer support.

The majority of support tickets are repetitive and answered by a simple formula the representative churns without thinking. Which is likely easily replaceable by chatbots.

Actually call volumes increased:

“ The union took CBA to the workplace relations tribunal earlier this month as the company wasn’t being transparent about call volumes, according to a statement Thursday from the Finance Sector Union. The nation’s largest lender had said that the voice bot reduced call volumes by 2,000 a week, when union members said volumes were in fact rising and CBA had to offer staff overtime and direct team leaders to answer calls, the union said.”

This was my read of this as well! What a stupid metric. The first thing I thought when I read that was "Yeah, that's probably because people stopped calling and started looking for another bank."

That'll be what I'll do if my bank starts replacing people with AI. Take my money out and go somewhere that isn't trash.

True, but the union protected its workers from those at the bank. That is the value in the union. In jurisdictions without a union or parity labor policy, these workers would have no recourse for this fraud and the lies.

Robinhood's support bot is basically like that. Completely useless LLM that marks cases resolved when you give up.

I'm sure in some board room some exec is bragging about its smashing success rate.