Its an immense problem that it got so little attention until someone made a TV dramatisation of it. There are definitely many other miscarriages of justice caused by the assumption that systems are always correct, and these are not being corrected.
The lack of consequences for those responsible also means there is no deterrent to others doing similar things.
tajd · 23d ago
I think hopefully it raises that awareness that these systems are imperfect and do not allow people to delegate their decision making totally. I'm afraid rise of AI and the increase in more plausible sounding yet incorrect behaviours won't help us there. We should use this as an instructive example on how we can built more transparent/useful systems for all of our users.
chrisjj · 23d ago
> these systems are imperfect and do not allow people to delegate their decision making totally
There's no evidence this scandal arose from software allowing any delegation of decisions.
tajd · 23d ago
Ah ok, potentially I was imprecise - the post office leadership appeared to assume that the software was completely correct. That is what I was trying to frame as "systems are imperfect and people should take care to validate what they are doing"
jemmyw · 22d ago
> the post office leadership appeared to assume that the software was completely correct
Not even that. The leadership appeared to not care or care to understand what was going on. They showed a lack of competence and tried to cover it up, badly, when they realized they might be in deep shit. The software was just one part, and really just further wrapped up in leadership failure.
chrisjj · 23d ago
Thanks for the clarification. Yes, any system at fault here looks more likely to be corporate than computer.
The criminal prosecution will hopefully determine the extent to which the leadership did genuinely make that assumption.
jmkni · 23d ago
If you think about it, it's crazy how unregulated software development is compared to other industries, considering every industry runs on software.
If you screw up as a Doctor or a Solicitor or an Accountant etc, you can get done for malpractice or struck off or go to prison.
Somebody will basically say, 'you did such a terrible job, you aren't allowed to do that job anymore'.
Screw up so badly writing software that 13 people commit suicide and sweet fa happens.
chrisjj · 23d ago
The cause of these suicides is not software. It is extortion facilitated by UK courts of law.
I'd like to hear what will be done about the judges responsible.
Spooky23 · 23d ago
Agreed. Often the “solution” laid out is more accountability for the engineer, which I find equally disgusting.
I’ve been in big company/gov tech for a long time. The companies building these types of projects are almost universally incompetent and often malevolent. Having had the occasion to “rescue” a few big projects, they are usually infused with a culture of incompetence and ambiguity in accountability for all things besides billing.
The problem is that it’s what I call “mutually assured incompetence”. The customer is bad at writing RFP/tenders, and often accepts substandard work out of ignorance. Contractors are disincentivized from delivering a sensible outcome as a result. Contractors only understand pain as a motivation once the deal is done. Garbage in, garbage out.
The failure of accountability is endemic as more and more tasks are automated and outsourced. Something as trivial as cash room accounting is a proven practice, traditionally with many eyes keeping all parties honest. While the personal tragedy isnt a factor, in the US, Bank of America “lost” tens of millions of dollars of physical cash in their outsourced counting facilities. The company’s response? Spend a fortune to physically shift cash around ahead of audits to pretend the money was there.
HPsquared · 23d ago
These are the natural limiting factors on outsourcing and the service-based economy in general.
chrisjj · 22d ago
How does out v. in sourcing make a difference?
HPsquared · 23d ago
It's covered by whatever activity they are writing the software for. It's like saying "it's crazy how unregulated reading and writing are" (copyright notwithstanding).
Its an immense problem that it got so little attention until someone made a TV dramatisation of it. There are definitely many other miscarriages of justice caused by the assumption that systems are always correct, and these are not being corrected.
The lack of consequences for those responsible also means there is no deterrent to others doing similar things.
There's no evidence this scandal arose from software allowing any delegation of decisions.
Not even that. The leadership appeared to not care or care to understand what was going on. They showed a lack of competence and tried to cover it up, badly, when they realized they might be in deep shit. The software was just one part, and really just further wrapped up in leadership failure.
The criminal prosecution will hopefully determine the extent to which the leadership did genuinely make that assumption.
If you screw up as a Doctor or a Solicitor or an Accountant etc, you can get done for malpractice or struck off or go to prison.
Somebody will basically say, 'you did such a terrible job, you aren't allowed to do that job anymore'.
Screw up so badly writing software that 13 people commit suicide and sweet fa happens.
I'd like to hear what will be done about the judges responsible.
I’ve been in big company/gov tech for a long time. The companies building these types of projects are almost universally incompetent and often malevolent. Having had the occasion to “rescue” a few big projects, they are usually infused with a culture of incompetence and ambiguity in accountability for all things besides billing.
The problem is that it’s what I call “mutually assured incompetence”. The customer is bad at writing RFP/tenders, and often accepts substandard work out of ignorance. Contractors are disincentivized from delivering a sensible outcome as a result. Contractors only understand pain as a motivation once the deal is done. Garbage in, garbage out.
The failure of accountability is endemic as more and more tasks are automated and outsourced. Something as trivial as cash room accounting is a proven practice, traditionally with many eyes keeping all parties honest. While the personal tragedy isnt a factor, in the US, Bank of America “lost” tens of millions of dollars of physical cash in their outsourced counting facilities. The company’s response? Spend a fortune to physically shift cash around ahead of audits to pretend the money was there.