Post Office Horizon Scandal – Computerphile – Blog

People lost their lives because of this. And people who paid up unfairly never were compensated properly. It is utterly disgusting.
Nuts n Proud
Our local sub-postmaster used to print out a duplicate receipt for every transaction even somebody wanting a stamp and keep a copy. Post Office once said he was short and had the duplicate paper receipts to prove he wasn’t. It cost him a lot of money for the extra paper rolls and ink cartridges but he said it was worth it.
Robin Nixon
I followed this case in Private Eye over the years, and it was clear from the start that the Post Office knew it was more than a coincidence that so many postmasters were coming up short, but it seemed that they just didn’t want to lose face by admitting there was an error, so carried on prosecuting and prosecuting. Absolutely scandalous.
One of the worst things in life is being accused of something that you didn’t do. I can’t imagine how horrible it must feel to go to jail over a software glitch…
Sven Rötters
The real scandal is that the court automatically assumed that the digital data presented is correct and trustworthy. That is the main topic of our times: Are our digital systems trustworthy enough to count as evidence in court? Perhaps Computerphile should invite Ross Anderson again for some basic security engineering lessons. 😉
It’s a disgraceful story. Distributed transaction processing with two-phase commit was sorted and implemented in the mid-1980s. If systems today are failing the ACID test it’s due to incompetence by the system designers.
Dag-Erling Smørgrav
You got the “isolation” in ACID wrong. It doesn’t mean different parts of the system are isolated from each other, it means concurrent transactions don’t interfere with one another.
These kinds of bugs are remarkably easy to get. I had to fix a bug in a client’s webshop that would sporadically get way more stock in thewebshop than they actually had. Turns out there was some sort of denial of service-ish thing where robots where trying to reserve as many items as they could. But ofcourse if you never actually purchase then the system will remove your shoppingcart and return the items to stock. Turns out that if people where reserving items while it was getting returned to stock the deleting of the cart would fail and a little later the same cart would be deleted again and presto; more stock than you started with. That disappeared when I implemented transactions. And yes I notified the author of the shop and basically told him to go and learn how databases work.
Anthony Jewell
This was an absolute scandal and ruined people’s lives. The power the post office had under an ancient law to privately interview the accused in police stations supposedly under caution and hide other cases from them was egregious and compounded the problem. My heart goes out to the victims who have suffered from these baseless accusations. I recommend all SW engineers and system designers listen to the BBC and guardian podcasts on the subject to appreciate how important it is to get these things right and show true due diligence in their profession. Don’t be afraid to call out issues you might see in any systems you work on. Anthony (semi-retired SW engineer).
The Magpie
This whole story warrants a lot more conversation in the software industry. Going forward it should be required reading alongside the Therac-25 scandal. It also demonstrates the issues with blindly trusting computer programs without any compassion for real people. The many stories of those who were prosecuted, fined, jailed, some of them dying before their verdicts were overturned, are heartbreaking.
George Anderson
I am pleasantly surprised by how many times I enjoy watching videos of the British postal system. Thanks to Tom Scott for introducing me.
Aidan Crane
This was huge, people lost their livelihoods and were accused of theft and fraud and every case wrongly won by the post office against the sub postmasters was a gross miscarriage of justice. There was a woman local to me who’s even own family thought she was taking money and this is a perfect example of when you should know your users. There was also some evidence of the post office and Fujitsu the supplier knowing that there were issues and remained silent while people went to jail and in one case ended their own life. (Allegedly)
This is an example of why all software that touches public funding should be free, and, in the case of internal government systems, should be released to the public. Being able to audit Horizon would’ve given people a fair defense in court. (See the FSFE’s “Public Money, Public Code” initiative.)
I’ve had a problem similar to this before, a meal payment transaction system would charge you without checking to see if the order had actually been successfully received. Bad code does show up often in the world.
Nick Norton
Closed source, poorly audited, faceless unaccountable/uncontactable software devs. Execution by Software rather than Executable Software.
Seems like the people at the post office responsible for the cover should now be accountable for ruining people’s lives. Some served substantial prison sentences for this. Also, thoughts and prayers for any of the devs with Horizon on their CV :😬
I see everyone in this comments section is commenting on the (shameful) scandal itself, but I wanted to take a moment to say that this is also a great video explainer! Clearly explained so that all the main issues are understandable to even the slowest of us… great job, both Professor Murdoch and Computerphile!
It's Only Me
The behaviour of the Post Office was atrocious. Has anyone been held accountable?
donkeydunn Dunn
Good video, definitely want to see more stuff about databases, even if it is historic information i.e. development of ACID. Steven seems like a good presenter so look forward to more from him.
I’m surprised this sort of thing doesn’t happen more often. Once you replaced the mainframe guys who were very, very careful about ACID compliance with a bunch of Java guys in the 90s and 00s who had never heard of ACID correctness and then tried to get it to work across multiple, disparate systems with no global transaction rollback was sure to break somewhere.
Kendawg McAwesome
A perfect example of how we can all expect to be treated as “infallible” machines and algorithms control more and more of our lives.
I once ordered at Burger King through these touch screens. Right at the point of paying, the system froze and did a reboot. No receipt printed, no order placed, but money withdrawn from my account… What a struggle that was to explain.. 🤦‍♂️ Bad code is out there, even at big companies..
Forthright Gambitia
The explanation of isolation wasn’t exactly correct. It is more about concurrent transactions, and the need for serializability (i.e. order of operations doesn’t change final outcome) regardlessly of whether they are on the same system or not.
Daniel Davies
A lot of managers, executives, and lawyers working for fujitsu and the post office should be spending a looooooooong time in prison for conspiracy to pervert the course of justice over this
Leonardo Felippine
Let me guess: This system cost hundreds of thousands or maybe millions of pounds to be developed?
Adam Brown
Please tell me a massive scandal isn’t going to be the exact same problem used to introduce concurrency problems to freshman CS students.
A great explanation of the Horizon ‘logical error’. My heart goes out to those that suffered as a result.
As a DBA, the gross incompetence — from bottom to top, tools to management — disgusts me.
Creepy Chris
thanks for this, this is exactly what i wanted to learn about – what bugs causes what issues. poor people who went to jail and lost their jobs
Miguel d'Oliveira
Some people in very high positions knew about the mistake that put inocents in jail. Life sentences for those psychopats would be a reasonable start to making them atone for it.
I followed this via Private Eye and as a computer science graduate I was always very suspicious of the guilt attributed to the people prosecuted. It just didn’t make sense that people were brazenly robbing the Post Office of money in the manner that they were supposedly doing so. Also it should have been fairly straightforward to look at the personal accounts of the defendants and seen either more cash being deposited into their accounts or them spending less from their accounts due to more cash in hand. Of course the argument could have been made that they gave the money to others but this would be unlikely. One of the biggest problems was that the prosecution hid from the defence the fact that there were many other postmasters and sub-postmasters who were being prosecuted for the same issue, so the defence could not then see the commonality in it being the computer system that was at fault. Thank you for this video because it shows just how complex transactional systems and are how difficult they are to maintain in sync – it reminds me of the Two Generals’ problem which was also excellently covered on Computerphile I think.
Wilco van Beijnum
Nobody mentioning the video quality of Steven Murdoch? Looks so good!
Harry Moorehouse
This scandal destroyed people’s lives!
Gareth Milsom
This is a wonderfully calm explanation of the technical failures behind an absolutely outrageous scandal.
Qwerty Plm
Keeping this system running this long was criminal!
Tim Bo
I worked for a company where the accounting software had a variance of about $2.49 between the net assets and the total equity. They must have had a weakness at some point that allowed an unbalanced transaction. To get that fixed we would have had to give the file back to the software provider. We just lived with it as it wasn’t material.
Swagato Chatterjee
As a SWE I have to say we need more regulations in SWE field. Until we treat software like a nuclear reactor, we are going somewhere very very wrong!
Absolutely mindboggling I can absolutely believe that people got locked up for this, though Lots of arguments can be made for why software should be prohibited from prod haha
Sebastiaan Hols
Computerphile is so great. Bringing light to these kind of things.
Anant Mehta
One of the most shocking scandals I have heard of. I hope those wrongly accused get the justice they deserve.
It seems like this was a kind of perfect storm of novel technology. In 1999, the internet was definitely established as a viable commerce platform, but hasn’t reached the level of maturity where you could expect software like Horizon to be implementing basic standards of design competence. It’s up to humans to eagerly question the technology when inconsistencies arise. And never assume malice what can be attributed to incompetence.
Reggie Benes
So the Postal Service bought untested software from the lowest bidder, and assumed it would work perfectly? Then it never seemed strange that shortages appeared that weren’t there before the new software was installed? This is so incredibly incompetent that it sounds like something that would happen in the US, and that should be embarrassing. I’m amazed people were convicted on the evidence from a new and untested buggy system. If I was a person affected, after I was done suing the Postal Service, I would sue the F@C% out of Fujitsu because they obviously covered up their pathetic system problems while peoples lives were being ruined because of it.
Marc Ridders
Famous IT-saying: “Avoid duplication of volatile information”.
Johnathan Wilko
Notice how the large directly operated branches (not the franchised village counters run by a couple of biddies) – the really large ones with many counters. They had the same issues with “being short” by tens of thousand as the innocent biddies. But, the Post Office never prosecuted anyone from their own branches. And the Post Office lied about there being no problems whatsoever. Funny that.
Stafford Campbell
The story is a good example of the danger in giving too much power to bureaucracies. They do terrible things, not because they are run by evil people, but because good, but flawed people don’t want to admit mistakes.
Vincent Groenewold
Not at all surprised, I see this happening today as well during the crisis in the Netherlands, everyone is trying to cover their * and a lot of intimidation is going on from top to bottom, totally illegal, but i simply happens. I wonder if that is simply what management trainings are about these days as it seems so normal. I do hope that all sub-postmasters and especially those in jail got a huge sum of money in return.
David Futschik
The designers and maintainers of the system itself should be facing trial and be held accountable for this. I say that as a software developer and researcher. You make software that deals with serious problems, you better be damn sure you give it serious consideration. The story is remarkably similar every time the government contracts out a software project, low skill SWEs, ridiculous prices, long lasting contracts – perfect kickbacks.
Eliot Mansfield
they hung out loads of postmasters out to dry over this
Chris Clarke
Does anyone know what happened to the Horizon system? Is it still in use (presumably patched) or has it been replaced?
Willis WAN, Chun Yu
Main takeaway: 11:41 “Distributed system makes everything harder.”
David Cronan
One such example of the flawed system is when a post-office suffered a thunderstorm nearby and the power went off. When the power was restored the Horizon system then told the Postmaster they had an extra £32,000 worth of stamps in their shop that had appeared out of thin air.
Konstantin Semionov
Yes computers always count correctly, but people program computers and people make mistakes all the time. The fact that someone would come out and say this software is like a tank is just stupid. I hope those who have covered this up will themselves go to jail
R Jones
“Computers don’t make mistakes”. True, but programmers do.
Gwallter Rixon
If these mistakes were a random wouldn’t they have been equal number of cases where the postmaster was in surplus? If so what did the post office do about that? If that was not the case it implies the existence of some corrective mechanism that was built to only work in One Direction.
John H
Lesson learned: If there is a way to do something wrong, government will do it the wrong way almost every time
developers! listen and learn very carefully ^^ up to date, even banks screw up transactional logic and its not that hard tbh (if you pay attention from the start)
Abraham Samma
I read an essay about this. It made me wince loudly. My God, when I think about AI and how we’ve convinced so many people that algorithms and other AI technologies are seemingly infallible, I get shivers. We may see similar cases in the future on that front.
Nicolas Rivollet
Could you please enable Auto-Generated English Closed-Captions? Even if it’s not perfect it’s really useful for non-native English speakers. Thanks
I learned these things in my third semester of computer science! How can such a large corporation make such awful software?
Alberto DeLaRaza
Were innocent people sent to jail? This is terrifying. I hope justice prevailed in the end and the innocent were exonerated to the fullest and properly compensated — though I have no idea how Britain’s tort laws compare. But given history, I have my doubts. It’s not too far a stretch to equate this disaster to the modern reliance, and unmerited trust of AI and machine learning systems. How long before some AI flags some innocent as a ‘potential malefactor’ that must be pre-denied, pre-ostracized, pre-cancelled, or otherwise denied their right to exist, just because some AI said so?
Sionyn Jones
Steven Murdoch great guy he did a great talks at ccc about chip and pin.
Rick Ellis
Agile approach can sometimes foster these situations because it discourages solving “future problems”, and it happens more often than anyone will admit. Experienced engineers can mitigate such problems but they sometimes have to fight an uphill battle to do the right thing.
꧁Mike Sully꧂
I work in the UK public sector in the IT industry, this is an episode I’m gonna have to miss, it will just make me too angry
Brandon Link
As a former bookkeeper this sounds like a technology failure compounded by a bookkeeping failure. All of these shortages should have been easily catchable with daily or even weekly checks.
jungle man
Please add some more videos on computer security 😭
So big question is: Who bought this shitty system? I’m not even specialist in these systems, but I saw so many wrong things (synchronizing logs? What? Why? It’s point for logs to be untouchable so you can fall back on it if something goes wrong). If you think about it, banks do pretty much similar stuff, arguably on even bigger scale, these systems should be worked out since forever… Feels like someone was trying to save some money, or someone contracted a new company to give a friend nice cozy job to me. And people paid for it…
Peter Anderson
So this is what happens when you accidentally program an accounting database to act out the plot from Superman III or Office Space.
Dmon !
Remember everyone, lives were actually lost thanks to this scandal. And you can blame not only Horizon but also senior post office managers who allowed this to happen. The company needs to be sued beyond imagination and people need to go to jail. Incompetence has led to irrepairable damage and even loss of life
Richard P
I worked in IT in the 1990s, not for the PO or Fujitsu. It was an open secret in the industry that the system was an utter shambles. The BIG question is that The Board of the PO rejected the distemper in September as having major issues that they could not implement it. However, 4 weeks… 4 WEEKS later they approved it and signed it off… WHY? I smell corruption and this may explain the 2013 cover-up.
Menachem Salomon
One of the issues with writing robust code is the difficulty in simulating hardware or network failures. Bugs might only show in very specific circumstances, and recreating these circumstances during QA or UAT is a rare skill. That’s besides for the Heisenberg Uncertainty Principle as applied to computers: Putting software through a debugger often changes the software (or the way it runs) just enough to hide potential failures, making fixing bugs much more difficult.
Rock dweller
Prof Stephen might be the best dressed and we’ll framed guests on computerphile.
Selling stamps? Not the crazy stamp collecting machine? Seriously though, it sounds like it was way too common to be a real thing and should make the post office consider it might be their system’s fault. I’m surprised I hadn’t heard about this, this level of stuff usually comes to our news as well. Sounds like absolutely terrible and usual government level software and system purchase.
I wonder if the software was old enough to have been written on an ICT 1900?
I would love to know, when Horizon was put out to tender (I assume it was a bespoke system written for the Post Office) how many bids were there and why was the winning bid chosen? Who was the person\consultants who created the system specification, did these person/s have an established track record and did they hang around for long in the project? What was the timeline on the project and did it rollout in phases? Did the project have test criteria that covered all possible use cases? How many changes did Post Office Management make to the project mid-stream? Were there changes of Project Management and were there disagreements between IT and Post Office non-IT management? I suspect somewhere in all of this some non-technical person got some great hooraa for a totally botched product but it also wouldn’t surprise me if the system supplier took existing system code that wasn’t suitable and used this as a basis to fast track the creation of something that would appear to tick all the boxes just for the sake of winning the contract.
andrew allen
The post office management should be sent to jail for this!!!
Now remember how many critical systems run on things like COBOL and MUMPS/Cache, and that many of the original developers are dead at this point…
Me Peter Nicholls
There was also a legal assumption built in to legislation that computers did not error.
Lawrence D’Oliveiro
5:58 Technically, “atomic operations”.
Helmut Zollner
Wow. So the basic architecture of the Horizon System was not up to the task.
Johnathan Wilko
Such a simple bit of software. And they screwed it up. In this day and age?
John B
To err is human, but to really foul things up requires a computer. How true those words are.
Paŭlo Ebermann
Breaking all 4 parts of ACID is quite a feat.
Richard P
731 Absolutely shocking, disgraceful, the arrogance and hubris of the management. Someone needs to go to prison.
04:23 shouldn’t that second -10 be +10 in order for the 2 sides to balance ?
Lots of armchair IT experts on this one 🤣
Old Man Doing High Kicks Only In Black Socks
I always wondered why British gangsters claimed to rob post offices. I always thought that was petty and juvenile.
11:12 “It seems like such simple problems” -> welcome to software development. Nothing is simple, the existence of CSS, Overleaf, JavaScript and dozen of IDEs for the same programming language are physical evidence of this fact. The only simple problem is printing “Hello world!” locally in 25 attempts or less within an 8h time limit, everything else is not.
vulnerabilities that corrupt people can exploit is a feature, not a bug
Wait wait wait wait… how is concurrent logging a hard problem? Aren’t multiplayer video games with 64 people essentially a big concurrent log, updated 60 times a second? That doesn’t sound like a hard problem.
james caley
So did real money go missing? My bank balance is a complex number: there is a real part and an imaginary part.
Duncan Taylor
Accountancy made interesting! It’s a miracle!
I really hate making up names for this kind of stuff
The black background makes it look like the speaker is explaining the British post office in the void of intergalactic space
Were any errors in favour of the sub post offices?
Can you do a video on the Microsoft Print Spooler vulnerability “PrintNightmare” ?
This is Skynet, version 0.0.1. When people trust computers more than they should, don’t understand how they operate, and end up harming other people as a result.
11:40 distributed systems don’t make everything harder, trust makes everything harder. in a non distributed systems you trust the parts of that system and never build in failure because it’s so rare. in a distributed systems you have to address these issues up front. distributed systems are superficially harder, for example think about a sudden dropped connection, that’s the same problem as sudden power failure. a operation was stopped at some point before it could move to the next part. it’s a frequently something that happens in a distributed systems so it’s frequently addressed and dealt with, the power failure is super rare and a waste of time to deal with it, so it’s not dealt with, but over enough time the rare becomes likely. you trusted the power to always be their. because distributed systems have both the power loss and the connection loss they are “harder”, but being forced to deal with these issues makes everything easier. because you don’t trust a node when it does something funky, so even a unexpected unknown for failure gets dealt with as odd behavior. P2P systems are hard to build because they make you deal with it all, but are robust.
Alex Landherr
We had a similar problem with the library’s computer system at my elementary school, a book would be marked late or not returned even when we personally handed it to the librarian.
DaWei IsGood
No system is better than its programmers who built it. Seems this was a shitty programming from the beginning and this is just horrible if people got blamed left and right. Poor organization just spiral things. This doesnt suprise me…but poor employees. Management should be in jail for this crap. They were incompetent for sure and probably overpaid. It goes hand in hand.
On the logs, you can encrypt the logs. That restricts access. Secondly you can sign each log entry, with a crypto check sum to prevent modification. Then you have the question of adding and removing log entries. Here you need to generate a crypto sequence of numbers, where the generation from the previous to the next isn’t easy, if you don’t have the key. Then you can check for deletions and insertions into the log
So they went 4 out of 4? wow what was going on in there
Rob Fielding
This is why cryptocurrency is based on a distributed ledger. You can’t just jam the entire world into one ACID database.
So, as a layman, I guess this is what blockchain is supposed to fix?
Couldn’t listen to him for long, but he muddled through okay.
Tim Beaton
Hal 9000 :- “It can only be attributable to human error”  Horizon:- “Hold my beer…”
“What’s PostgreSQL? Best practices? It’s like you’re talking another language!”
Matheus Xavier
The fact that a Person went to jail because the government does not update an ancient computer system is unacceptable. Shame on the law people involved on this case.
Bob Robertson
In a nutshell, this is pretty basic and lame. Stock reconciliation is a problem for all systems – everywhere, due to the transient nature of goods and basic stock management. This video assumes, in defence of ALL PO counter staff, that post offices staff do not ‘pocket’ money. This is simply not true and there is evidence for that. Post office balance sheets are often wrong, the processes needed updated because this is either mismanagement or theft, right? Horizon tried to do that, by making the counters ‘accountable’, but as there are tens of thousands of ‘counters’ processing millions of daily transactions and its very easy for post office staff to steal money, make routine mistakes and of course investigations are very labour intensive to investigate. Horizon did that investigation with very smart technology. People just don’t like getting caught with their ‘fingers in the till’, so no matter how much wittering on this laddie does, he’s missing lots of details on how the Horizon System actually works. Of course, everyone is entitled to their opinion these days, but that doesn’t make it true, or right and on this occasion he is very wrong about so many things.
Dmon !
Horizon need to not only be sued but people need to go to jail. Negligence and incompetence caused lives to be impacted, some permanently so …….
Andy Hall
so was everyone accused a victim of bugs in the software then ?
Philip Mottershead
So what you’re saying is that we need the post office block chain🤣
Benjamin Philipp
F-ing “L-fronting”, making people pronounce “jail” like “J O” 😁
Jonathon Jubb
Double entry book keeping, not rocket surgery….
Kesa Mek
Nice try but computer science doesn’t explain the ongoing cover-up.
James Potter
classic case of companies skimping on QA
Would blockchain contract technology solve alot of the issues in this video?
jed loveday
Top video answer just what i was looking for
Andrew Kilpatrick
So the moral of the story is don’t run a franchise for the government.
Mayukh Purkayastha
Sir i m invented my imagine machine Shakti Prokriya karan yantra or power acceptor large machine or yantra. This power acceptor help update new type quantum computer. I m invented tree sensor Ai computer system 🌳🌱india India Bihar
Mayukh Purkayastha
Sir i am invented my imagine machine Shakti Prokriya karan yantra or power acceptor large machine or yantra. This power acceptor help update new type quantum computer. I m invented tree sensor Ai computer system 🌳🌱india India Bihar India Bihar
What!? Damn. The UK post office speked into civil service hard lol
Mayukh Purkayastha
Sir i m invented my imagine machine Shakti Prokriya karan yantra or power acceptor large machine or yantra. This power acceptor help update new type quantum computer. I m invented tree sensor Ai computer system 🌳🌱india India Bihar India Bihar
It seems to me that some kind of blockchain could solve most of these problems.
Steve Roger
Computerphile we need subtitle, please at least turn on the auto cc subtitle.
Swapnil Lonkar
Effects of crapy software!
Lawrence D’Oliveiro
9:45 YouTube’s comment system does not have durability.
José Samuel Produções
I think the videos of this channel can be more presented. Im interested in the subject of this video, but i cant understand eveerything of it because it was presented in a boring way (just talking). Consider maybe putting more images and demonstrating how the things gone.
Jack Kraken
So how much of the ACID test did you fail? Horizon Systems:”All of them(?)” “¯\_(ツ)_/¯“
Suki Paul
abso outra horizon still in use,maybe the fatcats can now stop licking the cream
What was the podcast? It’s not in the description.
Catarina Pereira
anyone else read the title as “after” + (microsoft) “office”? something to do with after using MS office?
Jack Kraken
So what exactly is wrong with the Horizon systems? Horizon Systems:”Yes”
Jacob LaMountain
why is lemonad stand sell stamP??
ZX Renew
People served time in Jail for this. The coders should be in jail now!
Penny Lane
Oh look, a video where the audio and video quality isn’t complete Zoom garbage!
Simon Tay
What the hell is a lem needs stall? Ive never heard of that before.
Richard Clarke
In the UK people can be jailed based on no evidence except a computer printout. Think about that !
These guys ever heard of the Block-chain????
Let me save you 16min: Accounting software was bad because it didn’t keep track of things properly. That’s all this video is, with a few examples of what the things it didn’t keep track of properly were. I’m finding Computerphile videos more and more shallow recently, and increasingly devoid of anything approaching technical explanations.
love from Nepal 🇳🇵
Jake 28
the seventh first?
Niklas Paulsson
Can the BBC be trusted to report anything correctly these days?