For those not familiar with recent UK scandals, there has been one ongoing for a few years where there was a bug in a system designed by Fujitsu that was calculating the finances for post offices. This resulted in the system quoting wrong balances for post offices, and when that didn't match the actual cash in hand, resulted in prosecutions of the postmasters responsible for those offices which in turn led to convictions, dismissals, fines, imprisonments, and suicides over a 15 year period as the computer system was presumed correct.

Anyway, it was later proven the computer system was incorrect but the government there dragged their heels on exoneration and compensation.

https://en.wikipedia.org/wiki/British_Post_Office_scandal

The fact that this misnomer of infallible computer systems was ever enshrined in law is pretty damning of the whole UK legal system and the relationship between technical people and law.

Every person who has ever programmed a computer or worked in any complex system knows they can't be relied upon 100%.

Not least because it seems to go against the core concept of "innocent until proven guilty" that the whole legal system is meant to rest upon.

For a long time I was pushing a campaign that every piece of software paid for with public funds should be made FOSS (unless national security etc).

I struggle to work out why a post office point of sale should be vital to Britains security and we should have been able to see the code.

On top of which I believe that making such code open means there will become a eco-system of ISPs who will be able to support, integrate and improve the software and provide local government users (ie postmasters) with worthwhile consultancy Under these circumstances it’s hard to see how this would have gone uncovered for so long.

(Or rather, not uncovered, but unbelieved. The great tragedy of this affair is that us was known and reported on for years - but nothing happened. You know those films where the hero manages to get the proof to the newspaper / tv station and the film ends as the bad guys are bundled into police vans - yeah not so much.

That will be wonderful with AI:

   Sir, I have this conversation with chatgpt where the assistant tells me that you are guilty. Based on the fact that chatgpt is able to correctly count raspberries we can now legally consider that it is reliable and so that you are guilty! Game over.

Computers always do exactly as they're told. Unfortunately, there are so many people telling them to do so many things at so many layers of abstraction that the operator can't be sure of anything anymore.

edit: The thing is, had I prefaced it with "ASSUMING A PROPERLY FUNCTIONING COMPUTER," someone would still roll in to pick it apart. You can't please pedants and make a worthwhile point at the same time.

Do circuits get weird? Can a stray cosmic ray flip a bit even in a system with ECC RAM?

Sure.

Does this meaningfully affect a bit about the hazards of abstractions on top of abstractions?

No.

The point is that abstractions amplify computer problems whether they're human error or ghosts in the machine.