Ask HN: When was the last time you broke production and how?
1cristaloleg25/30/2025, 11:48:09 AM
Inspired by a similar thread on Lobste.rs https://lobste.rs/s/ytefme/when_was_last_time_you_broke_production
Comments (2)
daly · 6h ago
Old Mainframe story... We had a VM/370 system with 8 Meg of memory used by 82 programmers.
VM/370 used virtual machines, one per programmer. The virtual machines were faithful
copies of the hardware.
I was a systems programmer. In the virtual machine you could issue meta-commands such as
'shutdown' by putting the prefix "#cp shutdown". The prefix "#cp" could be changed at boot.
Being curious I tried booting VM/370 in a virtual machine. The embedded VM/370 could then
create sub-virtual machines. Which, of course, raises the question... Will VM/370 boot
in one of the sub-virtual machines? Well, it certainly worked. I set the command prompt
to be "?cp" in the sub-virtual machine.
Which raises the question... Will VM/370 boot in the sub-virtual machine? Well, it
certainly worked. The command prompt prefix was "@cp".
Rise, Repeat... I was 8 levels deep and it still worked. Time to get back to work.
So I typed "#cp shutdown" instead of the latest chosen command prompt...
I had absolute priviledge on the real hardware so "#cp shutdown" was executed by
the real, main VM/370 image. Which promptly shutdown the real machine.
Of course, only I and my co-worker systems programmer knew what actually caused
the outage.
incomingpain · 17h ago
Couple weeks ago I needed to update the wildcard SSL cert on my app. They updated various servers but not my stuff. Fine, I went to 1pass and grabbed the .crt and .key
Installed it and the .crt was the new one.
The .key was the old one that wouldnt work with the refreshed cert. The new .key wasnt in 1pass. I had to go find it. Eventually was able to export the cert off another server and then use openssl to extract the private key.
I was a systems programmer. In the virtual machine you could issue meta-commands such as 'shutdown' by putting the prefix "#cp shutdown". The prefix "#cp" could be changed at boot.
Being curious I tried booting VM/370 in a virtual machine. The embedded VM/370 could then create sub-virtual machines. Which, of course, raises the question... Will VM/370 boot in one of the sub-virtual machines? Well, it certainly worked. I set the command prompt to be "?cp" in the sub-virtual machine.
Which raises the question... Will VM/370 boot in the sub-virtual machine? Well, it certainly worked. The command prompt prefix was "@cp".
Rise, Repeat... I was 8 levels deep and it still worked. Time to get back to work. So I typed "#cp shutdown" instead of the latest chosen command prompt...
I had absolute priviledge on the real hardware so "#cp shutdown" was executed by the real, main VM/370 image. Which promptly shutdown the real machine.
Of course, only I and my co-worker systems programmer knew what actually caused the outage.
Installed it and the .crt was the new one.
The .key was the old one that wouldnt work with the refreshed cert. The new .key wasnt in 1pass. I had to go find it. Eventually was able to export the cert off another server and then use openssl to extract the private key.