We had a disaster yesterday. It was pretty catastrophic, it involved our product out there in the wild world. As a team we failed to ensure that our core and essential features were stable and as a result we had to pull the latest product from the distribution lines. As it happens, most customers either didn't know we messed up but could have. We very swiftly replaced copies publicly available, reverted POCs to the new modified/fixed version. Basically it sucked real bad. Anyway, from the ashes the phoenix arises and so we realised the gaps in our process and are now gearing up to never do that again. It hurts, but it's essential to highlight where we need to improve and nothing teaches you like failure. So I'm kinda terrified and enthusiastic about this.
The great thing is that it gets things happening that perhaps was highlighted but fell on deaf ears and now because of it - stuff's happening.
We've designed a disaster - hope it helps.