Recently I changed some configuration on a client's site that triggered thousands of emails to different customers, forcing our client's Customer Service to handle the same amounts of customers calls on a day that they would have on a normal week, and finally having our client writing an apology email for all affected users saying that there was no data breach in the site.
It was a long day.
Admit it
Shit happens, and people make mistakes. I did on that occasion, I probably will again in the future or somebody else will, but I will be happy if we as a team don't make the exact same mess I did.
The first thing to do when you screw up is to admit it: tell somebody that you screwed up and tell that person how you screwed up exactly.
If there's a big problem, and you alone caused that problem, chances are people will be chasing ghosts to fix something they won't understand completely how it started (if you don't come forward and tell them exactly what happened).
You are the only one that knows, basically (until everybody finds out).
The idea is to change everybody's mind from panicking about a mystery error to focus on how to put down the fire and doing some damage control. Back to my example, everybody was wondering how the emails went out but when I told them it was me changing a setting then everybody move towards finding a solution (emails were still going out at that point).
Really, trying to cover up something is really a stupid idea because, again, shit happens and it shouldn't be the end of the world, and people will find out rather sooner than later.
Say sorry as you should be
Being on the defense on this type of situations is pretty common, but you shouldn't be. You made a mistake, you admitted it, now apologise without making excuses or blaming something (or somebody) else.
As individuals and as a team we should learn some stuff starting with the fact that human error is an actual thing. Saying sorry is as important as learning to accept someone's apology.
Keep in mind that if you didn't screw up this time you could be the one causing the mess next time, so don't believe you are so perfect. At the same time, if you were the cause of all the problems today, relax, somebody else will take the leading role next time.
Not the same mistake twice
When the adrenaline is over and the problem is solved do everything within your reach to avoid the same mess to repeat itself.
Let's try to have new problems, not always the same ones (it's boring that way).
While all human errors can't be eliminated (unless there's no human in the equation) we can always reduce the chances for them to happen by identifying what mechanism we (the team) can put in place to prevent them.
Back to my emails, the setting I changed wasn't on the Live site but on a testing environment, that happens to contain real customer data. In this case the problem was on how we create those testing environments (a fault on the process we have in place for doing that).
The problems could be prevented by improving the processes, adding any necessary documentation, and most important by spreading the information across the team.
Learn from a mistake is not a cliche phrase but instead it's something "tangible".
If you really learned from a mistake you will ended up with more documentation available to the team, a better process in place, and everybody informed of what happened and what can be done to avoid it happening again.
Always keep in mind that La La Land was announced the winner of the Best Movie category in the The Oscars ceremony because somebody handed the wrong envelope. And they improved the process for the following events.
