An “Oh Crap” moment

[tweetmeme source=”chenelaine” only_single=”false” service=”bit.ly”]

Today I had an “Oh Crap” moment.  An alert coworker discovered that a product that was released and on the docket for a much-anticipated marketing launch had developed a fatal flaw.   Said coworker basically saved my behind, because it would have been very, very bad if that product got launched on schedule (and on my watch too – unimaginable).

The product release was a textbook example of excellent teamwork and extraordinary efficiency in execution.  The fatal flaw was mainly my fault.  That kind of fatal flaw had happened before on a different platform.   The lead engineer on the project hasn’t seen it before, but I have.  I should have anticipated it and made sure SQA tested for the condition that would have triggered it. But I missed it.  I was colossally lucky someone caught it before it was released into the wild.

To our credit, we didn’t run in circles, scream and shout. We called all the key people, triaged the situation, and in 30 minutes we had a calm and rational plan of record to fix the problem. To quote the lead engineer, who turned around halfway through his drive home to partake in this quest: “We are going to do this right; we aren’t going to do this as though our hair is on fire.”

Fortunately, or unfortunately, I’ve lived long enough to have had several “Oh Crap” moments.  So I know from past experience that speedy acceptance of responsibility leads to a speedy resolution of the crappy situation.  So I sent a mea culpa email, explaining what happened, apologized, and explained what we were doing to fix this.

So it’s all out of my system and we are moving on to a fix.  While I feel like crap,  I am very proud of how my team behaved in a time of crisis, and how we are able to mobilize and deal with the situation.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




Twitter Updates


%d bloggers like this: