Shedding Some Light

I have a streak of good luck when it comes to the stereotypical tech support call in the middle of the night. I have never taken an unexpected phone call from a customer between the hours of midnight and six in the morning. You know the call, the one that wakes you out of a slumber when the only thing you can think of is something bad has happened to someone you know! I have supported customers and users of my software in some way shape or form since January 1984. The streak really is 23 years.

I should say was 23 years – broken this weekend on Saturday morning at 2:01am.

Now I guess I should consider myself lucky, but the streak would still be alive if someone had done their job. I should have expected the call because of a conversation I had with the client earlier in the day Friday. His big concern with the latest conversion was that the end users were calling the Help Desk with requests for unsupported printers. I reported to him my concern is the onsite consultants and end users were not looking at the data we have converted for their review no less than 5 times before, including one last Tuesday. It was too quiet. I had repeatedly asked if the data looked good and got nothing but positive confirmation, but the signs were all there. You all know conversions are quirky, and every one of them is unique. This is the 21st different conversion I have run for this client and every one of them reveals something new. Data conversions are never smooth, never normal, and rarely clean, especially on the night they are processed for production.

Sure enough I get a call at 2:01am. I answer my phone on the second ring, which is real slow for me. Normally I get it on the half ring in the middle of the night, and definitely before the second ring starts. But I was dead tired and in the middle of an excellent dream that had nothing to do about coding or natural disasters. I asked who it was three times before I understood who was calling. Dead tired. Nothing like being greeted with “Hi Rick, the conversion you ran did not work. There is missing data and we think you pulled the wrong data or converted an older set of the files.”

Unlikely. Not that I am infallible (just ask my wife, kids, customers, friends, enemies, well – any human and even some non-humans), it is that I follow a process and the process has checks that make this nearly impossible. Things happen though, and I normally accept it as my problem. I am not a fan of finger pointing.

In the mean time while I was scrambling to determine what went wrong several emails went out to every important person on the planet for this project proclaiming my failure to process a clean conversion.

Come to find out there was data missing. Sure enough the review of the data showed certain records after January 11th did not get converted. But we ran a conversion on 1/30, strange? The fact is: if someone had reviewed Tuesday’s conversion the problem would have been recognized and the streak would be alive. The problem happened because someone did not follow the documented process and run a report that archived the data that we use in the conversion.

I really hate when that happens.

Not a single email was sent proclaiming the real reason for the failure. So my reputation takes a beating and the real troublemakers skate free without showing even a scratch. I am guessing they will get bonuses or some other recognition for finding the problem and getting the goofy Fox guy out of bed to fix his problem. Ah, the joys of being the hero. {g}

The good news is the conversion went into production on time, and I am on day two of my new streak.