A certain statistic has it that every developer messes up thoroughly at least once in his career, that the earlier the better, and that those who haven’t made big blunders yet are laying the foundation for a horrendous error of enormous proportion. Some have dropped entire production databases (poor souls), others have leaked confidential client information by no direct fault of their own, and others yet have locked everyone out of a production server, including themselves. We may laugh to hear such stories, but we know the sheer dread and anticipation that is accompanied by such errors. I used to laugh too, until I made a production-grade error a few weeks ago.
I was on this project for many weeks and since the beginning, I was on a roll, baby. Every day above ground was a great day. The development was swift, the meetings were happy, and progress was good. I smiled at everyone in the office and could not understand why on earth some developers looked miserable. “Cheer up man.”, I would tell them complacently, almost condescendingly, and the looks I got back would have sent me running for the hills were I not having an amazing time in my project. Alas! Little did I know some little devil was lurking around, waiting to pounce on me.
This project of mine involves working a great deal of the time directly with databases. That in itself should have discouraged me from getting comfortable. The integrity of the data was such that I had to switch back and forth between the production and development databases to ensure that everything is fine to a tee. Yet at the same time, it was necessary to run clean up scripts before running each test lest the test results be inaccurate. I was in a particularly trigger-happy mood one fateful morning, when I forgot to switch back to the development database before running the clean up script. Oh, the horror! One statement executed and it returned slightly more affected records than I expected. I didn’t do anything. The next statement executed with a similar result. I frowned. This is weird. Yet I still did nothing. The third statement had executed slightly more than a minute when my stomach tightened, for the only way execution could have taken that long was if I were working on production data. Boy how a face can change when a heart knows fear!
I straight halted execution. Yet I had this sick feeling that I was done for. It was like crossing a muddy stream but realizing mid-step that what you thought was a solid rock is really a sponge and there is absolutely nothing in the whole wide world that you can do to stop what’s coming. I debated an hour or so whether to come clean or lay low and hope no one will notice. Yet I was not really debating, because the whole time I was a nervous wreck. I was sweating, I was panting, I was white with dread. My thoughts were racing so much I had to take a break. Outside, some developer friend of mine took one look at me and knew straight away I had fucked up. He smiled and said, “Cheer up man!” Some nerve!
By the time I came clean, the client had already realized something was not right. Goes to prove honesty is the best policy. The meetings that followed were among the worst meetings of my life. Every word from my teammates stabbed me to the quick. Every look from my boss shook me to my core. In all this I was far from innocent: it was all my fault. I had no defense, no excuse, no nothing. To speak bluntly, I was fucked.
Fortunately, the gods had not completely forsaken me. Automatic backups were set up, so I was not completely screwed. Before all this I knew vaguely about transactions but after that mistake I go to extra pains to use them even if they slow me down. I do not commit unless I explicitly must, and even then, I count to ten before clicking Execute. This hesitation, this thinking and overthinking, has sipped into my life, for my tea gets cold ere I get to ten. Yet it is a small price to pay. The client is happy, and I am saved from diarrhea. If that is not a “win-win”, I don’t know what is.
