Holiday IT Horror Stories
For many system administrators, working in IT has a host of tangible benefits: Good pay, access to a never-ending stream of new and updated technologies, camaraderie among IT professionals, the chance to solve complex problems, and the opportunity to fundamentally impact how efficiently businesses and organizations can function.
On the downside, being a system administrator can sometimes require long hours at the office or being on call during vacations and holidays. Now that we’re in the midst of the holiday season, I recently asked Petri IT Knowledgebase readers to submit their holiday IT horror stories, tales of when they had to solve an IT-related problem at the office — even while on holidays or vacations.
More than two dozen of you replied to my call for IT tales of woe (and some that were just adventurous), and we’ve included excerpts of some of them here. If you’d like to submit your own IT horror story, please drop me an email or add a comment to this blog post.
Say Goodbye to Traditional PC Lifecycle Management
Traditional IT tools, including Microsoft SCCM, Ghost Solution Suite, and KACE, often require considerable custom configurations by T3 technicians (an expensive and often elusive IT resource) to enable management of a hybrid onsite + remote workforce. In many cases, even with the best resources, organizations are finding that these on-premise tools simply cannot support remote endpoints consistently and reliably due to infrastructure limitations.
IT work ain’t always pretty — especially around the holidays.
Bad Santa Visits the IT Department
Many years ago I worked for a large corporation here in the US who shall remain nameless. (They used static IP addresses – it was hell!) Myself and another admin found ourselves working Christmas Eve doing patching, upgrades, etc. and were literally the only two people in the building… so we did as all good techs do when doing this kind of work: We broke out a projector and watched a movie, because, let’s face it, a lot of this work is just hurry-up-and-wait. We picked for our holiday patching endeavors the movie Bad Santa. It just so happened that the Sr. VP of our department came to check up on us to make sure we were actually there and working and saw us watching this movie. Needless to say we got in some serious trouble for that one, despite the fact that we had to come in on Christmas Eve — technically a day off for us — and we were salaried and were the only two in the building. This is what we get for “volunteering.” Needless to say I left that job shortly thereafter for the position I have now. – Brian Dower
The Bank Holiday
It was a Friday in Ireland of a bank holiday weekend (which means that we are off work on the Monday). On Friday evening I began to do an update recommended by Microsoft for Windows 2000. I applied it at 4pm as users logged off, then proceeded to the server room… I noticed that the Citrix server had a funny message on its LCD panel on the hardware. Upon investigating, I realized that the Windows OS patch had overwritten or corrupted a driver that the server needed and it now had a major fault. After spending the entire night surfing the web, and the Saturday morning talking to Dell customer support, the advice was to rebuild from scratch, with new firmware, operating system, the works. So I backed up user data and got up to the bit where Citrix can be installed. Then I hit a snag: The Citrix licensing and support site was down for maintenance until 5 am Monday morning (my time), and I needed to get particular updates and licenses. I saw my plans of a romantic getaway with the missus — which looked unlikely Friday night — completely shot. I came in early on Monday and had the server up at 7 am, just as users in the UK (who didn’t have a bank holiday) started to log in.
The upshot was that noone noticed, as there was no downtime from the users’ point of view — and the missus got a big bunch of flowers to say ‘I’m sorry.’ I now will NEVER EVER plan any sort of migration/install/patch on the Friday of a Bank Holiday weekend! — Ger Dunne
A Burning Issue
A couple of years back I was administrator for a government agency in Victoria, Australia. It was bushfire season and I did all the account and access setups for Mobile Incident Control Centres (ICC) and their staff. ICCs are set up at the nearest safe point to the fire and the local fire fighters (usually volunteers) assist. A third of Victoria was on fire – it was a really bad season. All the fires at that time had fully functioning ICCs.
On Friday night, I’d flown to Brisbane (about 3000 kms away) for a wedding. On Saturday, I was sitting in brilliant sunshine, with my legs in the swimming pool, having a long cool drink and thinking about getting dressed in the wedding clothes. My personal mobile rang: The person on call needed to setup 50 new fire fighters for a new ICC – he had never set up anyone in Novell Netware and couldn’t contact our other team members.
So there was I stepping him through all the setups from memory (no computer at the rental property we were all staying). I got him to take screenshots of each step and put them into a word document, so he could follow them again for the next 49. He successfully setup the Mobile ICC and lives and property were saved. That was the best and most rewarding support call I’d even taken. I wish I could have the pleasure of doing all support while lazing back at a swimming pool! — Vivieene Laverick
The Christmas Eve Cooldown
A few years back, the giant AC unit in our server room went on the fritz. It was around 4pm on Christmas Eve. I was supposed to be leaving work for my annual family Christmas Eve dinner. It was a tradition for a few decades where all extended family gathered to celebrate at my aunt’s house. Now I was missing it!
Instead of going I was staring angrily at about 70 servers shutting down from heat triggered events. We put in an emergency call and had the server room door and building door wide open, trying to cool everything off.
Thankfully, by 8 pm cooling had been restored and I was able to leave and begin my holiday celebration! One of the guys I worked with hung in there to power on all of the servers and insure all services were restored. Maybe he should be writing you an e-mail too!! — Bob Zabaga
On New Years Eve of the millennium (Dec. 32, 1999) I was working for an online bank which was located on the 15th floor of a downtown building. I was the primary server support person. We had spent weeks if not months preparing for “Y2K” but the risk management people had other ideas. In order to say that “we had done everything humanly possible,” I was instructed to physically monitor our servers. Physically. That meant sitting in the 10 x 15 room where we kept the servers while midnight rolled past in each ConUS timezone. Holding a flashlight. – Marty Bogen
What I Did During My Christmas (Sewer Line) Break
“About 5 years ago, I was working for a local university. Just before the Xmas break a main sewer line which ran from a large men’s bathroom to the street sewer hookup had collapsed. This translated into the need to dig up and replace the drain line before classes resumed in January. The work was going to involve digging through the floors in several classrooms, and the campus bookstore had to be closed because the line ran under the middle of the main display area.
Naturally, no one thought that it would be necessary to inform anyone in the IT department — or the book store manager for that matter. So we only began to learn of this “holiday break project” when the plumbers arrived onsite, two days prior to the commencement of the winter break, with equipment to dig up the floors in the classrooms and bookstore.
What was particularly irksome was the attitude of the plumbers who were gleeful about any sort of delay. When they were asked if they could at least give us a day to clear out and secure the computers they informed us that a delay was certainly fine with them. They pointed out that they “would get paid for watching us work – and probably earn more per hour than we did.” Nice guys. Great sense of humor.
Bottom line: Our Christmas “break” largely was burned up relocating a couple of classroom’s worth of computers to a secure location. Plus we needed to move all of the bookstore workstations, rewire some network connections in the book store, then put everything back, test connections, printers, etc. – all during the campus “down time.” Bah Humbug. – Curtis Christensen
Pass the Turkey, Reboot the Server
One Thanksgiving we were just arriving at friend’s house for a day of feasting when I got a call from my coworker. We had done some upgrades to a huge monster of a Unisys server the day before and now the server was down. I was in charge, so I didn’t want to make my minion work on this day, and being nice means it’s all on you. I dropped my spouse off at the friends’ place and drove to the office, where I spent the rest of the day and most of the evening trying to recover the server. At this point I don’t even recall the specifics of the crash, but I do recall missing turkey and dressing! Bummer. Since that event I try to avoid holiday upgrades; however, I do spend lots of late Saturdays working on stuff that is in production and cannot be down during the week. – Penny Rand
In addition to all of the stories printed here, I’d like to thank all the other Petri IT Knowledgbase readers that submitted their holiday IT horror stories that we didn’t have space to publish. Thank you for your submissions!
Have any IT holiday horror stories of your own to share? If so, drop me an email with some of your favorites.