Today was "one of those days". We had a major failure of one of our servers. This server operates our in house software and controls all order entry, pricing, and our website shopping cart system.At 7:30am the server does a day-end reboot to prepare for all new orders, and this time it just died! Darrin our controller and our IT Manager Frank were very surprised this happened and explained it to me like this.
Imagine flying in a plane and without any warning both engines just stop. That's essentially what happened to us - both hard drives failed at the same time. You see, we have spent a significant amount of money on our computer systems and have built in the best possible redundancies, back ups and even auxiliary power. So when a server has both hard drives fail simultaneously without any blips, hiccups or warnings it truly is a one-of-a-kind "bad luck" situation.
We scrambled to order in parts and repair the main server. That took a few hours and we were powering up our server. From there we needed to rebuild the server system and data from back ups. This took several more hours. What impressed me the most was that our contingency plans, back up data and technical help all ran smooth. I had always wondered - what if we had a fire, an earthquake, a major server failure, are we prepared?
Is your business prepared, or have you had to deal with the loss of all your critical data? Were you able to be up and running quickly? Maybe now is the time to do a test or dry run to see how you fare. We all rely so heavily on technology, how long can our companies survive without it? My policy is to spend money for the best and most current IT we can afford. Even then, as we experienced, there is no guarantee things won't go wrong.
I want to say a big THANK YOU to Darrin, Frank, David for their IT expertise and a special shout out to our sales staff and customers for their understanding and patience today.
Cheers,
Calvin
