Power Outage - Who Turned Out the Lights?
From a blown fuse to a lightning strike at an electrical substation, we've all experienced some sort of power outage before. In our homes, a blown fuse can easily be replaced or reset with little or no effect on our everyday lives. But a power outage at work, especially long-term, could cause technological disaster. Without a tested and proven recovery plan, the absence of electrical power could virtually shut down everything.
The following is a collection of lessons learned, gathered from several companies that experienced power outages which affected their computer / server rooms:
Before an Outage
Be sure to have action plans for responding to a power outage written into your Site Disaster Recovery Plan. These would include information on backup generators, Uninterruptible Power Supplies (UPS), recycle times, and contact procedures. Surge protectors and UPS should be in place on critical systems equipment. The sooner you notify the proper contacts within your company, the quicker the recovery can begin.
Even if your site has emergency lighting, flashlights with good batteries should be stored in designated locations known to key personnel. Flashlights should be kept both inside and outside the computer / server rooms. Emergency lighting cannot adequately illuminate the back of a server when you are checking a connection in the dark.
During an Outage
• Use flashlights instead of lighters, candles, and matches to work in darkened areas. Some UPS / battery systems can leak combustible fumes that could be ignited by a flame.
• Coordinate with local facilities and/or security personnel to determine the cause of the outage, and the expected return to service. Communicate any critical systems requirements to those people.
• Bring down servers, mainframes, and mini-frames gracefully according to system manual instructions; that is, if you are on-site or are notified in time to do the shutdowns.
When Power Is Restored
• Bring up systems one at a time (most critical ones first) to avoid flooding the power lines all at once. Test to ensure system software, databases, and application files come back up. Work with users to ensure that their hardware / printers are operable.
• Advise users of any lost data that they may need to re-enter.
• Update your Site Disaster Recovery Plan with lessons learned from the power outage. Notify the appropriate management representative of any unusual lessons, so they can be shared with other company entities.