Response Considerations for Data Centers

BY DANIEL J. NEAL, GHORLEY GREGG, AND PATTI COLLINS

Responding to large computer data centers is a relatively new concern for fire departments. These facilities maintain large server rooms, redundant power backups, large cooling systems, and secure access. Firefighters should be familiar with the common emergencies at these facilities and should also understand the importance of maintaining the operation of these facilities despite emergency incidents. Computer data centers have a vested interest in maintaining their business continuity, because the accidental shutdown of these facilities can have nationwide security and economic impact. Therefore, fire department personnel’s familiarity with these facilities and their operations is essential.

Loudoun County (VA) Fire, Rescue, and Emergency Management (LCFR), located in northern Virginia, protects a 525-square-mile area. The county is the home to several large data center complexes that handle large portions of Internet traffic, including America Online, credit card transactions, online companies, and Verizon.

Construction. Data centers are typically of Type I (fire resistive) or Type II (noncombustible) construction. Some of these buildings are reinforced to withstand winds of up to 200 miles per hour (mph). This construction accommodates the large server rooms, and these facilities typically also include heating and ventilation facilities, generator areas, battery backup rooms, and a network operating center (NOC).

Server rooms. The server rooms are the heart of a data center and consist of large computer servers that accommodate large volumes of computer transactions and traffic. Since the computer equipment in these rooms creates large amounts of heat, raised floors and large-capacity cooling systems keep the rooms cool. Ventilation and cabling are placed under the raised flooring, which is often elevated as high as 36 inches. This poses significant potential firefighter hazards, such as hidden fire and entanglement (photo 1).

Click to Enlarge
(1) Photos by Patti Collins.

The fire detection systems in these server rooms emphasize the importance of maintaining operations. Highly sensitive detectors placed throughout the ceiling and under the raised flooring will detect any particles from combustion immediately and provide immediate notification of any fire in a server room. Often, these detection systems are linked to the building’s engineers by text messaging or pager. Thus, building engineers are often notified of an emergency before the fire department is summoned.

Fire alarm kiosk. To further protect these server rooms, each room often has one or two fire alarm kiosks immediately inside the entrance that support many important fire suppression functions. An annunciator panel displays the location of any malfunctioning or activated detector and often includes a schematic of the server room layout for quickly locating a detector. These kiosks also house a suction-cup floor tile puller to access through the raised floor (photo 2).

Click to Enlarge
(2)

The kiosks may also house an emergency power-off (EPO) button. The EPO will immediately shut down all power to the server room and is often housed behind a clear protective housing with a hinged access hatch, which may also be monitored for security. Pushing the EPO has a tremendous impact aside from shutting down power to the server room. During a familiarization tour, one building manager warned our company, “If you press the EPO, millions of dollars could be lost.” Consider activating the EPO only if there is an immediate threat to life.

For other emergencies, the facility can often switch traffic to other server rooms before power needs to be shut down. When conducting facility familiarization tours, remind your firefighters to be respectful and considerate, especially around the EPOs. During one of these walkthroughs, I observed a firefighter simulating pushing the EPO. Understandably, the building engineer did not like it.

Typically, to protect the server rooms, a preaction system requires the activation of two detectors. This prevents an accidental sprinkler discharge, which could severely damage the servers. The sprinkler risers may be in plain view in a caged area in a main hallway outside the server room, which allows ready access to the riser (photo 3).

Click to Enlarge
(3)

Backup power supplies. Maintaining constant data center operations requires redundant power systems. The local utility provides the main power to the data center; if the municipal power is interrupted, the data center must switch to backup power through a series of steps to maintain an uninterrupted power supply (UPS) system. Initially, backup batteries provide backup power while the facility’s generators start up. In data center language, this is called “ride-through.” These battery rooms often consist of rows of large lead-acid batteries. It is not uncommon for these battery rooms to consist of scores (or even hundreds) of backup batteries on six-foot-high racks.

A data center’s diesel-fueled generators are massive, have tremendous capacity to ensure uninterrupted data center operation, and often are programmed to rotate generator operation to supply power. A data center with eight generators may be able to power the entire facility by operating only two or three generators. Since the power rotates among the generators, be aware that these generators can start up at any time, posing a danger to a firefighter and a patient in close proximity to secured electrical equipment or the generator itself (photo 4). Since uninterrupted power is an essential data center need, arbitrarily shutting down a generator may not interrupt power. This may only cause the center’s backup power systems (i.e., batteries and additional generators) to activate, thus maintaining electrical equipment.

Click to Enlarge
(4)

Diesel tanks for fueling the generators are common; they may be underground, aboveground, or even built in under the generators themselves. The sizes of such tanks vary significantly, depending on the size of the facility and the number and sizes of its generators. It is important to know the location of these diesel tanks. Fuel may be piped from an outside storage tank through a wall into the building.

Knowing where the fuel shutoff is located is important if there is a leak. One large data center had a control valve for the diesel storage area that isolated the fuel tank from the generator. Valves also exist to isolate the runoff from the diesel tanks and generator areas. Whenever a diesel truck replenishes the tanks, this valve is closed. In the case of a leak, the spilled fuel is automatically isolated from entering the normal rainwater runoff.

Cooling systems. Another important characteristic of a data center is its extensive cooling system (photo 5). The servers generate a large amount of heat, for which massive chillers provide cooling, which requires a significant amount of refrigerant. These cooling system rooms or their entrances may be equipped with refrigerant leak warning alarms. One data center boasts an extensive alerting system if a refrigerant leak develops. A light tree with blue, yellow, and red sectors is mounted above the door to the chiller room (photo 6). The blue light indicates a refrigerant leak of 50 parts per million (ppm); yellow, 150 ppm; and red, 300 ppm or more. If the leak activates the red light, it triggers an automatic exhaust fan. The exhaust systems can also be started remotely to remove any leaked refrigerant.

Click to Enlarge
(5)

Click to Enlarge

(6)

Water tanks. These cooling systems also require a large amount of water to operate. Data centers also maintain large water tanks behind their facilities in case of a municipal water supply interruption. These tanks can store up to several hundred thousand gallons (photo 7). They may also have a hydrant or fire department connection attached to them. However, this hydrant is merely a drain fed by head pressure and is not a routine source of water. In the case of fire, use this water only as a last resort. The fire department may advise the facility manager to paint this hydrant a distinct color (e.g., blue) or place a sign indicating that the water is not for routine fire department use (photo 8). Remember, however, that if there is a fire and there is no other source of water, use this water storage tank.

Click to Enlarge
(7)
Click to Enlarge
(8)

Converted-use buildings. Not all data centers exist in new construction. Many data centers may occupy buildings that have been converted to their use. These centers may have evolved over years as data centers and switching facilities. It is not uncommon for these centers to exist in multistory buildings. Although there may have been a “change of use,” these buildings still require the operational systems of their newer relatives. To accommodate cabling and other connections, there may be large holes between floors. Some of these facilities that serve as large connection hubs can have miles of wiring that were installed over a period of years. This plastic-coated wiring creates a significant fire load, not to mention the years it may take to rewire these networks. Backup generators may be installed with diesel storage in unexpected locations that may not be code compliant. It is most critical for the local fire department to conduct a preincident survey in these converted-use buildings.

 

RESPONSE GUIDELINES

 

There are several guidelines to consider when responding to a data center. A fire department’s first trip to a data center should not be on an emergency response. These facilities often have tight security. Fire company officers should set up facility familiarization tours, during which firefighters should meet the building engineers and security personnel. Department personnel should explain the fire department’s response to their facility and ask facility personnel about their emergency response plans. Facility personnel are essential when responding to any emergency and are well versed in their facility’s operation. You must coordinate any decisions regarding these building systems with the fire department and the building engineers. Fire department/facility manager coordination, a detailed preincident survey, and regular tours make for an effective response to a data center.

When responding to data centers, firefighters should be prepared for the most common emergencies. With business continuity a top priority, the most common emergencies are readily managed.

Emergency medical incidents are still the most frequent emergencies. Personnel in these facilities may be “essential” personnel and may not be willing to leave until their replacements arrive.

Data center fire alarms may be frequent. The facility’s policy may be to immediately investigate the alarm but to continue the fire department’s response. The on-duty building engineer may already have investigated the alarm prior to the fire department’s arrival. A typical alarm may be the result of detectors that need replacement. The building engineer may want to confirm this determination with the fire department before resetting the alarm system.

A third common emergency is an overheating fan in the air handler in a server room. In one case, a fire alarm initiated the fire department’s response. On arrival, the fire personnel found that data center personnel had already located the problem, secured the air handler’s power, and summoned someone to repair it. Hence, this incident did not require the server room to be shut down. Although the electrical equipment produced a significant amount of smoke, the problem was readily contained. The operation of the servers in the room continued without interruption.

In the rare case of fire in a data center, locating and accessing the fire are critical. The centers have limited access because of security concerns. Additionally, these centers cover large areas and will have long central hallways. This will require making up very long hoselines or locating auxiliary access (photo 9).

Click to Enlarge
(9)

Responding to data centers can be intimidating for firefighters unfamiliar with their operation. However, firefighters can become well versed with their operations and hazards through facility familiarization. Also important, they can get to know the facility’s building engineers and security personnel, which aids the fire department in responding effectively while supporting the data center’s business continuity.

DANIEL J. NEAL is a captain with the Loudoun County (VA) Department of Fire, Rescue, and Emergency Management, assigned to Fire Station 6. He has served with the department for 11 years. He has a master’s degree in emergency health services and is a graduate of the National Fire Academy’s Executive Fire Officer Program.

GHORLEY GREGG is a lieutenant with the Loudoun County (VA) Department of Fire, Rescue, and Emergency Management, assigned to Tower 6. He has served with the department for seven years.

PATTI COLLINS is a firefighter with the Loudoun County (VA) Department of Fire, Rescue, and Emergency Management, assigned to Fire Station 23. She has served with the department for 15 years.

 

More Fire Engineering Issue Articles

 

 

Fire Engineering Archives

 

Hand entrapped in rope gripper

Elevator Rescue: Rope Gripper Entrapment

Mike Dragonetti discusses operating safely while around a Rope Gripper and two methods of mitigating an entrapment situation.
Delta explosion

Two Workers Killed, Another Injured in Explosion at Atlanta Delta Air Lines Facility

Two workers were killed and another seriously injured in an explosion Tuesday at a Delta Air Lines maintenance facility near the Atlanta airport.