Recently, many of us within the mission critical industry got together in the Bay Area to support a fundraiser for the Boy Scouts, chaired by Bill Mazzetti. We had a great turnout with people from construction, engineering, design, electrical, mechanical, general contractors, components, hardware, and all manners of data centers from enterprise to colo and cloud. As the fundraiser involved sporting activities we took health and safety measures seriously and modeled our activities after the Boy Scout’s motto – be prepared.
That motto, ‘be prepared,’ has certainly been challenged recently by the unprecedented hurricane season of 2017, as well as global seismic, flooding, weather, and fire events. Data centers, networks, communication systems, industrial complexes of all manners, and the civil infrastructure that connects them and our communities together have all been put to the test.
There is no way to predict what region will be the next to experience record heat, 500-year flood levels, raging fires, or 7.0+ earthquakes, but that doesn’t mean we shouldn’t be prepared. Sure, we think our mission critical facilities are ready because we designed them for redundancy, extended power outages, 125+mph winds, etc. but have they really been tested? And, more importantly, have we taken steps to be prepared for the far too unpredictable elements mother nature can throw our way? The answer is often no we have not.
For the most part, when it comes to data centers or any other mission critical facility we can’t just pick it up and move it. We need to take steps now to assess our susceptibility to some heretofore unimaginable sequences of events that can find us isolated, vulnerable, and even out-of-business despite the best laid plans…
So what can we do to be better prepared? Start with the building, expand outward to the surrounding campus / real-estate, telecommunication networks, supply chain (fuel, water, people), out-of-band communications (for emergency use), and include all your personnel as well as contract support resources. Yes, that is a daunting task but you don’t want to find out that your employees and/or support personnel were either unable to reach the facility (civil infrastructure) or that did not want to leave their family. And let’s not forget the potential loss of concentration, thereby increased risk of error due to concern for those outside the confines of the facility when communication to the outside world is lost. Or worse, when the CNN report shows near Armageddon outside the confines of the data center.
This may not be what you want to hear but the easy steps are not all that expensive or difficult to manage. A good audit and multi-scenario analysis will identify the areas you need to bolster such as perimeter walls improved for flood protection, more pumps on more circuits with back-up, raised generators and fuel tanks, larger fuel tanks, more/larger water storage, more emergency food & supplies, bedding, clothing, etc., out-of-band communications, seismic shoring, exterior fire & smoke protection, and even a few canoes / boats for those in a flood plain. Do not forget to run a few test cases once you have your new resources, systems, and policies in place!
Testing the human and civil infrastructure elements is far more complex. The resolution can range from supplying inexpensive two-way radios to your personnel to integrating emergency preparedness strategies and solutions into key personnel’s homes as well as for your mission critical support personal and civil infrastructure. Perhaps as far as a public-private partnership to improve roadways, rights-of-way, flood control, emergency response infrastructure and systems, utility infrastructure, and training.
Although 2017 is far from over there are already a lot of lessons to be learned from the impacts of local, regional, and national disasters. These stories will continue coming forward and they will no doubt be emotionally charged as the bravery and heroics of our compatriots and first responders are more fully understood.
The data center as a mission critical facility, can serve as a test case for being better prepared. Planning now means being better able to handle catastrophic destruction of civilian and enterprise architecture by putting safeguards in place to minimize or even eliminate downtime.
Until then let’s put the Boy Scouts’ motto to practice and Be Prepared!