Meltdowns at NSA spy data center destroy equipment, delay opening

October 9, 2013 in News by The Manimal

Source: ARS Technica

A massive data center being built by the National Security Agency to aid its surveillance operations has been hit by “10 meltdowns in the past 13 months” that “destroyed hundreds of thousands of dollars worth of machinery and delayed the center’s opening for a year,” the Wall Street Journalreported last night.

The first of four facilities at the Utah Data Center was originally scheduled to become operational in October 2012, according to project documents described by the Journal. But the electrical problems—described as arc fault failures or “a flash of lightning inside a 2-foot box”—led to explosions, failed circuits, and melted metal, the report states:

The first arc fault failure at the Utah plant was on Aug. 9, 2012, according to project documents. Since then, the center has had nine more failures, most recently on Sept. 25. Each incident caused as much as $100,000 in damage, according to a project official.

It took six months for investigators to determine the causes of two of the failures. In the months that followed, the contractors employed more than 30 independent experts that conducted 160 tests over 50,000 man-hours, according to project documents.

The 1 million square foot data center, filled with supercomputers and storage equipment to maintain surveillance information, is slated to cost $1.4 billion to construct. One project official told the Journalthat the NSA planned to start turning on some of the computers at the facility this week. “But without a reliable electrical system to run computers and keep them cool, the NSA’s global surveillance data systems can’t function,” the newspaper wrote.

Project officials are still trying to determine the cause of the meltdowns, and they disagree about whether proposed fixes will work. Backup generators have failed repeated tests, cooling systems “remain untested,” and “there are also disagreements among government officials and contractors over the adequacy of the electrical control systems.”

The Army Corps of Engineers is overseeing construction and promised to make sure the data center is “completely reliable” before allowing it to go online.