A Look inside Verizon's Flooded Communications Hub
By ANTON TROIANOVSKI
The Wall Street Journal
Kevin Hagen for The Wall Street Journal
Several floors of Verizon's headquarters are flooded at 140 West St. in Manhattan.
Eleven years after the 9/11 terrorist attacks, Verizon Communications Inc. VZ +0.78% is once again scrambling to repair severe damage to a key switching facility inside its historic headquarters building in lower Manhattan.
Verizon saw severe damage from flooding at 140 West, the same key telecom hub destroyed by the 9/11 attacks. Anton Troianovski on The News Hub has the story of what happened, and gives us a good read on the state of telecom in lower Manhattan. Photo: Reuters.
The massive facility for interconnecting key communications lines sustained heavy damage after planes struck the Twin Towers more than a decade ago. This time the enemy was water shoved ashore by Hurricane Sandy.
The building is one of the worst hit of a number of facilities that carriers were rushing to fix Wednesday. The Federal Communications Commission said the number of cellphone tower outages dropped on the second day after the storm made landfall, with just over a fifth of the sites in storm-affected areas in the Northeast offline, down from about a quarter on Tuesday.
Phone companies supplemented those efforts with extraordinary measures to bolster service. Wireless carriers AT&T Inc. T +0.46% and T-Mobile USA, for example, said they would switch each others' customers between their networks depending on which was in better shape in a particular area.
The steps came as cellphone users around the New York City area reported more trouble keeping and completing calls as businesses started to open back up and people tried to go about their workdays. The FCC said pockets of serious damage remain that will take a long time to repair.
The depth of the challenges facing telecom providers was on display inside Verizon's facilities in lower Manhattan.
Kevin Hagen for The Wall Street Journal
Verizon chief technology officer Tony Melone stands amidst efforts to remove water from the company's headquarters in Manhattan.
The company's headquarters—a key communications hub just north of the World Trade Center site at 140 West Street—was in a state of crisis not seen since the 9/11 attacks, which partially destroyed the building.
Mud still covered parts of the ornate lobby. Down below, 3½ floors of the building's five-level basement were still submerged, the brackish water sloshing up the walls of the stairwell.
Verizon employees said Monday night's storm surge was so powerful that it breached the protective plugs that surround cables coming into the building. As a result, water flooded the critical basement "cable vault" that takes in communications cables and directs them to switching gear upstairs, which wasn't damaged. On Wednesday, Verizon employees took a reporter inside the vault, illuminating the huge dark room with flashlights. The cables were still damp, and leaves littered the floor.
Kevin Hagen for The Wall Street Journal
Crews work to remove flood waters caused by Hurricane Sandy from Verizon's Manhattan office.
Thanks to changes made in the aftermath of 9/11, when some critical equipment was moved above ground level, Verizon was able to get parts of 140 West operating. Verizon's Dominic Veltri, who helped direct the building's recovery after the attacks and now oversees engineering, design and construction in critical Verizon facilities, had emergency generators brought on Tuesday and threaded the electrical cables through a second-story window, to restore some functionality to the building. Now he is working on installing additional temporary fuel tanks on the street to power more generators.
"Unfortunately we've been through this before," Mr. Veltri said.
Like the terrorist attacks in 2001, Hurricane Sandy has exposed one of New York City's most vulnerable points: the concentration of telecommunications infrastructure in Lower Manhattan. Carriers adjusted after those attacks by moving equipment to more secure points within buildings and ensuring their networks could keep running even if parts weren't functioning.
Yet it is clear in the wake of the storm that key facilities remain exposed to disruption by disasters—especially in areas like Manhattan that are dense with high-volume telecom users.
"You have no choice but to concentrate on where the need is," said Tony Melone, Verizon's chief technology officer.
Much of lower Manhattan is still without wireless or wireline service as wireless providers struggled to restore service, Thomas Gryta reports on digits. Photo: Getty Images.
At another Verizon switching facility downtown, on Broad Street, the situation was even worse. Water had flooded critical electrical equipment, leaving the entire building inoperable.
"There is nothing working here," said Mr. Melone. "Quite frankly, this is wider than the impacts of 9/11."
Telecom hubs like Verizon's flooded buildings downtown act as collecting points for phone and data traffic, serving as key "nodes" in the phone network. Mr. Melone said that area landline customers, cellphone users, and businesses like the New York Stock Exchange were served by Verizon's hubs downtown.
Verizon said it had built in enough redundancy to be able to operate its network in a limited way. Mr. Melone said it was still able to provide service to the stock exchange on Wednesday, and that cellphone service was improving.
Outside the Broad Street facility, water being pumped out of the basement was rushing down the street as Verizon executives, including Mr. Melone and Verizon CEO Lowell McAdam, oversaw the efforts of scores of workers. The entrance to the darkened building was held open with a sandbag.
At 140 West, Mr. Veltri and other Verizon officials had to take unusual measures. Mr. Veltri said no one expected the water to flood as much of the building as it did. The force of the storm surge was so strong that it bent the 86-year-old Art Deco skyscraper's steel and bronze doors. To pump out the water, workers stuck a pipe down an elevator shaft.
On Sept. 11, 2001, the destruction of 140 West St. severed key links in the country's telecom network. Some businesses were without landline phone connections for weeks. After the building was rebuilt, critical equipment such as backup generators were moved above ground.
But fuel tanks and pumps for the generators were left below ground and were rendered inoperable when Sandy's storm surge overwhelmed Verizon facilities downtown. Now, Mr. Veltri said, Verizon will look at new ways to improve the reliability of its infrastructure—including figuring out how to protect the fuel pumps at 140 West.
After 9/11, critics said the damage to 140 West showed the vulnerability of U.S. telecom networks in an industry dominated by a few players—and in which key facilities were concentrated inside a small number of buildings. Verizon and others said they have improved the redundancy of their networks, meaning calls can be routed around trouble spots even if some facilities are taken offline.
That redundancy has been put to the test this week. FCC Chairman Julius Genachowski said Wednesday that telecommunications were improving, but noted "serious outages" in the New York and New Jersey area and warned that "the crisis is not over."
Getting a precise fix on the damage to the region's telecom networks was difficult. The major wireless telecom WTT +1.65% providers have revealed few details of their service problems other than to acknowledge that issues exist and to say they are working to fix them. People moving in and out of affected areas can make it difficult to estimate the number of people without service, companies said.
Outside Verizon's headquarters Wednesday, workers were welding vertical metal panels set on top of a wooden platform. Mr. Veltri said they were preparing to install temporary fuel tanks to power generators—just as Verizon did at 140 West after 9/11.
—Thomas Gryta contributed to this article.