Gerry Smith
Posted: 10/31/2012 8:03 pm EDT Updated: 11/01/2012 9:56 am EDT
They call themselves “the bucket brigade.”
In a lower Manhattan building evacuated during Hurricane Sandy, more than a dozen people on Wednesday carried 5-gallon buckets filled with diesel fuel up 17 flights of stairs. Each of them carried two buckets -- one in each arm -- up two flights of stairs, then handed them off to the next person. The stairwell was pitch dark, slippery and reeked of diesel.
The diesel was powering a backup generator on the roof that fueled a data center run by Peer 1 Hosting. Inside, rows and rows of Peer 1's servers provide the computing power for about 100 companies like Squarespace, a Web publishing software business, and Fog Creek Software, a company that develops tools for the software industry. Those companies rely on Peer 1's data center to enable thousands of customers around the world to run their websites.
Now, to keep their customers online, these high-tech companies were relying on a very low-tech feat: carrying buckets of gasoline up stairs and pouring the fuel into a generator. The stakes were high: If the rooftop generator ran out of fuel, the servers would fail, and thousands of websites would go dark.
"This is a precarious situation," said Anthony Casalena, founder of Squarespace, who was helping carry diesel fuel up the stairs on Wednesday. "People have really come together here to keep things online."
The potential for disaster had already played out at another data center this week in lower Manhattan. On Monday, Hurricane Sandy flooded a data center run by Datagram, taking down several major websites -- including The Huffington Post, Buzzfeed and Gawker -- that depended on them to run their businesses.
Employees at Peer 1, Squarespace and Fog Creek were determined not to let that happen to them.
The trouble began after ConEd cut off power to the building at Peer 1's data center at 75 Broad Street. The company had generators in the basement, but the storm flooded the basement. Peer 1's last backup generator was stored on the 17th floor.
As Peer 1 began notifying customers that its facility may lose power, employees from Squarespace and Fog Creek arrived to see if they could help. The only way to power the rooftop generator -- and keep the data center running -- was to carry diesel fuel up the stairs. Thus began “the bucket brigade.”
The diesel-filled jugs were heavy, and hauling them up the stairs was physically draining, said Babak Ghahremanpour, a software developer at Fog Creek.
“Especially because we’re not all that fit,” he said.
They carried the buckets up and down the stairs from Tuesday afternoon until 1 a.m. Wednesday morning. Several employees of Peer 1 slept on cots at the data center, where air conditioners were blowing cool air on the servers to prevent them from overheating. Ghahremanpour said he only slept three hours Tuesday night while trying to keep the data center running. He said he was staying awake by drinking coffee.
“Do I smell like diesel?” he asked.
Once the buckets made it to the roof, their carriers poured the fuel into large green barrels, then piped the fuel into a generator that holds about 200 gallons of diesel. When the fuel in the generator got low, it set off an alarm in Peer 1's office on the second floor to alert employees to restart the bucket brigade.
By 2 p.m. Wednesday, Peer 1 estimated that it had eight hours of diesel left to keep the data center running. But Ghahremanpour was worried. At least one million customers use one of Fog Creek's products, Trello, which is a tool that makes it easier for developers to collaborate on projects.
He said Fog Creek was considering moving its 65 servers to another data center in midtown Manhattan that still had electricity.
“That’s a last resort,” he said. “I just don’t know how long we can keep doing this.”
Jesse Hertzberg, a senior vice president at Squarespace, stood on the roof of the building and watched the diesel fuel being pumped into the generator. Hertzberg said Squarespace was updating its customers about the conditions at the data center "so they'll know when their websites go down."
"But hopefully that won't happen," he said. "We've already made it 28 hours longer than we thought we would."
The company sent Twitter updates about its efforts to keep the data center running, which generated praise from customers and even those who had never used Squarespace before. One Twitter use wrote: "@squarespace I may not currently be a customer, but the lengths you're going to try to keep customer sites up makes me wish I was. Good luck."
*************************
The Huffington Post is eager for insights from our community, especially people with experience in power, infrastructure and engineering, on the adequacy of emergency preparation in advance of Hurricane Sandy, and the degree to which past disasters have informed adequate planning and construction. Please send a note to sandytips@huffingtonpost.com with insights and suggestions for the important questions that need to be asked of relevant private sector and government officials, and point us toward stories that need to be pursued.