this post was submitted on 08 Feb 2024
1 points (100.0% liked)
Very Real Tech Pics
1286 readers
1 users here now
There's no place like 127.0.0.1
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
It's all fun and games until the halon gets triggered
First day working on a data center:
"You see that red light next to the door? If that turns on and you hear a horn, you have 45 seconds to leave the room or you will die. Also, if it goes off, you must not for any reason whatsoever try to enter the data center, actually just leave the whole building at that point."
Scariest caveat to a security brief. Funny story, that data center actually burned down while I worked there. No, I didn't do it, I wasn't even on the building at the time. But I was part of the cleanup crew. I can still feel the smell of burnt plastic if I try real hard to remember.
Oh yeah, Halon will fuck you right up.
If anyone happens to end up in a datacenter and the alarm goes off, leave the building.... Just go right outside. Do not stop and collect tools, do not pass go. Do not collect $200.
There are many centers that don't use Halon anymore and it's "safe" to stick around, but heck, if that alarm goes off, I'd still get out. Nothing I'm in there to do is more important than my continued survival. If it's actually safe, I'll get delayed by what? An hour? Two? Maybe a day? Who cares.
$200? Back in my days it was 4000. ;)
The fuck happened?
An investigation concluded that a power line next to a vertical ventilation overheated, melted the wires and created a short circuit electrical arc. The power properly went off but a flammable outer lining in the ducts caught fire. The insulation correctly slowed down the fire, but the plastic on the wires caught fire. The wires went up into the data center and carried the flames inside through the ducts. All in all, the fire preventions worked to some extent but still several million dollars in servers got ruined and rendered inoperative. A whole rack burned until the halon triggered and mostly snuffed the fire, but the furniture in an office below, adjacent to the duct lines burned into a crisp further damaging the data center with smoke and heat.
It was the most surreal week on that job, taking all the equipment to the dining hall building next door, tearing everything apart, cleaning and testing every component to separate what was damaged beyond repair and what could still work at least as contingency. The funny thing is that the backup servers where in an office next to the one That burned down so the service was off for good. They got lucky and the tape archive was properly sealed from the fire, it was inside the data center. But we still had to verify each of the tens of thousands of tapes for data corruption.
I guess the moral of the story is, off site backup means more than 20 meters away.
Add: oh I forgot to mention. The reason you will certainly die in a data center fire is not just the oxygen displacing gas. The doors are designed to create an airtight magnetical seal and when the fire suppression triggers it locks the doors permanently. It needs power to break the seal and open. There's a big red emergency button next to the door, but in case of a fire it just buys you an extra 30 seconds to scram out of there before the system rearms the gas release. Miss the window and you're trapped inside in an oxygen depleted environment. Firefighters had to tore the entire door frame off the concrete to access the data center.