r/talesfromtechsupport • u/kenf22 • 19d ago
Medium The fire alarm story
As promised in my first post, here it is. This one was not my call, but my friend's. I was there and he told me as it was happening and I heard the follow up from the manager as well. My other friend referenced it last night, coincidentally.
On a normal day, expecting a normal call, my friend answers. Before saying hello, or introducing himself, the customer just says "I need the name of the tech that was just onsite"... Well this can't be good. Despite being new, my friend handled it well. He explained that he can see the request for a tech, but not details like names. He can, however, get our manager to reach out and speak to the subcontractor. I'm not sure exactly how he said it, but the customer took a breath.
"While I get the manager, can I ask what happened so I can get this recorded?" Apparently, the tech came out, was let into the datacenter, and left to do the work (this is a theme with these stories). He did the work fine, and solved this issue without a problem as far as I know. He put everything back together, cleaned up, and headed out.
When leaving, he pushes the door and it does not move. In his head, presumably... "Oh, there is a button here. This must be the exit button"... NOPE. Fire alarm... The entire DC power is cut, alarms blare, emergency lights come on. What does our intrepid tech do? He tries pulling. Door opens. Cool. Time to admit your mistake and limit the damage? NOPE. GTFO!
Now, our understandably irked admin has to run in and search for the fire. Finding none he reviews the cameras and sees what really happened. He starts power on procedures and gets things up again and then makes the call.
Well, this is a problem for us. This guy just cost the DC downtime, which has SLA violations, and I don't know how much money. Leaving made it much worse since they had to search for a while instead of just starting things back up immediately. Now legal needs to be involved.
Manager does not provide the name, obviously, but does call the subcontractor to get their side. Funny, tech has not reported any issues ... Awesome.
Lawyers, fun... I don't know exactly what was said, but this is the update I heard after all that:
Yes, he hit the button, but...
The button was unprotected (no cover)
The button was not labeled
The button was not red, but Blue!
So, we have an unlabeled, blue button next to a door that did not open while he had his hands full or tools/parts. Do you really want to bring this to a judge?
Now, I have no idea if it did go to a judge, or there was a settlement, or if they just dropped it. Moral of the story, use proper, labeled emergency buttons, and stop trusting techs to know their way around your datacenter.
163
u/Speciesunkn0wn 19d ago
...I would have thought the button was a wheelchair access button. Did they have their wiring fucked up or something???
67
u/Distribution-Radiant 19d ago
Unfortunately DCs aren't really wheelchair friendly. The last one I worked in, you had to badge in, walk in a few feet, then badge in AGAIN at a 2nd door. There was no way to get that 2nd door open for someone in a chair; just not enough room for it to swing open.
And the server racks are usually 7-8 ft tall.
BUT blue is usually an exit button in the ones I've been in. The emergency shutdown has had a cover and been clearly labeled in all of them. Some also had a halon release button; those were also covered and clearly labeled. They were usually red or yellow.
95
u/ozzie286 19d ago edited 18d ago
I am a printer tech. I just worked on a printer in a room where a badge was required to get in, but getting out just requires hitting a button on the wall. A red button.
EDIT: and again today! I never realized how common this setup actually was. At least this one said "push to exit" on the wall plate, yesterday's was just a random red button on the wall.
73
u/Tieraslin 19d ago
I have a friend who worked as an insurance adjuster.
One day he got a call from the client to go to their site to assess an incident. When he asked for the details, he was told that it was best that it be discussed on site.
He agreed, and went to the site. The manager told him that they were hoping to claim for their fire suppression system (argon) discharge.
My friend asked if there was had been a fire incident, and the manager responded with, "No..."
Manager took him into the control room where there were two security guards present, one senior, one junior.
The manager explained that the junior had just been assigned to work in the control room and the senior had been showing him how everything works, i.e. "Press this for xxx", "push this for yyy".
When he came to the manual fire suppression button he pointed out that the button had a flip perspex cover in order to avoid accidental triggering.
"So if you need set off the fire suppression system, you do this", senior proceeds to flip up the perspex cover, "and then you do this", senior then proceeds to push the now uncovered button. The junior guard stared at what happened whilst the senior was left with an "Oh shit!" look on his face.
The fire suppression system discharged its contents, and they were hoping to claim back a five-digit sum to refill the supply from an insurance payout.
The insurance claim was denied.
41
u/arcimbo1do 19d ago
Are we certain the guy realized the power cut was a consequence of him pressing the button? If the button clearly doesn't look like the emergency button he might have thought that a fire started right while he was trying to exit, and that would explain why the door would not open with the button...
14
u/Cyborg_Ninja_Cat 19d ago
If he did think there was an alarm (fire or drill or false alarm) and an evacuation, just running away would be even worse. In a real fire it could put lives in danger searching for him. Not to mention, he'd have nothing to fear because he wouldn't imagine he'd done anything wrong, so why would he want to disappear?
The guy panicked, that's all.
25
12
u/Wise_Use1012 19d ago
Oh yes not leaving a burning building is a excellent idea.
10
u/Cyborg_Ninja_Cat 18d ago
You leave the building. Not the entire site. Have you never done a fire drill in your entire life?
3
u/Wise_Use1012 18d ago
Gotta go get other jobs complete. Unless they going to spend money paying him to wait around while they get their shit together. Have you never worked a job in your entire life?
0
u/Loading_M_ 18d ago
It's an emergency. The fire alarm went off - you have to assume the building is on fire until proven otherwise. This takes priority over your next job.
6
u/Beginning-Painter307 18d ago
if they had bothered to tell him where the muster point was then they would have told him the blue handicap button next to the door is actually the fire alarm
1
u/ozzie286 16d ago
Priorities:
Get out of burning building
Get to next job
[...]
Trim toenails
Learn fire alarm procedures of every site you go to on the off chance an alarm happens while you're there
21
u/Wise_Use1012 19d ago
Dude probably didn’t even know he was the cause of the alarm. Probably just heard the fire alarm and like most people will get out of the building. Not his fault they can’t figure out how to properly label shit.
35
u/MindlessPhilosoper 19d ago
There's fault on both sides. The tech shouldn't have just bolted. The data center should have clearly marked and protected the E-stop.
31
u/ozzie286 19d ago
"It's so weird that the fire alarm started going off as soon as I hit this button, but it unlocked the door, so it must have been the door unlock button"
21
u/Distribution-Radiant 19d ago edited 19d ago
Full agree. Every DC I've been in had 2 clearly marked buttons by the door (one to exit, one emergency shutdown), and a cover over the shutdown button that you had to lift.
And the emergency shutdown button was always red, as they should be. Exit button was usually closest to the door.
At least they didn't have a halon system tied to it. That could get ugly real quick. You only see that in pretty old DCs though - the US banned production in the late 80s, though there's still a lot of it out there.
11
u/Dougally 19d ago
One DC at a place I used to work for had the fire alarm go off. It dumped CO2 on the equipment. Ended up with unrecoverable snap frozen hard drives and days of data recovery from backups. Some data was lost.
3
u/Distribution-Radiant 19d ago edited 19d ago
Yikes. That shit should only go off from heat IMO, like regular fire sprinklers. Them going off should also trip the fire alarm.
Regular ones have a glass bulb stopper - you can tell what temp they trip at by the color in the bulb. The most common (red) trip at 155F, and they make versions for CO2 as well. And there's a flow meter somewhere that sets off the building fire alarm.
5
45
u/cerrera 19d ago
The problem is not that he pushed the button. (Well, that's a problem, but not THE problem.) The problem is that he left. That's what I'd bring in front of a judge.
54
u/ThatUsrnameIsAlready 19d ago
He thought it was a door release button, and being unlabelled might not have even realised that pushing it caused the alarm.
Alarms typically mean GTFO, and he was already leaving anyway.
I doubt this DC ran this third party contractor through on site emergency protocol either, like what the blue button does, or where to report/muster.
37
u/handlebartender 19d ago
There's also the "well that's a weird coincidence" factor, where he may not have connected the cause of him pressing the "exit button" with the effect of the alarms going off. Other than to think "wow, good timing, what with me heading out now", maybe.
13
u/TheBasilisker 19d ago
If a singles location estop will lend you to mess up SLAs your SLA where dookie to begin with. If it was important enough you should have a fail over location that picks up in exactly such a situation.
6
22
u/techtornado 19d ago
What country is this because that is a strange choice to put the blue “exit button” by the door…
In the few DC’s I’ve been in, there are always posted instructions on the red fire EPO along with the yellow gas discharge arm/disarm button
10
u/soupie62 19d ago
Place I worked had a security door - swipe in, swipe out.
Emergency exit button had glass over it.
Rather than break the glass (and need to replace, write incident report etc) someone slid out the glass.
The new report they had to write, and the security investigation, were significant.
9
u/Matir 19d ago
A university I worked for did have a request to exit button for the door within a foot of the EPO (emergency power off) button. The EPO button was not covered. Somehow I never hit the wrong one, even when trying to hit the exit button with my hip because my hands were full. A while after I left I heard they finally covered the EPO button.... after the inevitable happened.
8
u/MountainMark 19d ago
We had a vendor on the floor and, walking out, he hit the recessed, labeled, bright red EPO button. Dropped the entire DC.
He did it just for the hell of it.
4
u/ThunderDwn 19d ago
And were there sufficiently painful consequences to this individual? Enquiring mind wants to know.
1
u/MountainMark 18d ago
I don't remember the details well. I know he was blackballed as an individual. I don't remember if the entire company was. This was like 25 years ago in the dot-com boom.
9
u/dickcheney600 18d ago
An unlabeled fire alarm that's not even red? That's literally dangerous in a completely different sense! Like if there really is a fire, you need to find the alarm RIGHT FUCKING NOW! The "installer" is lucky no one was hurt from a real incident and inability to find the button, or they'd be behind bars by now. (Particularly one that's an EPO as well, like if there was someone getting zapped, again, you must find it NOW or they're dead)
6
5
u/ac8jo 18d ago
The button was not red, but Blue!
I worked at an office and both doors had a red "PUSH TO EXIT" button that would unlock the door. They also had a light on them because someone (smartly) put a motion detector above the doors that would unlock the door. So even red isn't safe for this.
5
u/Ha-Funny-Boy 18d ago
One place I was working for built a new data center. The person that designed it was a company employee and bit of an idiot/ass-hole. He put an emergency power off button right above a water fountain. Shortly after going live he was in the room and needed a drink. Goes to the fountain, leans over and braces himself on the button. Guess what happened!
3
u/thelittlestmouse 18d ago
Haha, I might be remembering the same incident if it happened in PA about 20 years ago. Otherwise there's a second company with an unlabeled, wrong color switch by the door to the data center that was triggered by a contractor. Ours might have been an electrical contractor though, hard to remember.
4
u/gorgeousaurus 18d ago
I used to work in a lab that had an emergency power stop button right by the exit door that wouldn't kill the lights, just all the wall outlets. Which meant any equipment plugged in would die and usually kill whatever experiment it was running. The placement and design of the button (circular red button) was pretty much identical to the one most hospitals around here have that you have to push to unlock the door to leave. At least a couple times a year a visiting repair person who also works in hospitals would push it without thinking while leaving the lab space. By the time I left that job we had like 3 different printed signs around it begging people not to push the button and explaining the door wasn't locked and you could just leave. Didn't seem to help much in all honesty
5
u/OldGeekWeirdo 18d ago
Seems like they need a dummy exit button so the absent minded can hit that instead.
2
u/OldGeekWeirdo 18d ago
There's one data center that always makes me leery. My office uses a button to open the door to exit. This data center has a button next to the door for an emergency shutdown.
I'm a creature of habit. It hasn't bitten me yet, but ...
5
u/coyote_den HTTP 418 I'm a teapot 17d ago
I know of a room where the exit button is blue and right by the doors. Can’t miss it. Can’t mistake it for the EPO… which is unprotected, red, and if you open the doors fully they will hit it!
3
u/unixuser011 PC LOAD LETTER?!?, The Fuck does that mean?!? 15d ago edited 15d ago
I’ve been in a similar situation to this. Many moons ago I worked in a DC, in the handover from dayshift to night, it was shown that the dayshift didn’t test the fire alarms (they usually do them every Monday morning, but for some reason, it didn’t happen, so we had to do it), so another DC tech goes down and tests them.
Usually, the standard test procedure was, pick a random area, put the alarm box in test mode (which doesn’t send the alert to the fire department) and hit the button and disable it from the main panel.
Guy goes down to test, we hear the alarm go off as expected but instead of lasting for the usual 30 seconds, it keeps going… and going. Then all the windows in the NOC fly open and we hear a massive bang come from the DC and a horrible smell coming from downstairs
So we grab our shit, perform a full evacuation, call the boss and wait outside.
Turns out, the tech was slightly stoned and instead of enabling the test mode, he just hit the random call box, which set off every alarm in the building and enabled the gas suppression system in the DC, thankfully he realised his mistake and hit the suppress button and the DC’s power wasn’t cut. The tech is also very lucky he wasn’t killed because in the early days, we had a Halon suppression system and the tanks were over pressurised and the first time they tested it, one of the tanks exploded and blew the DC’s door of it’s hinges
That was fun explaining to the bosses. All in all, £60,000 and 6 months later, the suppression system is replaced and all is well and management swept it all under the rug (surprisingly the tech didn’t get fired for it)
2
u/CaesarOfYearXCIII 18d ago
Thank you for the story.
Man, whoever thought that a fire alarm button must be blue and unlabelled is a doofus.
2
u/mrs_binfield 16d ago
A "pull" door on an exit route (if it was one) would be a no...
1
u/Wells1632 11d ago
This is what I was thinking. I am also thinking this is not a data center in the US, because if it were, the fire marshall should be investigated as well. I know if we had a setup like this, our fire marshall would shut us down instantly.
2
u/noO_Oon 18d ago
I was once fairly new to this one industry company, but not new in my field. We re-structured the existing software to be programmable by clicking. For that, we needed to know what the departments were referring to when saying „Emergency-Off“ and „Emergency-Stop“. You can only have one of the two: Power cut or kinetic energy cut. Anyway… the definitions varied between the team leads, there was no unified button design and naming and all of us had to read up on emergency standards to even identify the words named above. The company had been in operation for half a century by that point… I‘m mentally shaking my head to this day!
1
u/D0ct0rIT 17d ago
Am I the only one that thinks it's crazy that a company would allow a non-employee to just hang out in their data center alone?
1
u/lurker1B 16d ago
Holy compliance hell batman, I mean yes, but a sub sub contractor you don't know the name of no, but then again what about sign in sign out, checking credentials, etc. they also seem to have forgotten about, unknown unvetted anonymous people having access to critical infrastructure and sensitive data, I think I know the real reason the data center would not want court, this gets out and their clients are liable to flee and possibly sue the DC for more than the sla violation
1
u/Tasty-Mall8577 12d ago
Blue buttons are for Blue Badge holders to open doors in the U.K. (disabled). He should‘ve reported it so they could’ve reset faster, but I understand the running off!
1
404
u/e28Sean 19d ago
I was right there with the datacenter until it was revealed that the e-stop button was unlabeled, unprotected, and blue.
To be clear; that tech absolutely should not have just run off... But, I'm gonna say this is on the datacenter. E-stops should be clearly labeled, have some kind of dual action (i.e. have to lift a cover to hit it) activation, and should be clearly labeled and very obviously an emergency button.