This started as a comment when another Tale reminded me of the fun part, but it got a bit long for a comment, so here we are
In the process of moving into a new building, our company made a bunch of pre-existing infrastructure work well enough to get us up and running. The IT equipment started out in a former janitor closet, but after (repeatedly) coming in on the weekend to babysit overheated servers, IT buddy decided it was time to upgrade to a proper server room complete with its own air conditioner. Nobody was really using it, so the kitchen next to the janitor closet was converted into a server room. The wall of cabinets was converted into miscellaneous IT crap storage, the counter was covered with miscellaneous IT crap, and the sink . . . I like to tell myself nobody uses the sink for anything but I really don't know. The refrigerator moved out, and a couple of full-height racks of IT kit were plugged into the wall with a severely under-sized UPS powering both racks.
The kitchen had a typical power service for a kitchen, so a new power line was put in by the maintenance guy. The maintenance guy had his own way of doing things, which maybe weren't always the smart way. The building has a really big mains circuit breaker panel, a few multi-hundred-amp sub-panels, and a bunch of sub-sub panels for end users. Maintenance buddy ran power from a sub-sub panel to the kitchen. Another 30 feet of wire and he'd have tied into one of the higher-powered sub panels but nosir, he didn't go there.
The widget testing buddies have a sub-sub panel feeding their test bench. Said widget-testing did not always go to plan. As it turns out, some circuit breakers work faster than others. Every once in a while, there would be a flash from the test bench and a small number of lights would go out. The technician who just blew something up would check the mains at his bench. Dead. The tech. goes to his breaker box, and finds all the 20A and 30A circuit breakers are still set to on.
But this is a Tech Support tale because about the same time as our test buddy was figuring out he'd just popped the 200A breaker on the much larger panel, the door at the other end of the room would open, and an IT buddy would pop out like a wack-a-mole, and holler out
"Hey, did you guys just trip a breaker?"
Why yes, how did you know?
In later years, a new home run was pulled to the bigger panel like it should have been. A pair of much-larger UPSs were installed. After a small melting event, even the mains outlet in the kitchen server room was upgraded. After losing a few months of work when a backup system failed to meet expectations, they even managed to get a backup plan to backup the backup. They're so on top of things now, almost nobody in the building is still running Windows XP, even!
Submitted September 17, 2016 at 09:17AM by iamonlyoneman http://ift.tt/2cQDVns talesfromtechsupport
No comments:
Post a Comment