It came from the server room: Halloween tales of tech terror
- Thread starter JournalBot
- Start date
I have no stories this good. I just sit in my cube having my soul sucked out of me.
Upvote
198
(198
/
0)
That in itself is horrific. And highly relatable.[url=http://meincmagazine.com/civis/viewtopic.php?p=27867685#p27867685:2pfwqu6t said:
Upvote
142
(143
/
-1)
I don't think I can top those but my go-to horror story is about the manager who insisted we stopped using databases and instead implement everything in Excel because he knew Excel (or so he claimed). *shudder*
Upvote
115
(115
/
0)
Once upon a time, someone told me if you played the Windows Vista install disc backward you can hear satanic messages. So I tried it. But... I accidentally played it normally AND IT INSTALLED WINDOWS VISTA!
Upvote
145
(166
/
-21)
I was on a cruise ship when I got an email from my boss asking me to immediately block a certain user from a share containing many subfolders and a substantial amount of data. I remoted into the server over the ship's very, very slow satellite Internet connection and went to change the permissions but, due to connection issues, the mouse was jumping around enough that it was difficult to control with the necessary level of precision. When I went to set the Deny permission for that user, I got a pop-up saying something like "do you really want to set the permissions for the Everyone group to Deny on D" Some miracle of good fortune caused me to pay attention to that pop-up, rather than just OK'ing past the routine message as I normally would have, thereby blocking the whole staff's access to the whole partition.
I sent my boss a reply message indicating that I couldn't block the user remotely, but I'd do it when I got back. That was apparently good enough for him, and it was a long time before I admitted to very nearly blocking all access to that mountain of data.
I sent my boss a reply message indicating that I couldn't block the user remotely, but I'd do it when I got back. That was apparently good enough for him, and it was a long time before I admitted to very nearly blocking all access to that mountain of data.
Upvote
45
(45
/
0)
Y2K was not a problem.... Feb 29, 2000 huge problem that almost no one saw coming. (Leap day that only happens once every 400 years.) Days of explaining why everything was date stamped wrong.
Upvote
82
(82
/
0)
May God bless all our brave SysAdmins, Programmers, System Analysts, Systems Administrators, IT Managers and IT Directors out there. Their efforts have carried the computing and technology world (and thus entire world economies) forward over the generations past and present.
Upvote
47
(47
/
0)
The only ones I can think of are...
Back in the day when RAM came on chips, instead of on DIMMs, my brother was installing an upgrade to his, I want to say 386 16MHz (it may have been an early 486, I don't recall exactly). I think it was from 256 to 512KB of RAM. The conversation...
"Hmmm, I wonder if this have a "right way" to go in. I mean, they fit both directions..." 10 seconds later and the power button is flipped. The cloud of smoke that immediately started emminating out of the case told us that, yes, they had an insertion polarity that needed to be followed (after the fact, we noticed the tiny white dot on each, helpfully on the UNDERSIDE of the RAM chips). Amazingly, after much coughing, choking and yanking out of power cords...it still worked once we swapped the chips around.
Next one, more smoke. Around abouts 2000 helping a friend upgrade an old machine (Pentium 166, or actually might have been a K6-166) with a new SCSI HDD in it. Again, polarity is queen. No keying on the cable, inserted it wrong, powered it up...all was fine for a few seconds, and then cloud of smoke coming out of the case. Yanked the power, one of the wires melted clean through the sheathing on the SCSI cable. Reversed it, continued to work (despite the wire being bare. We just wrapped the whole thing in electrical tape so that the bare wire couldn't contact any metal in the case).
Final hijinx. A friend was working on an old CRT monitor that had ceased working. It had just been plugged in. I warned him that before monkeying around in the thing, he might want to wait a half an hour or so for any capacitors to discharge. He thought I was being silly. His screw driver crossed a capacitor. The resulting current discharged caused his muscles to contract so hard he lept backwards (involuntarily) 5ft in to a concrete wall and then lay on the ground for about a minute before I could get anything coherent out of him.
I'll admit about 45 of those 60 seconds I spent doubled up with laughter. Suprisingly we are still friends.
*Edit* A couple of post-posting corrections supplied by my brother who both did the surgery and remembers it better. Apparently it was the L2 cache (now THERE are memories, off-chip replaceable L2) and a 486sx25/33. Also apparently it was depressions on the chips, not white marks. Otherwise, same lead in and result.
Back in the day when RAM came on chips, instead of on DIMMs, my brother was installing an upgrade to his, I want to say 386 16MHz (it may have been an early 486, I don't recall exactly). I think it was from 256 to 512KB of RAM. The conversation...
"Hmmm, I wonder if this have a "right way" to go in. I mean, they fit both directions..." 10 seconds later and the power button is flipped. The cloud of smoke that immediately started emminating out of the case told us that, yes, they had an insertion polarity that needed to be followed (after the fact, we noticed the tiny white dot on each, helpfully on the UNDERSIDE of the RAM chips). Amazingly, after much coughing, choking and yanking out of power cords...it still worked once we swapped the chips around.
Next one, more smoke. Around abouts 2000 helping a friend upgrade an old machine (Pentium 166, or actually might have been a K6-166) with a new SCSI HDD in it. Again, polarity is queen. No keying on the cable, inserted it wrong, powered it up...all was fine for a few seconds, and then cloud of smoke coming out of the case. Yanked the power, one of the wires melted clean through the sheathing on the SCSI cable. Reversed it, continued to work (despite the wire being bare. We just wrapped the whole thing in electrical tape so that the bare wire couldn't contact any metal in the case).
Final hijinx. A friend was working on an old CRT monitor that had ceased working. It had just been plugged in. I warned him that before monkeying around in the thing, he might want to wait a half an hour or so for any capacitors to discharge. He thought I was being silly. His screw driver crossed a capacitor. The resulting current discharged caused his muscles to contract so hard he lept backwards (involuntarily) 5ft in to a concrete wall and then lay on the ground for about a minute before I could get anything coherent out of him.
I'll admit about 45 of those 60 seconds I spent doubled up with laughter. Suprisingly we are still friends.
*Edit* A couple of post-posting corrections supplied by my brother who both did the surgery and remembers it better. Apparently it was the L2 cache (now THERE are memories, off-chip replaceable L2) and a 486sx25/33. Also apparently it was depressions on the chips, not white marks. Otherwise, same lead in and result.
Upvote
131
(133
/
-2)
The year was 2008.
I took over the IT department at the company and we were in the process of cleaning up the mess left from the previous administration. The most horrifying moment was when we discovered that all of the financial data for the company was stored on a dedicated file server that hadn't been backed up in living memory. The last definite record of any backup whatsoever was in 2004.
Now, our financials people made their own backups...and stored them on the file server with the live data. Perfectly secure, right?
Already knowing this was a terrible situation, I investigated further and found that not only had the server not been backed up, it also had a single non-redundant hard drive. And a single power supply. And it was a decade old. But, hey, at least it was rackmount, right?
We scrambled as quickly as possible to move all of the data off and onto our main fileserver, and the entire time I was paranoid that it would finally give up the ghost right then because we were poking it too hard. When we got all of the data moved and powered it down, it was given a place of honor. I keep one of its CPUs (slot-based Pentium !!!s) on my desk to this day, and deep in our underground storage unit the rest of its body lies in state, while other, younger servers have been sent to the scrapyard. It's a department pet.
I took over the IT department at the company and we were in the process of cleaning up the mess left from the previous administration. The most horrifying moment was when we discovered that all of the financial data for the company was stored on a dedicated file server that hadn't been backed up in living memory. The last definite record of any backup whatsoever was in 2004.
Now, our financials people made their own backups...and stored them on the file server with the live data. Perfectly secure, right?
Already knowing this was a terrible situation, I investigated further and found that not only had the server not been backed up, it also had a single non-redundant hard drive. And a single power supply. And it was a decade old. But, hey, at least it was rackmount, right?
We scrambled as quickly as possible to move all of the data off and onto our main fileserver, and the entire time I was paranoid that it would finally give up the ghost right then because we were poking it too hard. When we got all of the data moved and powered it down, it was given a place of honor. I keep one of its CPUs (slot-based Pentium !!!s) on my desk to this day, and deep in our underground storage unit the rest of its body lies in state, while other, younger servers have been sent to the scrapyard. It's a department pet.
Upvote
102
(102
/
0)
Upvote
178
(178
/
0)
Can't beat a story where things caught fire. Worst was maybe when our store was closing one location, their network was a leased-line WAN back to the main store. 2 days before the location closed, in the middle of a huge clearance sale, the line breaks. With no time to repair the line or install any other lines, our network guy cludged together a laptop with a cell modem (PCMCIA card with I think EDGE speed) to run as a VPN router back to the store. It was enough of a connection to allow them to run registers, which is all that was needed.
Upvote
31
(31
/
0)
[url=http://meincmagazine.com/civis/viewtopic.php?p=27867731#p27867731:2mhgyv5c said:
I was once told by an manager that Excel is a great database tool. People later asked me why my face had red outline of a hand print on my forehead.
Upvote
54
(54
/
0)
The company I worked for reorganized and a local facility became part of the same division as the facility I worked at. I was now the IT tech for two facilities. They also decided to change the WAN service for that facility.
It all went well until one day the service just stopped. We were dead in the water because most of servers were off site. People just started doing housekeeping for a few hours until the service started up again. We in IT called our support contacts and did all we could. Nothing worked. The service just started up by itself and nobody knew why.
It happened again the next day, and the day after. And it went on like this for weeks and weeks. Not every day, but more than once a week. I would be there for hours on the phone trying doing all sorts of stuff. This would happen at all times of the day.
The facility got very clean.
The worst part about this was the the outsourcing of this service went 5 levels deep.
The company had outsourced its IT to an IT firm, for whom I worked.
The IT firm outsourced its WAN service to a US telecom.
The US telecom outsourced the Canadian WAN service (where our facility was) to a Canadian Telecom.
The Canadian Telecom didn't really do business IT at the time (?), so they just outsourced it to the other Canadian Telecom.
Who outsourced the last mile service to a local company.
So the support calls were very long and convoluted and just bizarre. We would spend forever on hold waiting for person A to call person B. Joe would tell you to do this, but Sally would tell Joe to stop this because it wasn't his equipment. And as the problem got worse, more and more senior executives from the company who owned the facility got on. The CEO got on the calls once.
And still nobody could figure out the problem.
One day, the problem occurred when somebody from Canadian Telecom #2 was in sub-station that nobody normally went to. He happened to notice an amber light one one piece of equipment. He swapped it and the problem went away.
It all went well until one day the service just stopped. We were dead in the water because most of servers were off site. People just started doing housekeeping for a few hours until the service started up again. We in IT called our support contacts and did all we could. Nothing worked. The service just started up by itself and nobody knew why.
It happened again the next day, and the day after. And it went on like this for weeks and weeks. Not every day, but more than once a week. I would be there for hours on the phone trying doing all sorts of stuff. This would happen at all times of the day.
The facility got very clean.
The worst part about this was the the outsourcing of this service went 5 levels deep.
The company had outsourced its IT to an IT firm, for whom I worked.
The IT firm outsourced its WAN service to a US telecom.
The US telecom outsourced the Canadian WAN service (where our facility was) to a Canadian Telecom.
The Canadian Telecom didn't really do business IT at the time (?), so they just outsourced it to the other Canadian Telecom.
Who outsourced the last mile service to a local company.
So the support calls were very long and convoluted and just bizarre. We would spend forever on hold waiting for person A to call person B. Joe would tell you to do this, but Sally would tell Joe to stop this because it wasn't his equipment. And as the problem got worse, more and more senior executives from the company who owned the facility got on. The CEO got on the calls once.
And still nobody could figure out the problem.
One day, the problem occurred when somebody from Canadian Telecom #2 was in sub-station that nobody normally went to. He happened to notice an amber light one one piece of equipment. He swapped it and the problem went away.
Upvote
85
(86
/
-1)
I was once tasked with figuring out why one of our product's reports didn't match to the penny with a separate report containing the same total. This involved debugging a 29 page (checked in word with no line spacing) SQL query that was wrapped in 3 loops and had entirely cryptic variable names, no comments/documentation, and no one knew what the business logic was supposed to be like. If you're going to ask "why didn't you just look at the working report and copy the functionality?", they were written so radically differently that this approach was the equivalent of rewriting the whole thing. I spent days looking through that one SQL query trying to find out where the start of the calculation I needed was. It sort of felt like trying to read the minified version of a large javascript file in order to pinpoint how a particular variable is calculated, except the javascript logic was written three people who thought the business logic worked six different ways and no one knows which variable you're looking for.
Happened to be that I was already on my way out for separate reasons, so I never did figure out how it worked. Part of me wants to try that challenge again, but the sane part of me never wants to see anything like it again.
Not much of a horror story I suppose, but it really felt like it when I came in the morning after spending a day looking at it with hardly any progress.
Happened to be that I was already on my way out for separate reasons, so I never did figure out how it worked. Part of me wants to try that challenge again, but the sane part of me never wants to see anything like it again.
Not much of a horror story I suppose, but it really felt like it when I came in the morning after spending a day looking at it with hardly any progress.
Upvote
51
(52
/
-1)
It was my first week at my first IT job. A few coworkers got up all of a sudden and quickly started heading for the server room. I asked what was going on, and their response was "there's a fire in the server room, come help!" When we got there, the halon fire suppression system hadn't gone off, and we couldn't smell smoke, so we carefully opened the door to look inside. Nothing. Everything was working fine. In fact, almost the exact opposite happened. After checking the situation out, the main A/C unit had failed, which kicked on the backup A/C unit which hadn't been tested in months. It had frozen over in the short time it was alive, and when the ice buildup began to melt it flooded the false floor where a smoke detector was, shorted it out, prompting the system to think there was a fire. Interesting first week.
Upvote
71
(71
/
0)
About 10 years ago I was working desktop support at a DOE facility. Some "Cyber Security Specialist" drone suddenly discovered that we had a major security problem, in that it showed the last username on the login screen to all our Win2k and XP desktops. THAT'S HALF OF OUR SECURITY he proclaimed.
So at 2PM on Halloween, desktop support received an email indicating we had an emergency, mandatory, SMS push coming tonight and to alert the users to leave their desktops on. We did.
What got pushed was the entire HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon key and all its values that the "Cyber Security Specialist" had just exported off his desktop XP machine with the "don't show last user" value turned on. Not just the one value, but the entire key and its many, many entries. The SMS folks were directed to push out that patch exactly as provided to every single desktop.
The XP desktops had major problems. The 2k machines, since they'd just been passed a bunch of values they didn't understand, came up to a blank wallpaper screen with no login box. Every. Single. One. More than 1000. None would get onto the network, none were usable.
We had to pull the HDD out of every one of those 2k machines, do the remote registry edit thing, fix a bunch of entries, and mount them back in the old boxes. Plus fix anything with the XP machines that may have broken.
So at 2PM on Halloween, desktop support received an email indicating we had an emergency, mandatory, SMS push coming tonight and to alert the users to leave their desktops on. We did.
What got pushed was the entire HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon key and all its values that the "Cyber Security Specialist" had just exported off his desktop XP machine with the "don't show last user" value turned on. Not just the one value, but the entire key and its many, many entries. The SMS folks were directed to push out that patch exactly as provided to every single desktop.
The XP desktops had major problems. The 2k machines, since they'd just been passed a bunch of values they didn't understand, came up to a blank wallpaper screen with no login box. Every. Single. One. More than 1000. None would get onto the network, none were usable.
We had to pull the HDD out of every one of those 2k machines, do the remote registry edit thing, fix a bunch of entries, and mount them back in the old boxes. Plus fix anything with the XP machines that may have broken.
Upvote
120
(120
/
0)
When I was (much) younger, and much more naive, I had an old computer running Windows 3.1. I was attempting to install a game, but was out of hard drive space. So I went into MS-DOS mode, and was cleaning out files, when I came across "progman.exe". This having been before widely-available Internet, I thought to myself "Progman? I have no idea what that is" and deleted it. Needless to say, that computer very soon got Windows 95 installed on it.
While actually working IT, I got a call from a customer. Below is a rough recollection of the conversation.
Me: "IT, this is *my name*, how may I help you?"
Customer: "None of the laptops in our building will connect to the Internet!"
Me: "What building are you in?"
Customer:"Building X"
Me: *Looks at my notes about known issues* "Ma'am, it appears as though your building has a power outage due to some electrical work"
Customer: "Yeah, as soon as the power went out, we lost Internet access"
The switch they connected was on an UPS, but the outage was scheduled to take longer than the battery life of the UPS.
Me: "Ma'am, we cannot do anything to get you back online until the power is back"
Customer: "But our laptops are working just fine! It's just the Internet that's down!"
Me: "Ma'am, your laptops have built-in batteries. The switch connecting you to the Internet does not have a built-in battery like a laptop has."
Customer: "..." *CLICK*
While actually working IT, I got a call from a customer. Below is a rough recollection of the conversation.
Me: "IT, this is *my name*, how may I help you?"
Customer: "None of the laptops in our building will connect to the Internet!"
Me: "What building are you in?"
Customer:"Building X"
Me: *Looks at my notes about known issues* "Ma'am, it appears as though your building has a power outage due to some electrical work"
Customer: "Yeah, as soon as the power went out, we lost Internet access"
The switch they connected was on an UPS, but the outage was scheduled to take longer than the battery life of the UPS.
Me: "Ma'am, we cannot do anything to get you back online until the power is back"
Customer: "But our laptops are working just fine! It's just the Internet that's down!"
Me: "Ma'am, your laptops have built-in batteries. The switch connecting you to the Internet does not have a built-in battery like a laptop has."
Customer: "..." *CLICK*
Upvote
71
(71
/
0)
Not really a horror story per se, but an older lady came in with a laptop complaining that her facebook games were no longer working. Before she opened it up, I asked her which browser she usually used.
"Oh, I use the cheese one."
I was pretty used to people calling things by the wrong name, but I had no idea what she was talking about.
"The cheese one?" I asked her.
"Yeah, you know. The mozzarella Firefox."
I had to step away for a moment until I could keep a straight face again.
"Oh, I use the cheese one."
I was pretty used to people calling things by the wrong name, but I had no idea what she was talking about.
"The cheese one?" I asked her.
"Yeah, you know. The mozzarella Firefox."
I had to step away for a moment until I could keep a straight face again.
Upvote
131
(131
/
0)
[url=http://meincmagazine.com/civis/viewtopic.php?p=27867947#p27867947:12y9tbs7 said:
Sorry bout that, I probably wrote that query.
I was responsible for writing a lot of financial, shipping, production, etc reports that the executives and department heads used. Many of them I just threw together to show the bosses and to prototype my queries and logic. They would more often than not deem the prototype "good enough" and give me two days to polish it. A lot of my queries ended up being the same I used for prototyping, many of which were quite rough. I feel bad for whoever has to debug or add columns to some of those in the future.
Rapid Application Development is a double-edged sword for some companies.
Upvote
53
(54
/
-1)
Two tales of horror, one personal, one told to me.
Way back in my youth, when dinosaurs ruled the IT industry, I worked at LISP Machine Incorporated. One month we got a batch of power filtering capacitors delivered, in which some of them had been incorrectly labeled with positive and negative grounds. Those suckers could store quite a dollop of electrons in them before they decided to explosively destruct, and for a while it was a fun trick to play on the techs smoke-testing boards to walk up behind them quietly and clap your hands together hard. My, how they would jump! One day I walked into the testing area just in time to hear a loud crack and see a flaming capacitor land at my feet from a machine a good 100’ away.
I had a friend who worked at Dialog Systems (later Verbex), and they had a machine room full of Vaxxen. One weekend, the cooling system failed, and as designed, the alarm went off. The guard, concerned, entered the room to see what was what, saw nothing immediately wrong, and proceeded to silence the alarm and shut the door. When the staff came in on Monday, it was 120 degrees in the room and every machine had gone to its eternal reward.
Way back in my youth, when dinosaurs ruled the IT industry, I worked at LISP Machine Incorporated. One month we got a batch of power filtering capacitors delivered, in which some of them had been incorrectly labeled with positive and negative grounds. Those suckers could store quite a dollop of electrons in them before they decided to explosively destruct, and for a while it was a fun trick to play on the techs smoke-testing boards to walk up behind them quietly and clap your hands together hard. My, how they would jump! One day I walked into the testing area just in time to hear a loud crack and see a flaming capacitor land at my feet from a machine a good 100’ away.
I had a friend who worked at Dialog Systems (later Verbex), and they had a machine room full of Vaxxen. One weekend, the cooling system failed, and as designed, the alarm went off. The guard, concerned, entered the room to see what was what, saw nothing immediately wrong, and proceeded to silence the alarm and shut the door. When the staff came in on Monday, it was 120 degrees in the room and every machine had gone to its eternal reward.
Upvote
72
(72
/
0)
One morning, I received a call from a client.
After changing cache batteries on a disk IOA, one of their RAID-5 sets (there were three in the disk group) was bad, and they couldn't get it to recover.
I asked him where his system save was; he said that he had planned to do it after the machine came back up.
So, I asked for his most recent backups.
After taking them back to the office to review, I realized that he was missing a substantial amount of his data.
It turns out that he was doing a two-part backup, and that the second part was overwriting over the first.
According to the job logs, each part of the save was working just fine. He had never bothered to look at the contents of the tapes, though.
They ended up losing six weeks worth of data.
To finally top it off, the system sat on the floor of the server room (it was a tower config machine).
As I was well into the long process of restore, the client was messing around with a rack that stood next to the machine. As he was putting on a side panel to the rack, he dropped in on a disk expansion tower, and caused a drive (just one, luckily!) to go bad.
After changing cache batteries on a disk IOA, one of their RAID-5 sets (there were three in the disk group) was bad, and they couldn't get it to recover.
I asked him where his system save was; he said that he had planned to do it after the machine came back up.
So, I asked for his most recent backups.
After taking them back to the office to review, I realized that he was missing a substantial amount of his data.
It turns out that he was doing a two-part backup, and that the second part was overwriting over the first.
According to the job logs, each part of the save was working just fine. He had never bothered to look at the contents of the tapes, though.
They ended up losing six weeks worth of data.
To finally top it off, the system sat on the floor of the server room (it was a tower config machine).
As I was well into the long process of restore, the client was messing around with a rack that stood next to the machine. As he was putting on a side panel to the rack, he dropped in on a disk expansion tower, and caused a drive (just one, luckily!) to go bad.
Upvote
15
(15
/
0)
Probably the best was a call from a small staffing company run out of a shop front in a strip mall around 2003. They had gotten to me through the grapevine, and this was my first time working with them. The problem was described as "No one can print anything, the switch seems to be dead." So I asked how many machines they had, and they said just 5. So I told them I would pickup a replacement switch and drop by to install it.
When I got there I asked to see the switch. I was then taken to the back of the room where there was a microwave cart with a basic inkjet on it. Sitting next to it was a physical parallel port switch, with 4 selections hooked to 50 foot parallel cables, and 1 loose cable that I was told they swapped when that computer needed to print.
Further they literally had 10 phone lines coming in, 5 of them hooked to individual 56k modems for each PC. I was terrified I had accidently walked through a rift in time to 10 years before.
The good news is that I convinced the owner to invest in a basic ethernet setup, and to get cable internet installed. There were 4 much happier workers after that.
When I got there I asked to see the switch. I was then taken to the back of the room where there was a microwave cart with a basic inkjet on it. Sitting next to it was a physical parallel port switch, with 4 selections hooked to 50 foot parallel cables, and 1 loose cable that I was told they swapped when that computer needed to print.
Further they literally had 10 phone lines coming in, 5 of them hooked to individual 56k modems for each PC. I was terrified I had accidently walked through a rift in time to 10 years before.
The good news is that I convinced the owner to invest in a basic ethernet setup, and to get cable internet installed. There were 4 much happier workers after that.
Upvote
56
(56
/
0)
My favorite involves management, laptops, and HDD failures.
I had just gotten promoted as to an IT tech. I had been working at the company for a while, but was unaware of one little thing. Management had some type of bet to see who could keep doing their job on the oldest laptop.
A week into my new job, one of these management laptops failed. An old Sony laptop running windows 2000 (Vista was the hot shit at the time). The HDD was just dead, and of course the backup scripts setup by the previous IT tech had failed. Years ago, even.
So, me, the new IT tech has to explain to a touchy manager that we need to send this failed HDD out for costly, and possibly useless, data recovery. All because the backup scripting had failed, without notification, on an ancient laptop. Not fun times.
The outcome was ok, we got a new backup and monitoring system. Management got put on a laptop replacement schedule. I kept my job.
I had just gotten promoted as to an IT tech. I had been working at the company for a while, but was unaware of one little thing. Management had some type of bet to see who could keep doing their job on the oldest laptop.
A week into my new job, one of these management laptops failed. An old Sony laptop running windows 2000 (Vista was the hot shit at the time). The HDD was just dead, and of course the backup scripts setup by the previous IT tech had failed. Years ago, even.
So, me, the new IT tech has to explain to a touchy manager that we need to send this failed HDD out for costly, and possibly useless, data recovery. All because the backup scripting had failed, without notification, on an ancient laptop. Not fun times.
The outcome was ok, we got a new backup and monitoring system. Management got put on a laptop replacement schedule. I kept my job.
Upvote
27
(27
/
0)
In my first job out of university I setup a remote data collection box that pumped the data over isdn back to the clients head office. This had 2 isdn routers on 2 different telcos. To set it up we pointed the lines back to the bosses house. Once the lines in the clients office were in, I removed to the collection site and updated the end point.
Six weeks later we were called to the client. Turns out I had forgotten to change the backup line. The primary had died and the backup was trying to connect to the clients ftp server. This was failing and resetting the line....every minute for 10 minutes every half an hour.
The client dropped the bill on the table.... It must have been 7cm thick of A4, which promptly broke the leaf of the table.
I'd manage to get a £10k+ bill of minimum charges of 5p per call.....opps
Six weeks later we were called to the client. Turns out I had forgotten to change the backup line. The primary had died and the backup was trying to connect to the clients ftp server. This was failing and resetting the line....every minute for 10 minutes every half an hour.
The client dropped the bill on the table.... It must have been 7cm thick of A4, which promptly broke the leaf of the table.
I'd manage to get a £10k+ bill of minimum charges of 5p per call.....opps
Upvote
21
(24
/
-3)
Not sure if this counts:
On our only production MySQL server. I was attempting to manually delete a huge (>2GB) MyISAM database, and the quickest way to do it was command line. Except I left some of the path out of the command...
I've done this twice in my tenure here.
On our only production MySQL server. I was attempting to manually delete a huge (>2GB) MyISAM database, and the quickest way to do it was command line. Except I left some of the path out of the command...
I've done this twice in my tenure here.
Upvote
22
(25
/
-3)
Oh god I remember the Desktop III machines from my time in the Coast Guard. They were running concurrently with Windows 2000 based machines due to legacy software. If I remember correctly they were all daisy chained on our ship so taking one offline would destroy their primitive network. Not my deal I just fixed radars.
Upvote
21
(21
/
0)
Not really an IT issue exactly. I used to work at a retail tech support counter in a big box store. Most of our customers were elderly or uninformed because otherwise they would just fix the issue themselves, but the few times they came to the counter after doing a little research were cringeworthy. In particular, a woman asked me if having too many "cakes" on her computer could cause problems. It took me almost 2 minutes of questioning to realize she mixed up her baked goods.
Upvote
13
(14
/
-1)
I have a Zenith laptop manual somewhere.
Me I have no stories from my extremely short stint as a Microsoft support screener. No other than people losing extremely sensitive data because they don't keep backups. That and trying to get keys generated.
Me I have no stories from my extremely short stint as a Microsoft support screener. No other than people losing extremely sensitive data because they don't keep backups. That and trying to get keys generated.
Upvote
0
(1
/
-1)
Tales of computer humor from the late 80's.
Guy in one of the departments gets a new printer then his software stopped working (can't recall what it was) and whenever he tries to start it an error about hardware key error keeps coming up with a picture of a key, and he can't figure out what's wrong since he tried the key already.
So I headed down to his office and find out he's been trying to get it working by using the keyboard lock key and locking and unlocking his keyboard. Turned out when his printer was swapped they took it away, cable and all, and never noticed the parallel port anti-piracy dongle was still on the end of the cable, which had been haphazardly tossed into a large pile with a bunch of others that had been recently decommissioned. It probably took a couple of minutes of checking cables to find the right one, but the guy switching his keyboard lock on and off and turning to me with a bewildered look as to why it wasn't working still sticks in my mind after all these years.
Guy in one of the departments gets a new printer then his software stopped working (can't recall what it was) and whenever he tries to start it an error about hardware key error keeps coming up with a picture of a key, and he can't figure out what's wrong since he tried the key already.
So I headed down to his office and find out he's been trying to get it working by using the keyboard lock key and locking and unlocking his keyboard. Turned out when his printer was swapped they took it away, cable and all, and never noticed the parallel port anti-piracy dongle was still on the end of the cable, which had been haphazardly tossed into a large pile with a bunch of others that had been recently decommissioned. It probably took a couple of minutes of checking cables to find the right one, but the guy switching his keyboard lock on and off and turning to me with a bewildered look as to why it wasn't working still sticks in my mind after all these years.
Upvote
17
(18
/
-1)
I'm working with a colleague of mine on our research, in preparation for getting his doctoral thesis done. The beowulf cluster we're using starts sending messages about the temperature being too high... it's at this point that I notice the faint, white smoke curling in from the room next door.
The power supply on one of the nodes had started smoldering its way to an electrical fire.
No one ever really used the system after my colleague finished his thesis, but it kept running, and slowly losing nodes until the time I left.
The power supply on one of the nodes had started smoldering its way to an electrical fire.
No one ever really used the system after my colleague finished his thesis, but it kept running, and slowly losing nodes until the time I left.
Upvote
11
(11
/
0)
[url=http://meincmagazine.com/civis/viewtopic.php?p=27868035#p27868035:2v16nse8 said:
I guess I should clarify that I don't feel upset that it was written the way it was, the management in that company was the sort where everything the client wanted was considered mission critical, and the client wanted everything they could possibly think of. It was the kind of place where you'd work on a web application they were in dire need of for 2-3 months, and then when it was done they didn't even have time to log into it.The kind of place where the general manager honestly thought backing up our 300+ GB (probably more but I can't remember now) data center to CD-ROMs was a good idea.
Basically, everyone before me had been strongly pressured into writing spaghetti code at one time or another, so I don't blame anyone for it.
Upvote
37
(37
/
0)
Legitimate question. Should this scare me of from this line of work. I'm a freshman in college and I am planning on majoring in MIS with minor in Computer Science.
Upvote
18
(19
/
-1)
johnny_blvd
Ars Centurion
Around 2002 or so I got a frantic, near screaming call from one of the executive assistants. The exec's laptop cannot connect to the network and he has to get on now. So I dutifully rush over as none of the other techs would deal with either of them. The laptop didn't have a built-in ethernet port so we had installed a 3-Com PCMCIA network card in it. The exec had handed his laptop to his assistant and told her to get him docked and logged in, but she being clueless to anything technology related didn't know where to plug in the dongle for the network card. Somehow she managed to jam the dongle on to the top row of pins on the COM port that was directly behind the PCMCIA card. It took all my will to not laugh as soon as I saw that. Instead I calmly plugged it in the right port, made sure he was online, then laughed the entire way back to my desk.
Upvote
18
(19
/
-1)
My dad had a 386 toshiba laptop with I think 40MB harddrive and 4mb ram. My brother got his hands on warcraft 2. The floppy disk edition. I think 20 floppy's arj'd.
Offcourse there was not disk space but we really wanted to test it out. So we backup the whole system to floppies and started unpacking. When finally done, we fired it up.
warcraft 2, it turned out, needed 8mb of ram... bugger. So we restored the system. Except... one disk failed... Let's say that my dad was not happy.
I bought my first computer, second hand but it worked fine, but it missed an audio card. So I bought an ISA soundblaster 16bit. I was so proud. So I unscrewed the casing, installed the card and turned the machine on.
*sniff* *sniff*. Hmm, that doesn't smell good. As quick as I could I disconnected the mains, removed the card and tried again. No problem. Ok, one more try, same thing.
Later a friend of my brother came by, we discussed it and he took a look. Eventually opened up the power supply (which did had a bit of noise when moved but ok...) and found an screw in there. Apparently the soundcard required a bit more power and short circuited things. After that everything worked as it should. (well, it was an isapnp card.. but ok)
Offcourse there was not disk space but we really wanted to test it out. So we backup the whole system to floppies and started unpacking. When finally done, we fired it up.
warcraft 2, it turned out, needed 8mb of ram... bugger. So we restored the system. Except... one disk failed... Let's say that my dad was not happy.
I bought my first computer, second hand but it worked fine, but it missed an audio card. So I bought an ISA soundblaster 16bit. I was so proud. So I unscrewed the casing, installed the card and turned the machine on.
*sniff* *sniff*. Hmm, that doesn't smell good. As quick as I could I disconnected the mains, removed the card and tried again. No problem. Ok, one more try, same thing.
Later a friend of my brother came by, we discussed it and he took a look. Eventually opened up the power supply (which did had a bit of noise when moved but ok...) and found an screw in there. Apparently the soundcard required a bit more power and short circuited things. After that everything worked as it should. (well, it was an isapnp card.. but ok)
Upvote
7
(9
/
-2)
- Status