what did you learn today?

[afidel]

The company that used to support our Nortel and Avaya systems can no longer support these systems because they have decided to use Cisco internally. Which means the remaining 3 years of our support contract ends in a few months.

That is bullshit, and a contractual breach on the part of both Avaya and your vendor. I find it very difficult to believe that Avaya is doing that (perhaps it is because they fired their Avaya-certified engineers?), but you should be taking your vendor to court for the contract breach.

I wouldn't be surprised, they threatened to take us to court after we didn't renew our maintenance and went with a third party support provider. If I have anything to do with it this will be the last Avaya phone system I ever have to deal with, and my telecomm guy is of the same opinion. What makes that amazing is that he was previously a Navy trained Avaya expert with little experience outside of Avaya technologies. There are some really dense people running things over there.

 

[Spatula]

How do you incorrectly install RAM, very expensive RAM, and not notice? Maybe the same way you forget to buy a warranty for a really expensive computer.

 

[Barmaglot]

How do you incorrectly install RAM, very expensive RAM, and not notice? Maybe the same way you forget to buy a warranty for a really expensive computer.

Was there magic smoke involved?

 

[ronelson]

Was there magic smoke involved?

"Was" being the operative word.

 

[Paladin]

I'm assuming they managed to force it into a slot the 'wrong way' somehow? Nice.

I've reprimanded a few employees for that kind of thing. Getting past that mentality of 'just push harder' when you are dealing with other people's hardware/software is a pretty important thing. The last time I had to deal with something like that was an employee that was tasked with doing a clean install/upgrade on a customer's servers to go from win2k3 to 2k8R2. The customer 'said' they had backups so the employee wiped the 2 servers in question, installed the OS on both and then stopped to ask, "What IP settings, users and passwords etc should I put on these?"

My response: you didn't document anything first? you didn't make any backups?

Of course, the customer's backups were just a copy of their important website files. No backup of the databases involved, settings, anything.

Fortunately it turned out they had already been offline more or less for the last 4 months so the data in question was basically worthless and they welcomed us just starting fresh and doing things right.

 

[PVO]

The last time I had to deal with something like that was an employee that was tasked with doing a clean install/upgrade on a customer's servers to go from win2k3 to 2k8R2.

del *.*

Fortunately it turned out they had already been offline more or less for the last 4 months so the data in question was basically worthless and they welcomed us just starting fresh and doing things right.

I had that situation happen several times in the past, unfortunately, it was usually a mission-critical app or server with a specialized app on it, and special restore / recovery procedures. My favorite time was when the fellow that never documented anything and kept all his passwords, access, and knowledge known only to himself (job protection FTW) was out of the country, and totally unreachable for at least 72 hours.

Good times.

 

[Spatula]

No, this is a bit easier to mess up, but does excuse not noticing that the system doesn't recognize the RAM once in the OS.

It was a Mac Pro with FB-DIMMs. It was purchased with 2x1GB, and aftermarket 2x2GB was purchased. For 2x1GB, one stick goes on each RAM pull-out card. But to properly match two pairs of FB-DIMMs, you put one pair on each pull-out card.

But I imagine 2x2GB FB-DIMM was a hell of a lot more expensive 2 years ago than it is today.

 

[ronelson]

Was there magic smoke involved?

I think I cursed myself by responding to this. Nothing to do with work, but I had my AC replaced last week and kept the fan from the air handler. I planned to make it into a portable fan for the shop, since it has been over 100F all summer and it is unbearable down there. I plugged it in wrong (hey, I couldn't find the wiring diagram buried under the crud at first, then when I could it made no sense me) and I got some smoke, but it was the burning kind. Now that I have it wired properly...it still smokes. Just a bit, and the smell is hardly noticeable. Not sure if there is a problem just because (we did have to replace it, after all), because of how I plugged it in earlier, or because there is a bunch of debris that fell into the coils that is smoking. Maybe tomorrow I will have a chance to plug it in for 10 minutes and watch it, maybe it just needs to burn out the debris. *sigh*

At least I tried to wire it before I built the case for it

 

[PaveHawk-]

I learnt today that I'm working way too hard at my place (at least 2 hours work done every weekend for the last 6, but the average is sitting on close to 9) for no apparent reason; that is to say that my company doesnt appreciate the work I'm putting in to bring projects back on track and resolve other technical issues that affect our support operations (its an IT MSP/Consulting firm).

Yet for all my "ability", I sit here questioning myself wondering if I am half as good as I think I am and if I'm skilled enough to apply for that Senior engineer/Team Lead type role.

 

[BajanDude]

Apply. At worst you learn that you're not. At best, you learn that you are.

 

[WarheadsSE]

I learnt today that I'm working way too hard at my place (at least 2 hours work done every weekend for the last 6, but the average is sitting on close to 9) for no apparent reason; that is to say that my company doesnt appreciate the work I'm putting in to bring projects back on track and resolve other technical issues that affect our support operations (its an IT MSP/Consulting firm).

Yet for all my "ability", I sit here questioning myself wondering if I am half as good as I think I am and if I'm skilled enough to apply for that Senior engineer/Team Lead type role.

This shows you're worth taking the shot.

If you thought outright, "This is below me, I MUST move up!" we'd probably be telling you your full of it.

Take the shot.

 

[ronelson]

Apply. At worst you learn that you're not. At best, you learn that you are.

Technically, applying and getting the job will not tell you if you are skilled enough. But if you get the job, *then* you find out if you are skilled or not. My guess is that you are not, but you can grow into it. Why would you take a position that offers no challenges anyway? Go for it.

 

[Frennzy]

Ditto on the "go for it", nobody every got a promotion by being a wallflower.

Today I learned that even though Infoblox comes with more than one interface, you can only use one at a time.

 

[kennedye]

I learnt today that I'm working way too hard at my place (at least 2 hours work done every weekend for the last 6, but the average is sitting on close to 9) for no apparent reason; that is to say that my company doesnt appreciate the work I'm putting in to bring projects back on track and resolve other technical issues that affect our support operations (its an IT MSP/Consulting firm).

Yet for all my "ability", I sit here questioning myself wondering if I am half as good as I think I am and if I'm skilled enough to apply for that Senior engineer/Team Lead type role.

I have that same feeling every day, either that I'm going to be fired before the day is out, or that I should quit because I'm dragging the company down and they should replace me with someone more competent. You should see my end-of-year self-reviews...

 

[zyphos]

I learnt today that I'm working way too hard at my place (at least 2 hours work done every weekend for the last 6, but the average is sitting on close to 9) for no apparent reason; that is to say that my company doesnt appreciate the work I'm putting in to bring projects back on track and resolve other technical issues that affect our support operations (its an IT MSP/Consulting firm).

Yet for all my "ability", I sit here questioning myself wondering if I am half as good as I think I am and if I'm skilled enough to apply for that Senior engineer/Team Lead type role.

I have that same feeling every day, either that I'm going to be fired before the day is out, or that I should quit because I'm dragging the company down and they should replace me with someone more competent. You should see my end-of-year self-reviews...

A little something for everyone to ponder:
Dunning–Kruger effect

 

[bigmikebrooklyn]

For us, iPhone setup is quick, because all we need to do is setup Exchange on them. Apple has the iPhone Configuration Tool which means that all I need to do is plug in an iPhone, load the profile onto the device, then give it back to the user so they can enter their Exchange credentials. Blackberry setup is longer, but it's still a case of entering the activation information then leaving it alone.

My big issue is that I never hear from the iPhone users again, whereas for the past few months we've had users come down and complain that their calendar isn't syncing. Duplicate appointments, appointments not updating, all sorts of good things. And we (me and my co-worker) have no idea how to fix this. We haven't been able to isolate it to an Exchange thing (Delegates doing the wrong thing? Wouldn't be the first time) or a Blackberry issue. It only affects certain users, so lots of repeat customers.

And we don't have T-support so I can't call RIM for help

quick off the cuff for this: wipe handheld, delete BES account, recreate bes account, reactivate handheld.
that procedure will work wonders, especially on shakey systems that should have been replaced 3 years ago and have many voodoo hexs and charms running around on them (cough ahem cough).

also might want to check the users in question's OST files, and mailbox (db) integrity on their mailstore.
hope you kick that issue, i hate seeing the same people every 3 days...

 

[bigmikebrooklyn]

I learnt today that I'm working way too hard at my place (at least 2 hours work done every weekend for the last 6, but the average is sitting on close to 9) for no apparent reason; that is to say that my company doesnt appreciate the work I'm putting in to bring projects back on track and resolve other technical issues that affect our support operations (its an IT MSP/Consulting firm).

Yet for all my "ability", I sit here questioning myself wondering if I am half as good as I think I am and if I'm skilled enough to apply for that Senior engineer/Team Lead type role.

I have that same feeling every day, either that I'm going to be fired before the day is out, or that I should quit because I'm dragging the company down and they should replace me with someone more competent. You should see my end-of-year self-reviews...

A little something for everyone to ponder:
Dunning–Kruger effect

isn't this the second time someone's posted this recently, or am i just on converging intellectual persuit paths with you?

also, i learned i hate visio. just put the shit where i want it. stop drawing crazy lines automagically. why can't get the default size of type in a text box to stay where i set it or make changes to the default stick. when are they releasing the magic wythinkiwyg brain helmet? cripes, if i can't have my flying car, at least give me that.

deep breaths.

 

[Danger Mouse]

That old [Cisco] switches can have interesting failure modes:

2924XL-1: machine gun fire sounds from the switch as components on the main PCB exploded one after the other
2924XL-2: sparks like those from a sparkler, plus a little magic smoke smell (no smoke)

These were working pulls, but I suspect they weren't dusted before being brought to me so it was likely a dust caused issue. I dust switches I collect myself.

NBD, but I may have to get out a hammer/screwdriver to use on the flash chips. (Security policy - erase configs, secure erase drives, etc)



EDIT: Obviously, other electronics can have interesting failure modes or even the same/similar. I'm posting a somewhat (maybe not at all) humorous anecdote. Switch 3 is just blinky blinky over and over.

 

[zyphos]

isn't this the second time someone's posted this recently, or am i just on converging intellectual persuit paths with you?

also, i learned i hate visio. just put the shit where i want it. stop drawing crazy lines automagically. why can't get the default size of type in a text box to stay where i set it or make changes to the default stick. when are they releasing the magic wythinkiwyg brain helmet? cripes, if i can't have my flying car, at least give me that.

deep breaths.

You're correct. I did a search. Danger Mouse has brought it up before in this thread. It's also in a few other threads around the fora.

 

[SideAngleSide]

Today, I learned that if a vendor does not have a documented migration plan for a system upgrade that you are planning, you write one. And if they fully support you (and your upgrade), then the day is good.

 

[llib]

A "Test" button on the front of a transfer switch isn't really a good test.

I submit a largish, three-phase, 3000a, 480v, transfer switch that works every time the "Test" button is pressed. Fires up the Genset? Check. Source 2 available in less than 15 seconds? Check. Transition to Source 2? Check.

Enter power failure #1. Fires up the Genset? Check. Source 2 available in less than 15 seconds? Check. Transition to Source 2? Not so good. Error: "Fail to release Source 1". Hmmm... /me thanks the goddess for the Side-1 genset and cross-feed transfer switch.

Okay. Everything back to normal. Press the "Test" button. Fires up the Genset? Check. Source 2 available in less than 15 seconds? Check. Transition to Source 2? Check. WTF?!? Test three more times over the next week. Works every time. Well, maybe the failure was "just one of those things..."

Enter power failure #2. Fires up the Genset? Check. Source 2 available in less than 15 seconds? Check. Transition to Source 2? Not so good. Error: "Fail to release Source 1". %#*&#&% Oh, crap! Side-1 Genset took a dive. ARGH!! Power back three minutes later. Phew!!

Okay. Everything back to normal. Press the "Test" button. Fires up the Genset? Check. Source 2 available in less than 15 seconds? Check. Transition to Source 2? Check. Huh?!? Okay, let's think a bit. What's the diff between a "Test" and a power failure? Well, in a test, normal (Source 1) is still available. In a power failure, it isn't. So...

First, swing the load to Side 1 so we can dink with Side 2 infrastructure without messing with the data center. Next, walk over to big, honkin' 2,500 amp normal feed breaker to Side 2. Next, push the "Trip" button. After the godalmighty crash, hurry to the transfer switch. Loss of Phase. Good. Fires up the Genset? Good. Source 2 available in less than 15 seconds? Good. Transition to Source 2? Not so good. Error: "Fail to release Source 1". Ahah! We can duplicate the problem. (Anyone who has ever troubleshot any kind of complex system knows that duplicating the problem can sometime be a real PITA...) Call to service shop ensues.

Okay, we already knew that the switch's control boards have medium-sized on-board batteries to keep the control system running during power outages. But the switch mechanism actuators require a bit more oomph, so there are two 48vdc power supplies, one fed from normal, the other fed from emergency. The outputs are paralleled and feed the actuator control board. So at least one of the PSes will be up at any given time and will be able to power the actuators. Except the PS powered by the emergency bus was defunct due to some bad wiring from an earlier repair incident. So as long as we had normal power available, the switch would work (in either direction). But in a power failure, the Normal PS was no longer available and Emergency PS wasn't working, so there was no power for the actuators hence the "Failed to Release Source 1" error message. Simply mahvelous.

Lesson learned: When simulating a real-world event, make sure your simulation accurately reflects the real world event. In our case, pushing the Test button wasn't a real good end-to-end test. We were a little paranoid about bumping the big breakers, but if you want to simulate a power failure, you've got to by-God fail the power! The whole Diesel side was basically out of service for the past six months without us being aware of it! Eek!

Well, all is good now, and we'll continue to use the test button on our monthly and quarterly tests, but semi-annually we will dump the big breakers and have a by-God power failure! Of course, this being Oklahoma, either mother nature or OG+E might just do the test for us from time to time... One nice thing abut fully-redundant power systems, we can test one side at a time, leaving the other side on normal commercial power. Minimizes risk, but still lets us test/repair/dink-with one side or the other...

 

[Danger Mouse]

....that the 2 hour OWA downtime that mysteriously fixed itself earlier this week, was due to someone in senior management having their email account details stolen. (I was out of the office, so literally missed finding out about it, until I returned to my office, by which time it had fixed itself)

Why? Because they responded to a 419 scam, even though there had been at least 1 alert email every few weeks for the last few months because of such scams.

The response? "They didn't want to respond to the email, but were 'forced' to do so." [sic]


Really? Apparently, they were being held hostage in the Nakatome building by East German terrorists, whose leader was named "Hans"....

Or, they just fell for the financial hook of the scam. This guy makes $250k a year, but that's not enough to prevent him from falling for a 419 scam (aka "Nigerian" scam).

 

[Danger Mouse]

...that Cisco had a HPNA like product in their Catalyst 2950 LRE. Now to find a setup in reasonable condition because I can justify an equipment purchase but not a new cable run for just one user in a building that will eventually be torn down. Since there is existing copper (POTS wiring only), we can use that. Wireless is not an option, unless we're going to drill holes through the building (several walls) to get an antenna outside (not going to happen as that would be modifying a building to be torn down). There is almost no signal inside.

Then there would be the attendant distance issues, plus requiring connectivity to our Cisco VOIP setup, data security requirements, etc, etc.

Finding new POTS HPNA gear is a huge pain in the rear. I've only found one new router/bridge in wide availability and that's a Corinex system, which is likely a rebadge of some other products I've found to be vaporware in the USA but maybe available in other countries.

 

[Seany]

i've used these guys before to get me out of jams (how the @#$% do you get ethernet above a light fixture in the middle of the hotel ball room? oh the facilities guys can get us one pair of cat3... grreeaatt)

 

[PaveHawk-]

Replying to a few in a single go:

Apply. At worst you learn that you're not. At best, you learn that you are.

This shows you're worth taking the shot.

If you thought outright, "This is below me, I MUST move up!" we'd probably be telling you your full of it.

Take the shot.

Apply. At worst you learn that you're not. At best, you learn that you are.

Technically, applying and getting the job will not tell you if you are skilled enough. But if you get the job, *then* you find out if you are skilled or not. My guess is that you are not, but you can grow into it. Why would you take a position that offers no challenges anyway? Go for it.

Ditto on the "go for it", nobody every got a promotion by being a wallflower.

Cheers guys. My brain is so fragged at the moment, that everything feels like its in this haze/fog. To top it off, I've had this awesome tension headache for the last 6 or so days - mostly because I'm not looking forward to this weekend; I have to roll out a specific antivirus to a client because thats what they wanted, simply because it has an ISA plugin (swear to god, the ISA plugin sold it).

I need to take some time off and apply for jobs, this is starting to become ridiculous. I've worked 20% more hours than anyone else around me.

I have that same feeling every day, either that I'm going to be fired before the day is out, or that I should quit because I'm dragging the company down and they should replace me with someone more competent. You should see my end-of-year self-reviews...

Hah, you too? Its not so bad now, since I dont have the time to sit and think about it. It used to be that I'd expect to be fired at any given point; that management would suddenly wake up and go "hey wait a sec, we've been duped!". Now I feel a little safer because I have so much knowledge in my head that I want to document, but they wont allow me to since its not billable. I figure they should let me finish writing it up, it helps everyone in the long run anyway...

A little something for everyone to ponder:
Dunning–Kruger effect

Clicked on it and got a sense of deja vu. I must have clicked on it way back when it was posted before. How I didnt remember is beyond me.

--

I also realised that I'm holding myself to a weird standard; I'm reading these forums and thinking to myself that I must be behind the 8 ball because quite a few of the posters are crazy good at what they do. Take scorp508 for example, reading his posts you see how fucking good he is with Exchange, and I'm sitting there comparing going "wow, I dont know anything". Problem is, he lives/breathes Exchange so much that he got a job with MS.

Sometimes its hard not to compare to the talent in this forum, just have to remember that you cant be the best at everything.

 

[WarheadsSE]

I also realised that I'm holding myself to a weird standard; I'm reading these forums and thinking to myself that I must be behind the 8 ball because quite a few of the posters are crazy good at what they do. Take scorp508 for example, reading his posts you see how fucking good he is with Exchange, and I'm sitting there comparing going "wow, I dont know anything". Problem is, he lives/breathes Exchange so much that he got a job with MS.

Sometimes its hard not to compare to the talent in this forum, just have to remember that you cant be the best at everything.

I use this forum to keep my 'tude in check sure. It's a good reminder that yeah, I might be damn good at some things, this little world of IT is a whole lot bigger than it looks. Plus, it's a mild confidence boost when I can help out someone I've never even met without getting hands on with their equipment.

 

[sryan2k1]

Finding new POTS HPNA gear is a huge pain in the rear. I've only found one new router/bridge in wide availability and that's a Corinex system, which is likely a rebadge of some other products I've found to be vaporware in the USA but maybe available in other countries.

Why don't you buy a few cheap VDSL modems?

 

[Danger Mouse]

i've used these guys before to get me out of jams (how the @#$% do you get ethernet above a light fixture in the middle of the hotel ball room? oh the facilities guys can get us one pair of cat3... grreeaatt)

Thanks, unfortunately, I'm limited to Cisco for networking gear. I would dearly LOVE to purchase HP networking gear, but that won't make it past my boss. i.e. HP's PoE switch+ wireless access point in a wallplate would be a perfect fit for our older buildings, but that's not going to happen because of the Cisco requirement.

I would also have to deal with the networking contractor's requirements, which may preclude work on non-Cisco gear.

Finding new POTS HPNA gear is a huge pain in the rear. I've only found one new router/bridge in wide availability and that's a Corinex system, which is likely a rebadge of some other products I've found to be vaporware in the USA but maybe available in other countries.

Why don't you buy a few cheap VDSL modems?

Does Cisco make VDSL modems?

I'm also up against a vendor whose staff aren't technical enough to know what would be a good alternate product to fill a need.

 

[sryan2k1]

Yes, they do but I'm not sure in what packages, here is one example

http://www1.cisco.com/en/US/prod/collat ... _Item.html



Also, I don't know if they sell modems that can run point to point, they might need a DSLAM, but it's worth checking into.

 

[ronelson]

I also realised that I'm holding myself to a weird standard; I'm reading these forums and thinking to myself that I must be behind the 8 ball because quite a few of the posters are crazy good at what they do. Take scorp508 for example, reading his posts you see how fucking good he is with Exchange, and I'm sitting there comparing going "wow, I dont know anything". Problem is, he lives/breathes Exchange so much that he got a job with MS.

Chances are that if you lived and breathed Exchange, you would be pretty good at it, too. When something is only a part of your job, even a large part, it is difficult to focus on it. I take pride not in my specific knowledge, but in my innate skills. I know how to isolate a problem, which works just as well on something I know intimately as it does on something I have never seen. I know how to work with others to not only get what I want, but to make them feel they got what they wanted. I can work with customers during a crisis and keep them from panicking. I can pick up new material quickly and have learned to recognize the limits of my newfound knowledge before I *cause* a crisis.

I know that I am not the bee's knees when it comes to Exchange, but I also know that I can learn when I need to and keep from breaking it otherwise. That is far more important than any specific knowledge. If you have even a fraction of those skills, you should hold your head up high. Your humility also helps - as you can see, I have none

 

[Danger Mouse]

Yes, they do but I'm not sure in what packages, here is one example

http://www1.cisco.com/en/US/prod/collat ... _Item.html

Also, I don't know if they sell modems that can run point to point, they might need a DSLAM, but it's worth checking into.

Thanks for that. I also found some consumer grade ADSL/SDSL modem equipped routers. Again the problem would be requiring point to point use. Time for more research.

 

[Uhlek]

Also, I don't know if they sell modems that can run point to point, they might need a DSLAM, but it's worth checking into.

I've run Cisco routers back-to-back using ADSL and SDSL before and it's worked fine. Never tried their VDSL modules though.

 

[Danger Mouse]

Also, I don't know if they sell modems that can run point to point, they might need a DSLAM, but it's worth checking into.

I've run Cisco routers back-to-back using ADSL and SDSL before and it's worked fine. Never tried their VDSL modules though.

Thanks. Got any specific models?

EDIT: Okay, I fed the salesperson some specific models and told them to deal with Cisco, if they can't answer the question internally. I have ZERO time to waste on phone calls as we're doing an IT storage room move and I'm now covering for our entire IT staff by myself.

 

[sryan2k1]

I've never used the Cisco xDSL modules, but Can't you call your VAR and ask them what would fit best for your use case?

 

[Danger Mouse]

I've never used the Cisco xDSL modules, but Can't you call your VAR and ask them what would fit best for your use case?

I would refer you back to my previous comment about our chosen VAR's salesteam's technical skills. If I don't give them a specific part number, you can forget about it.

Yes, we are about to replace them with CDW-G for various other reasons.

 

[dodexahedron]

Yes, we are about to replace them with CDW-G for various other reasons.

You'll love that decision (truly).
CDW is one of the best vendors I've ever worked with, if somewhat pricy at times.

 

[sryan2k1]

Ah, I glanced over that part.


I love my CDW team. They get it right, without fail, every time.


Well worth the price.

 

[dodexahedron]

I love my CDW team. They get it right, without fail, every time.

And if they tell you they'll call either way, when something's iffy, by god they'll call.
My account rep has stayed after hours more than once to make sure things happened for us.

Definitely worth the premium, if you deal with them over the phone.
If all you do is online ordering, then you're not going to get much extra value from CDW other than same-day processing and shipping.

 

[afidel]

Ditto on the CDW love, my inside sales team has been great through two different rep's. My outside team has been a bit more iffy, but if I can't get an answer in time from them I can always call my inside team for a quick response.

*edit*
It's kind of funny, I was taking a survey the other day and they had direct order (CDW-G, etc) and VAR listed as options of who you purchase through, I selected VAR because to me CDW adds more V to VAR then anyone else I've ever dealt with.

 

[Mittens T. Cat]

That one of our VMs is causing a PSOD!

We had a PSOD yesterday afternoon, was kinda freaky. I moved the important stuff over to another ESX host, and the non-important stuff over to a third ESX host.

Well don't you know it, come in this morning and the third ESX host with the 'non-important' stuff (aka not mission-critical but boss-critical) has a PSOD! Not only that, but when we boot everything back up (on OTHER hosts), one of the VMs has been critically wounded in the registry or something. This fixed it: http://communities.vmware.com/thread/234841

For a while there though, having our OCS main server down was kinda tense.

We've had two suggestions so far.

One, the non-critical VM we are suspecting is SharePoint 2010, and a consultant was in who fixed the crawl service just yesterday morning. While it was crawling away, I had been trying to get some backups out of the way of our file server, on top of the regular production chatter - the suggestion is that the one array all of this was happening on kinda freaked out and dropped some iSCSI connections, then PSOD'd.

Two, there is something wrong with the server hardware. I find that hard to believe however - two different servers (1950 II and 1950 III) having PSODs within hours of each other?

Anyways...long day.