what did you learn today? (part 2)

[Sunner]

stash[/url]":2by0c2uz]

sryan2k1[/url]":2by0c2uz]I don't have any real issues with 5.1. Why all the hate?

I don't either, except for the web client. It's slow as balls, unintuitive and requires flash. Given that they are going to take away the C# client completely in the near future, they need to do better.

I hate the C# client as well, maybe I'm doing it wrong but to me it's always been slow as shit, has lots of general UI weirdness, and is just all around annoying. Still beats the web client but that's not saying much.

TIL that apparently the proper response to a security vulnerability(CVE-2013-2094 in this case, the Linux perf_event exploit) is to go into full on batshit panic mode, hold meetings with mid-high tier bosses, etc etc, rather than following a laid out patch policy which I would assume includes provisions for patching high importance vulnerabilities. Of course there I'm making the assumption that a patch policy even exists to begin with, maybe I'm being naive.

 

[hawkbox]

Oh the .exe sucks balls too. But it's leagues better than the web client. I swear they're trying to build Microsofts market share with every release.

 

[ramases]

euri[/url]":33zbwsnw]Also, if you were waiting to upgrade to vCenter 5.1u1 until the ridiculous AD group bug was resolved, it's now been resolved in update 1a.

Finally!
Thanks for the headsup, euri.

Sunner[/url]":33zbwsnw]TIL that apparently the proper response to a security vulnerability(CVE-2013-2094 in this case, the Linux perf_event exploit)

That one is a real doozy, both because it has lain dormant for so long and because of the funny way it is exploited (signed int32 to unsigned int 64 to unsigned int 32 to bypass a bounds check that incorrectly only checks for an upper (but not a lower) bound on the initial int32).
There was also a mitigation available in the RH bugtracker that would have at least blocked the public proof-of-concept.

I'm already grateful for the decision of Debian to not backport that set of changes to the perf_event-system back to 2.6.32 (unlike RH, which did backport the event system including the vulnerability from 2.6.37); really breathed a sigh of relief on that one.

 

[Danger Mouse]

Wheeeee, had a few hours to kill, so I'm about 80% complete on my SCCM 2012 build. A few more things and done. Then to build WDS/MDT on top of that and do systems imaging that way instead of Altiris when possible.

AND, will rebuild Altiris on proper 64bit OS and newer SQL.

But, after I finish up the last nasty bits of the Ex2010 setup, which is taking forever due to the "no downtime allowed" clause by my manager.

That plus the contractor for our HQ org not providing a basic diagram of their desired setup, means I'm going to have to guess at it

---

Speaking of which, can I use one install of TMG 2010 to be CAS load balancer as well as SMTP/Edge Transport load balancer? Assume extremely light traffic (less than 100 valid emails per hour on average via SMTP) Or do I need a separate instance? Or a HA/clustered pair to load balance the load balancer?

 

[xcrunner529]

dlp[/url]":1eqi9gay]

xcrunner529[/url]":1eqi9gay]

sryan2k1[/url]":1eqi9gay]They turned UAC off on most of the 2008R2 servers because they didn't like dealing with the prompts or the access restrictions on folders. I've ensured it stays on for everything I touch.

We leave it on, but MS really left it half broken for people. One big problem I run into daily is mapped drives not carrying over in elevated apps and command prompts.

What specifically do you need to run elevated daily? I'm an admin, leave it on, and rarely get it daily among all the machines I support. Alternatively, launch the executable from UNC paths instead of mapped drives.

Umm I am setting up new servers or installing new applications very frequently and media is stored on network drives. Microsoft's own ISO mounter in 2012 doesn't work with it without copying the ISO locally first. Do you really carry around CDs still?

 

[sryan2k1]

I've never tried it, it won't mount from a UNC path?

 

[afidel]

xcrunner529[/url]":2adgnslo]

dlp[/url]":2adgnslo]

xcrunner529[/url]":2adgnslo]

sryan2k1[/url]":2adgnslo]They turned UAC off on most of the 2008R2 servers because they didn't like dealing with the prompts or the access restrictions on folders. I've ensured it stays on for everything I touch.

We leave it on, but MS really left it half broken for people. One big problem I run into daily is mapped drives not carrying over in elevated apps and command prompts.

What specifically do you need to run elevated daily? I'm an admin, leave it on, and rarely get it daily among all the machines I support. Alternatively, launch the executable from UNC paths instead of mapped drives.

Umm I am setting up new servers or installing new applications very frequently and media is stored on network drives. Microsoft's own ISO mounter in 2012 doesn't work with it without copying the ISO locally first. Do you really carry around CDs still?

No, I upload the ISO to a VMWare datastore and mount it through the VM, doesn't everyone =)

 

[ramases]

afidel[/url]":3op34gvf]No, I upload the ISO to a VMWare datastore and mount it through the VM, doesn't everyone =)

And if its still on bare metal we just use the virtual media support of the service processor.

 

[Entegy]

I don't see mapped network drives not showing up in CMD as an issue really. Elevating the command prompt puts you in a different security context or even a different user account.

 

[Zapman987]

That our facilities team doesnt give jack for fixing stuff thats priority, and would rather follow procedure and regs.

Story:
We had a breaker trip which killed 2 strips (root cause, should have only had 1 on it) which then overloaded a third strip. I saw half a rack worth of stuff die, and figured correctly it was a power issue, ran over to said network room, and reset the third strip. Then called local L1 guys to confirm they couldnt do anything since the next bit was a breaker, and that I needed to call facilities. They confirmed, so I called facilities hotline. They give me a case, email me the case number, and say someone will be over.

20 minutes later, no ones shown up. Im thinking ok, maybe they got lost. Call hotline back, they are like well theyve been dispatched, heres the dudes name. I pull directory, find him, call his cell. He hasnt even gotten the case. Tell him whats up, he gets here. We fix breaker, and find the dual wired strips to one breaker. Fix that. Tech leaves.

His boss calls me while Im restoring servers. Tries to chew me out for calling the tech's cell. Im like dude, hes in directory, I didnt say anything other than to figure out where hes at and if he was lost. Not pissed at him or nothing. Boss dude is like well you shoulda called me. Then tries to bitch at me for not having an IT case open on this issue (i ran over so fast, didnt stop to open one). Then he looks at the facilities case ticket, its a p5 (a week time to fix). Hes like this was a priority issue right? I respond ya, it was at least a p2 if not a p1 issue. He finally shuts up a little, and is like well lemme do some investigation, Ill be in touch.

If he comes back to me, he can talk to my manager, and some VPs of IT on why he wanted to hold up a p1 restoration...

 

[afidel]

Zapman987[/url]":1womo6oh]That our facilities team doesnt give jack for fixing stuff thats priority, and would rather follow procedure and regs.

Story:
We had a breaker trip which killed 2 strips (root cause, should have only had 1 on it) which then overloaded a third strip. I saw half a rack worth of stuff die, and figured correctly it was a power issue, ran over to said network room, and reset the third strip. Then called local L1 guys to confirm they couldnt do anything since the next bit was a breaker, and that I needed to call facilities. They confirmed, so I called facilities hotline. They give me a case, email me the case number, and say someone will be over.

20 minutes later, no ones shown up. Im thinking ok, maybe they got lost. Call hotline back, they are like well theyve been dispatched, heres the dudes name. I pull directory, find him, call his cell. He hasnt even gotten the case. Tell him whats up, he gets here. We fix breaker, and find the dual wired strips to one breaker. Fix that. Tech leaves.

His boss calls me while Im restoring servers. Tries to chew me out for calling the tech's cell. Im like dude, hes in directory, I didnt say anything other than to figure out where hes at and if he was lost. Not pissed at him or nothing. Boss dude is like well you shoulda called me. Then tries to bitch at me for not having an IT case open on this issue (i ran over so fast, didnt stop to open one). Then he looks at the facilities case ticket, its a p5 (a week time to fix). Hes like this was a priority issue right? I respond ya, it was at least a p2 if not a p1 issue. He finally shuts up a little, and is like well lemme do some investigation, Ill be in touch.

If he comes back to me, he can talk to my manager, and some VPs of IT on why he wanted to hold up a p1 restoration...

Today I was reminded of why I stay at my current midsized employer, so little bureaucracy is worth giving up a bit of potential income =)

 

[llib]

Zapman987[/url]":1j1anwml]That our facilities team doesnt give jack for fixing stuff thats priority, and would rather follow procedure and regs.

Damn, dude. That sucks. If it's during working hours, any power issue here would see either me or my trusty coworker (or more likely both) at the problem site in about 60 seconds. If it was bad enough for us to call the campus sparkies, they'd be here in about 5 minutes. If we were home, 15 mins for my coworker, 30 for me (I live farther away). I know those are accurate because during power outages, we're on-site in 15/30 checking the gensets, UPSes, chillers, etc. And the campus folks would have been there in about 5 mins, attempting infrastructure first aid until we got there.

In the 15 years I've been here, I can't even remember a rack power failure, although there have been a few server PSes that let loose the magic smoke... I don't know how you guys do it, man. My hat's off to you. And it sounds like your IT guy and the facility guy need to have some face-to-face time...


EDIT: Although now that I think about it, I realize not everybody can afford to have a couple of people who specialize in taking care of the data center facility as a full-time job -- racking, stacking, wiring, fibering, rolling racks around and dragging power cables wherever we need 'em, breaker resets when necessary (none so far), load testing the gensets, transferring power, yada, yada...etc. And to make things even better, our management knows how lucky they are...

 

[ferzerp]

Not to rain on the facilities hate here, but why did you have so much on the pair of circuits that a failure of one would cascade and cause an actual outage?

 

[kperrier]

Because they don't have proper facilities management to prevent such an occurrence!

 

[Rick25]

llib[/url]":uhh5b4eg]

....
In the 15 years I've been here, I can't even remember a rack power failure, although there have been a few server PSes that let loose the magic smoke... I don't know how you guys do it, man. My hat's off to you. And it sounds like your IT guy and the facility guy need to have some face-to-face time...

llib's epic magic smoke thread
viewtopic.php?f=21&t=72913

[edit]

Fixed linkage

 

[afidel]

Rick25[/url]":soh60y7q]

llib[/url]":soh60y7q]

....
In the 15 years I've been here, I can't even remember a rack power failure, although there have been a few server PSes that let loose the magic smoke... I don't know how you guys do it, man. My hat's off to you. And it sounds like your IT guy and the facility guy need to have some face-to-face time...

llib's epic magic smoke thread
viewtopic.php?f=21&t=7291

Something's wrong with the link, it goes to a post "Industry study claims strong copyrights fuel economic engine"

 

[sryan2k1]

My hacking skills have determined the last digit of the topic ID was lost in a sloppy copy and paste

viewtopic.php?f=21&t=72913


Guessing starting from 0 turned up the right thread.

 

[Graeme K]

Going back a page: all of our server admins are Domain Admins here, because it's always been that way. I know I need to clean it up someday, but, ugh. Someday.

 

[Widger]

Today I learned that there are many many things to think about when planning a migration from one virtualization platform to another.


We're moving from ESX 3.5 to Windows 2012 Hyper-V and we have two different storage arrays: an Equallogic PS6000 and a Netapp FAS2020 both using iSCSI.

It just dawned on me...MPIO software. Can one install MPIO software from different vendors on the same server? It's going to really suck if I can't.......

 

[ncrand]

TIL that if you use IMAP with Outlook, it doesn't save Contacts and Calendars in the PST that it creates when you add an account. It saves them in another PST (Called "Outlook Data File"). That would have been really great to know 24 hours ago.

 

[afidel]

Widger[/url]":3qjohgxv]Today I learned that there are many many things to think about when planning a migration from one virtualization platform to another.


We're moving from ESX 3.5 to Windows 2012 Hyper-V and we have two different storage arrays: an Equallogic PS6000 and a Netapp FAS2020 both using iSCSI.

It just dawned on me...MPIO software. Can one install MPIO software from different vendors on the same server? It's going to really suck if I can't.......

You can, though with 2008R2+ you probably don't need it.

 

[Danger Mouse]

sryan2k1[/url]":2die87po]My hacking skills have determined the last digit of the topic ID was lost in a sloppy copy and paste

viewtopic.php?f=21&t=72913


Guessing starting from 0 turned up the right thread.

For some reason, makes me think of ye olde credit card number gen, from back before they checked whether the account existed or not and not just the check sum match

 

[Widger]

afidel[/url]":2klhpt53]

Widger[/url]":2klhpt53]Today I learned that there are many many things to think about when planning a migration from one virtualization platform to another.


We're moving from ESX 3.5 to Windows 2012 Hyper-V and we have two different storage arrays: an Equallogic PS6000 and a Netapp FAS2020 both using iSCSI.

It just dawned on me...MPIO software. Can one install MPIO software from different vendors on the same server? It's going to really suck if I can't.......

You can, though with 2008R2+ you probably don't need it.

Thanks Afidel. Much appreciated!

 

[Zapman987]

ferzerp[/url]":11mspjwf]Not to rain on the facilities hate here, but why did you have so much on the pair of circuits that a failure of one would cascade and cause an actual outage?

That was the root. The DC mgmt folks I dont know why had 2 strips on one breaker. Thats against standards. The third strip is just bad local awareness.

The whole place is scheduled for a renovation which should fix these issues, but I expect this to happen again before its done.

 

[Danger Mouse]

That my manager has scheduled my data center move for July 4 through 7.

IOW, the long holiday weekend when my nice and nephew will be in town, is dead.

No choice.

 

[MilleniX]

Danger Mouse[/url]":2ms1zfna]That my manager has scheduled my data center move for July 4 through 7.

IOW, the long holiday weekend when my nice and nephew will be in town, is dead.

No choice.

Or, you have a new job lined up before then, and you simply refuse to lose that time with your family.

 

[Matt Wallis]

Danger Mouse[/url]":25wz5vsv]That my manager has scheduled my data center move for July 4 through 7.

IOW, the long holiday weekend when my nice and nephew will be in town, is dead.

No choice.

Insist that the scale of the project (hopefully it's as large as it sounds), is such that his presence will be required.

 

[Danger Mouse]

MilleniX[/url]":3t9b3ifn]

Danger Mouse[/url]":3t9b3ifn]That my manager has scheduled my data center move for July 4 through 7.

IOW, the long holiday weekend when my nice and nephew will be in town, is dead.

No choice.

Or, you have a new job lined up before then, and you simply refuse to lose that time with your family.

Already made that clear to him. "As long as I don't have another job by then".

He looked like this

But he's had that look a few times now, with all the phone and in person interviews I've been having.

He's been more upfront about making sure I get promoted within the next 2 months (has to be after July 1st). That would be a decent bump, but still not what I'd be making on the outside.

Matt Wallis[/url]":3t9b3ifn]

Danger Mouse[/url]":3t9b3ifn]That my manager has scheduled my data center move for July 4 through 7.

IOW, the long holiday weekend when my nice and nephew will be in town, is dead.

No choice.

Insist that the scale of the project (hopefully it's as large as it sounds), is such that his presence will be required.

He lacks the requisite skill set/knowledge to be of use on the move. If anything, I'm going to have to intervene and explain to them WTF they're wrong when they put stuff in various places.

i.e. Experience has shown me that switches (especially fabric switches) need to go middle of rack and not top of rack. Top of rack means they crash during CRAC outages, which means no control messages are sent to systems to poweroff as part of the emergency shutdown.

 

[Matt Wallis]

Danger Mouse[/url]":25n5ejis]

Matt Wallis[/url]":25n5ejis]

Danger Mouse[/url]":25n5ejis]That my manager has scheduled my data center move for July 4 through 7.

IOW, the long holiday weekend when my nice and nephew will be in town, is dead.

No choice.

Insist that the scale of the project (hopefully it's as large as it sounds), is such that his presence will be required.

He lacks the requisite skill set/knowledge to be of use on the move. If anything, I'm going to have to intervene and explain to them WTF they're wrong when they put stuff in various places.

i.e. Experience has shown me that switches (especially fabric switches) need to go middle of rack and not top of rack. Top of rack means they crash during CRAC outages, which means no control messages are sent to systems to poweroff as part of the emergency shutdown.

Heh, well then he should make himself useful and supply the pizza and beers

I've had a couple of employers in the past who did that, being fully aware of what they were asking, and their inability to get the work done without us, would come in and bring food and take care of issues arising. ie. running out to buy cables, cards, hard drives.

 

[Danger Mouse]

His response is to get the less productive IT types interns.

Me? I've had arguments with him about it for the last several years. He's finally going to get me one. I suggested to him he should get one for everyone.

He doesn't buy me pizza or whatever. He gives me more projects which delay projects in progress then complains about the delays....that he caused and forgets he caused them

Beyond that, I've got a crapton of leeway about just everything else, including what time I show up at the office or last minute unannounced vacation or whatever.

 

[Matt Wallis]

Danger Mouse[/url]":3p5c16ru]
Beyond that, I've got a crapton of leeway about just everything else, including what time I show up at the office or last minute unannounced vacation or whatever.

At least there are these things that make it a little more worthwhile. Although eventually, even these things are not enough. My partner had 6 weeks paid annual leave a year in a former job. This was nice, except she could only take them Jan - Feb, the rest of the year, she spent being the only person in a 3 person company that did any actual work. The other two were sales, and the company CEO. When she had a cold, the company stopped. A lot of stress when you're trying to meet deadlines for magazine publications and the like, and it really really messed up her health.

 

[dotorg]

How old are the kids? During an install of new equipment, I once saw a rack re-wired to /r/cableporn quality by the twelve-year-old daughter of my boss, after being given just basic instruction on the desired result. (We were a dial-up ISP and our "server room" was a utility trailer. Nothing inside looked remotely nice and cabling was an afterthought. Except for one beautiful rack in the middle of the room. Filled with 56k modems, new web servers and the monster array that held usenet. I wish I had photos.)

A boss responsible for picking and planning an inconvenient move has an obligation to show up and help on the tasks he can accomplish, even if that is just going to get spare parts or stripping the cardboard off the equipment in the parking lot.

 

[ronelson]

That my manager has scheduled my data center move for July 4 through 7.

IOW, the long holiday weekend when my nice and nephew will be in town, is dead.

No choice.

What happens, what really happens, if you just say no?

I once saw a rack re-wired to /r/cableporn quality

Ugh. Does no-one label the wires anymore? I have nightmares about some asshole going into nearly any of those pictured datacenters and unplugging two random wires of the same color and how stupidly long it would take to fix that.

 

[ramases]

ronelson[/url]":268ypsm2]Ugh. Does no-one label the wires anymore? I have nightmares about some asshole going into nearly any of those pictured datacenters and unplugging two random wires of the same color and how stupidly long it would take to fix that.

Alex, I wish to solve, please: "What is the timeframe it would take me to find one of the cable testers that grok lldp/cdp frames?"

Its of course not a replacement for proper labeling, but if you have to work with people that sometimes change patches without also changing the labels installing lldpd (also supports CDP) in send-only mode on ALL the servers might be a good idea.
We also use information gleaned from CDP to automatically check that actual deployment matches documentation.

 

[Danger Mouse]

ronelson[/url]":2hapetkx]

That my manager has scheduled my data center move for July 4 through 7.

IOW, the long holiday weekend when my nice and nephew will be in town, is dead.

No choice.

What happens, what really happens, if you just say no?

I once saw a rack re-wired to /r/cableporn quality

Ugh. Does no-one label the wires anymore? I have nightmares about some asshole going into nearly any of those pictured datacenters and unplugging two random wires of the same color and how stupidly long it would take to fix that.

If I say no, he hires contractors to half ass it, which I them have to clean up afterwards, making even more of a hassle.

That's why I'm trying to finish all major projects so I'll feel no guilt when I leave.

 

[SandyTech]

Today I learned that when your hardware upgrade and DC move project starts with a 6U server deciding to detach itself from its rails and have a go at your head, everything else seems like minor BS. Including DOA RAM & hard drives, failed PSUs, someone forgetting the cage nuts back at the old office (~90 minute round trip), getting locked out of the new DC because security disabled all of your team's badges, and finding out that mgmt forgot to pay the rent at the colo and they were yanking power from our rack (CRM, exchange and websites) in the middle of our DC move.

 

[ronelson]

Its of course not a replacement for proper labeling, but if you have to work with people that sometimes change patches without also changing the labels installing lldpd (also supports CDP) in send-only mode on ALL the servers might be a good idea.

Never heard of lldpd. Still not sure how that solves labeling issues, labeling issues should be solved without resorting to cable testers. It seems it only helps in the situation where the proper end is unplugged; if it's the server-side then you're just getting the switching device's name right?

 

[ramases]

ronelson[/url]"iraifu2]Never heard of lldpd. Still not sure how that solves labeling issues, labeling issues should be solved without resorting to cable testers.

Of course labeling issues should be resolved by proper labeling, but until we get there we need some way to deal with mis- or unlabeled cables because they are there. A lot of them. :-/

It seems it only helps in the situation where the proper end is unplugged; if it's the server-side then you're just getting the switching device's name right?

Correct, the CDP/LLDP frame will give you information about whatever is on the other end (in terms of system name and NIC name/port number) of the lose cable; of course you still need to figure out into which port or panel the cable should go to, but in any case having information about one endpoint is already better by far than having absolutely no information about any endpoint, no?

As for ascertaining where the cable should be connected to: For every production system there ought to be a record in our network management database (the same that also feeds our Nagios installation) about to which switchport it is connected, including any intermediary patch-panels, so I can just look the switchport up there and get the cabling path and what is supposed to be on the other hand.

If there is no such information in the database it is considered an undocumented patch; at that point it ceases to be my problem due to a departmental policy that any and all undocumented patches should be assumed to be not in use. Unused patches may also be removed without further comment two workdays after discovery.
In an emergency situation we could also possibly dig ourselves out of that "missing documentation"-hole by just looking at all unconnected, undocumented or obviously mislabeled patches in the general vicinity of the loose cable and see what cdp tells us there, but doing that too often would simply be to repeat the mistakes that led to the current mess in the first place.

 

[PaveHawk-]

One of the skills I'd consider key in any support person, is identifying performance bottlenecks and knowing what tools to use to do so.

A client has some issues with Exchange (2010) right now, and instead of running perfmon and looking at all the relevant statistics, the support team are simply moving the server from RAID6 to RAID10 storage.

Seriously, no attempt has been made to identify why Exchange is performing slowly. Its performing slowly, lets just move it to faster disk - that'll fix it!!

It simply perpetuates this ridiculous cycle where our clients are forced to buy more hardware to resolve issues that are due to technical incompetency than actual performance or capacity issues.

 

[jshiplett]

It simply perpetuates this ridiculous cycle where our clients are forced to buy more hardware to resolve issues that are due to technical incompetency than actual performance or capacity issues.

Feature, not a bug.