what did you learn today? (part 2)

[Danger Mouse]

4 years into this campus modernization project and I uncover yet more hidden IT landmines.

The infoblox it turns out, was never properly configured for DNS integration with our DCs. That's now fixed.

Our APC ISX IfrastruXure Central server was incorrectly set up at the beginning and has been kind of limping along. That's mostly fixed.

Why do my current projects take 4x longer than they should? Because every time I start a new project, inevitably, I'll find a dependency on a project that was done wrong or never completed and THAT project or project(s) have to be done first in order for the current project to move ahead.

 

[Technarch]

Only 4x? You must be pretty good.

 

[Danger Mouse]

Technarch[/url]":1eyahv4l]Only 4x? You must be pretty good.

I am the all singing, all dancing, clap of the world!

 

[ncrand]

TIL that WSUS uses up waaaay more disk space than it thought it would. Also, it's pretty easy to move the WSUS database using wsusutil in the command line.

 

[Rick25]

Bad MS patch last week

http://www.theregister.co.uk/2013/04/19 ... pair_disk/

Bootable disk http://www.microsoft.com/en-us/download ... x?id=38435 for uninstall if you can't get into Windows.
We've seen at least 1 case of this this week.

 

[afidel]

TIL that I can get latency to creep up just a little on the new 3Par 7400 if I hit it with 600+MB/s of svotion traffic while backups are going on =) All in all I'd say I'm very impressed by it, now that I've moved the first production VM's over it will be fun to watch AO moving hot blocks up to SSD and see how effective that is at taking real workload IOPS off the 10k disks.

 

[Xon]

Rick25[/url]":mvbo65bz]Bad MS patch last week.

This is related to Kaspersky AV.

That said, the last patch cycle nuked the development database my group was working against. Trivial files like ci.dll and ntoskrnl.exe where corrupted zero length files after chkdsk was run. That ended up being a rebuild job.

 

[Dqua]

PaveHawk-[/url]":1x8mfkkh]Fuck Symantec and EV. I need to find some decent tools that rehydrate EV data back into Exchange so I can get rid of this shit. Quick audit of our client's licensing has revealed that their licenses no longer support Journal mailboxes. You have to buy an additional license for that. Apparently the license change occurred in EV9 or 10 (some of our clients are only now looking at their upgrade cycle) and will hurt them significantly.

[...words about Black_Obsidian getting screwed over...]

I was reading this, thinking "Poor DM, not again". Lo and behold, its someone else!

Welcome to the club.

PaveHawk-: shoot me a PM, I support an extensive EV install and may have some ideas.

 

[Rick25]

Xon[/url]":3gnotluv]

Rick25[/url]":3gnotluv]Bad MS patch last week.

This is related to Kaspersky AV.

That said, the last patch cycle nuked the development database my group was working against. Trivial files like ci.dll and ntoskrnl.exe where corrupted zero length files after chkdsk was run. That ended up being a rebuild job.

We're running Forefront and experienced a similar crapout on a reboot. I'll know for sure on Monday when we try the bootable ISO to recover the drive.

 

[Sunner]

Danger Mouse[/url]":2j3mq8vz]4 years into this campus modernization project and I uncover yet more hidden IT landmines.

The infoblox it turns out, was never properly configured for DNS integration with our DCs. That's now fixed.

Our APC ISX IfrastruXure Central server was incorrectly set up at the beginning and has been kind of limping along. That's mostly fixed.

Why do my current projects take 4x longer than they should? Because every time I start a new project, inevitably, I'll find a dependency on a project that was done wrong or never completed and THAT project or project(s) have to be done first in order for the current project to move ahead.

Stop complaining.
One of my projects has been underway for the better part of two years, me and some other techs know what to do and have been feeling pretty confident that we can pull it off despite some landmines along the way. Management however have had meetings. Then some more meetings. And some more. And so on. We're now closing in on the point of no return where a LOT of shit will unavoidably hit the fan. So they want to hold meetings to determine what to do about the upcoming disaster.
Another project was to simply setup a file transfer across a corporate WAN, nothing fancy, just send some files, receive receipts, all inside the same company. It drew on for ~1 year and took weeks of manpower from me alone.
A third project involves a web based system that does a bit of everything. Problem is, no one dares touch this system, it's an unholy mess of ASP. DNS? Bah, IP addresses spread all over the code base(yeah, not even a variable somewhere), no documentation, a database running on a Windows 7 box, etc. We're trying to move some functionality out of it and the designers of the new system don't even know the basics of SSL(as in, they have no idea what the CN in a cert is even for), SSL is used extensively in the new system.

FML.

Sorry about that, just had to vent a little...

 

[sryan2k1]

DNS? Bah, IP addresses spread all over the code base(yeah, not even a variable somewhere)

I ran a pfSense VM at my last job to NAT between an application like this and the rest of the network. Stupid.

 

[Danger Mouse]

Sunner[/url]":31tibyw9]

Danger Mouse[/url]":31tibyw9]4 years into this campus modernization project and I uncover yet more hidden IT landmines.

The infoblox it turns out, was never properly configured for DNS integration with our DCs. That's now fixed.

Our APC ISX IfrastruXure Central server was incorrectly set up at the beginning and has been kind of limping along. That's mostly fixed.

Why do my current projects take 4x longer than they should? Because every time I start a new project, inevitably, I'll find a dependency on a project that was done wrong or never completed and THAT project or project(s) have to be done first in order for the current project to move ahead.

Stop complaining.
One of my projects has been underway for the better part of two years, me and some other techs know what to do and have been feeling pretty confident that we can pull it off despite some landmines along the way. Management however have had meetings. Then some more meetings. And some more. And so on. We're now closing in on the point of no return where a LOT of shit will unavoidably hit the fan. So they want to hold meetings to determine what to do about the upcoming disaster.
Another project was to simply setup a file transfer across a corporate WAN, nothing fancy, just send some files, receive receipts, all inside the same company. It drew on for ~1 year and took weeks of manpower from me alone.
A third project involves a web based system that does a bit of everything. Problem is, no one dares touch this system, it's an unholy mess of ASP. DNS? Bah, IP addresses spread all over the code base(yeah, not even a variable somewhere), no documentation, a database running on a Windows 7 box, etc. We're trying to move some functionality out of it and the designers of the new system don't even know the basics of SSL(as in, they have no idea what the CN in a cert is even for), SSL is used extensively in the new system.

FML.

Sorry about that, just had to vent a little...

I see your ASP, hard coded IPs and raise you with:

-half baked SAP system implementation
-40 Year old codebase in our OpenVMS cluster (HP Superdome servers)
-Windows XP on the desktop (we're finally starting migration! wheeeee)
-.edu shenanigans
-Xerox MFD contract shoved down our throats, with no end of trouble until the last 6 months of fairly stable drivers

I could go on and on and on, but I don't care to rehash the last few years of posts

---

In other news, something screwed up a bit with the interview for a MEDIA COMPANY for a position that's 95% less work, but 40% more pay. The manager doing the actual hiring didn't get my landline number, AND I didn't get a confirmation of the interview slot until I called in.

And so of course the hiring manager's call went to my voicemail. Returning his call resulted in no answer other than voicemail.

Oh well, reschedule or it goes away.

I'm not that plussed about it, because the job seems to entail ONLY ANTI-VIRUS, which seems to be a career limiting move.

---

...TIL Infoblox's stupid context sensitive toolbar pane makes it a HUGE pain to get to where you want if you don't remember exactly what you want. AND, there's enough repetition to make your head hurt. It's not as terrible as it used to be, but it's certainly a far cry from where it should be.

 

[kennedye]

Today I learned (thanks to LinkedIn) that DM's "celebrating" 13 years on the job!
In that spirit, "congratulations" I guess? :bigdumbgrin:

 

[Rick25]

MDOP 2013 with MBAM 2.0 has been released. http://blogs.windows.com/windows/b/busi ... -2013.aspx

 

[ramases]

Danger Mouse[/url]":6meh0gwo]I'm not that plussed about it, because the job seems to entail ONLY ANTI-VIRUS, which seems to be a career limiting move.

I'm guessing this is not on the reverse-engineering part of the industry. And managing AV installations all day -- never mind the career-limiting potential, I wouldn't wish that kind of job on my worst enemy.

 

[hernias]

Standing up an elderly IBM AIX system (P5). It's good that I'm not too busy because this thing takes forever to do anything.

 

[Mike Bridge]

that i need to update the certificate(s) for vcenter 5.1's Single Sign On component. and it does not look pleasant:

http://kb.vmware.com/selfservice/micros ... Id=2035011
which refers to:
http://kb.vmware.com/selfservice/micros ... Id=2041600
(along with http://kb.vmware.com/selfservice/micros ... Id=2044696 )

 

[Danger Mouse]

ramases[/url]":eq8rb5vm]

Danger Mouse[/url]":eq8rb5vm]I'm not that plussed about it, because the job seems to entail ONLY ANTI-VIRUS, which seems to be a career limiting move.

I'm guessing this is not on the reverse-engineering part of the industry. And managing AV installations all day -- never mind the career-limiting potential, I wouldn't wish that kind of job on my worst enemy.

Yah, it's a large install (16000? seats), but it just sounds like a career ending move.

I don't know. Wound up missing each other on the call, so I'm tempted to just say screw it.

---

EDIT: :facepalm: fixing the DNS on ye olde infoblox, broke the PXEBoot for Altiris. I need to revert the changes and step through them one at a time

---

EDIT2: and I need to make a powershell script to deal with the broken output for the newuser file, before I can do the import/enable and then mail enable. YAARRGGGGHGHGHGHGHGHG

---

EDIT3: and I can't move on the freaking Ex2010 install, until the DNS is right on the money!

---

EDIT4: And another dependency, I'm not allowed to move mailboxes, until the backup server is fully functional with Ex2010 Sp3. Which is not supported in the current version, AND of course I wuld have to stop backing up the current mail server before I could pop to the new one, UNLESS I roll out the new version first.

 

[mdporter]

I'm guessing this is not on the reverse-engineering part of the industry. And managing AV installations all day -- never mind the career-limiting potential, I wouldn't wish that kind of job on my worst enemy.

Yah, it's a large install (16000? seats), but it just sounds like a career ending move.

I don't know. Wound up missing each other on the call, so I'm tempted to just say screw it.

I am about to roll out an AV upgrade companywide for our 3,000+ clients, spending time fixing the broken ones during our freeze period before I get the green light to push the upgrades out.

I could not and would not do only AV full time. Because there is no way to ever be "finished"

 

[Danger Mouse]

mdporter[/url]":2hhddh8a]

I'm guessing this is not on the reverse-engineering part of the industry. And managing AV installations all day -- never mind the career-limiting potential, I wouldn't wish that kind of job on my worst enemy.

Yah, it's a large install (16000? seats), but it just sounds like a career ending move.

I don't know. Wound up missing each other on the call, so I'm tempted to just say screw it.

I am about to roll out an AV upgrade companywide for our 3,000+ clients, spending time fixing the broken ones during our freeze period before I get the green light to push the upgrades out.

I could not and would not do only AV full time. Because there is no way to ever be "finished"

2200 stations here.

Yah, it would have to be stupid money to go. I don't think the $95k on offer will cut it.

 

[Pontiphex]

hernias[/url]":2a0t6q9s]Standing up an elderly IBM AIX system (P5). It's good that I'm not too busy because this thing takes forever to do anything.

I have some fond memories of AIX. Installing the OS from tape...

IRIX would be a runner up as far as nostalgia computing...

 

[sporkme]

TIL many things:

-DM's post about Infoblox makes me note that today while Googling for something DNS-related I discovered the Cricket Liu is in fact a man, not a woman. This has generated a few uncomfortable feelings.
-That while turning up IPv6 in a datacenter where someone else is doing all the routing is cool, turning it up on your LAN and then also getting a BGP session going is more cool and satisfying. "Hello World, I have 2^96 IPs now."
-That turning on IPv6 in IOS leaves your ssh and snmp open to the world until you create an ipv6 access list and apply it to your vtys and snmp config.
-That configuring ip6.arpa in BIND is a royal PITA.
-That this site is 100% work safe, but makes me giggle like a schoolgirl: http://cornholeworldwide.com/
-That what I thought was a dog hair on the bottom of my laptop screen is in fact a crack.

edit: Also my post count after this post made me thing of firewire and nothing else.

 

[Blacken00100]

...that Xen documentation expects you to already be a Xen expert before you start doing things. Fortunately I'm just screwing around in my home lab on my own time, because it's taken me three hours to get Baby's First VM started. Started building, that is, not started-started.

Also, the XCP ISO from the Xen folks themselves won't boot off a USB drive, so I ended up doing it manually on Ubuntu. Which in the long run is probably better, as I'll understand more of what the eff I'm doing.


Makes more sense to me than KVM does, though, I'll give it that!

 

[Danger Mouse]

OMG, it wasn't the Infoblox after all! Stupid Altiris.

Just work like you're supposed to, dammit!

Reverted back to the fixed up version I had already fixed up with proper AD-DNS integration, so the durn zones would update. Now to get the reverse lookup zones imported

EDIT: And, oh yeah, that's already done for the parts it's supposed to be done! OMG, it's mostly working like it's supposed to right now!

And now's the time on Sprockets when we dance!:

https://www.youtube.com/watch?v=ASO_zypdnsQ

 

[Danger Mouse]

Blacken00100[/url]":mjhtspo6]...that Xen documentation expects you to already be a Xen expert before you start doing things. Fortunately I'm just screwing around in my home lab on my own time, because it's taken me three hours to get Baby's First VM started. Started building, that is, not started-started.

Also, the XCP ISO from the Xen folks themselves won't boot off a USB drive, so I ended up doing it manually on Ubuntu. Which in the long run is probably better, as I'll understand more of what the eff I'm doing.


Makes more sense to me than KVM does, though, I'll give it that!

Yes, better to build it to understand it, but if you need to hammer away at things post-build process:

http://www.labguides.com/autolab/

Why reinvent the wheel? Then again, wtf should I try to build the ad-user import script when my colleague has one already?

So I can learn more, I suppose. Dammit.

EDIT: And zee script is done. Trim() needs the stuff in the parantheses defined :facepalm:

So to get rid of end spaces Object.TrimEnd(" ")

Or $Varname.TrimEnd(" ")

Or $Varname = $Varname.TrimEnd(" ")

but I'm not certain the assignment is needed. I need to doublecheck.

I need to do some cleanup on the script to enable the not so strong default passwords and a few other niceties AND to not dupe accounts based upon EmpNo, but that's another task.

I can probably condense it down a bit and remove a step or two, but only AFTER I've got all the intended functionality in place.

 

[ronelson]

I could not and would not do only AV full time. Because there is no way to ever be "finished"

You should try security!

...that Xen documentation expects you to already be a Xen expert before you start doing things.

I find it quite rare that the situation is the opposite. It would surprise me much more.

 

[sryan2k1]

I'm really not sure why anyone would suffer through the horrors that is Xen when ESXi is free.

 

[Big Wooly Mammoth]

ronelson[/url]":2xix73i2]

I could not and would not do only AV full time. Because there is no way to ever be "finished"

You should try security!

I am starting to understand this at a very deep level.

 

[Whittey]

Theoretically we can decom this next month. Yay!

 

[gradster]

Mike Bridge[/url]":mx4vrwm5]that i need to update the certificate(s) for vcenter 5.1's Single Sign On component. and it does not look pleasant:

http://kb.vmware.com/selfservice/micros ... Id=2035011
which refers to:
http://kb.vmware.com/selfservice/micros ... Id=2041600
(along with http://kb.vmware.com/selfservice/micros ... Id=2044696 )

Hopefully this can make it easier for you.

http://blogs.vmware.com/vsphere/2013/04 ... lable.html

 

[Blacken00100]

ronelson[/url]":schg079x]I find it quite rare that the situation is the opposite. It would surprise me much more.

I tend to find most stuff I work with to be quite well-documented. But I stay out of hardware and overly annoying server-y things, most of the time; right now, though, I need some local infrastructure for dynamically spinning up additional hosts and don't want to pay AWS or Rackspace for the privilege.

sryan2k1[/url]":schg079x]I'm really not sure why anyone would suffer through the horrors that is Xen when ESXi is free.

ESXi refuses to install on my test machine (which is weird given that it's all pretty vanilla hardware and nothing else seems to mind), so my options were a bit constrained.

 

[ronelson]

Whittey[/url]":2ord0iyv]Theoretically we can decom this next month. Yay!

For grins:

old-ass-router>sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-IK2S-M), Version 12.XXX, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Tue 03-Sep-02 22:58 by kellythw
Image text-base: 0x80008088, data-base: 0x80E23440

ROM: System Bootstrap, Version 12.XXX, RELEASE SOFTWARE (fc1)

old-ass-router uptime is 7 years, 6 weeks, 2 days, 21 hours, 2 minutes

I'm mildly surprised it's even in the v12 train. I'm hugely surprised we haven't had a power hit in all that time. Kudos to you, kellythw!

 

[WingMan]

Danger Mouse[/url]":hagh1n52]OMG, it wasn't the Infoblox after all! Stupid Altiris.

Just work like you're supposed to, dammit!

Altiris and work properly are oxymorons. It's gotten worse since Symantec took it over.

 

[ImRaptor]

sryan2k1[/url]":1phaenbd]I'm really not sure why anyone would suffer through the horrors that is Xen when ESXi is free.

Depends on if you are talking strictly Xen or XCP.
XCP is a much better platform when comparing free offerings.

 

[Whittey]

ronelson[/url]":2t0nqb98]
For grins:

old-ass-router>sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-IK2S-M), Version 12.XXX, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Tue 03-Sep-02 22:58 by kellythw
Image text-base: 0x80008088, data-base: 0x80E23440

ROM: System Bootstrap, Version 12.XXX, RELEASE SOFTWARE (fc1)

old-ass-router uptime is 7 years, 6 weeks, 2 days, 21 hours, 2 minutes

I'm mildly surprised it's even in the v12 train. I'm hugely surprised we haven't had a power hit in all that time. Kudos to you, kellythw!

I was going more with the 1993 vintage OS running on non-y2k compliant hardware that's used once or twice a day even now. I'm almost sad that it won't make it to drinking age here in the US.

 

[xcrunner529]

Blacken00100[/url]":n9lhq8xz]

sryan2k1[/url]":n9lhq8xz]I'm really not sure why anyone would suffer through the horrors that is Xen when ESXi is free.

ESXi refuses to install on my test machine (which is weird given that it's all pretty vanilla hardware and nothing else seems to mind), so my options were a bit constrained.

Is it because of your NIC or the SATA controller? Before I bought a vendor server I bought an Intel NIC for my generic HP Desktop and it then let me install and detected everything.

 

[Blacken00100]

NIC, yeah; it's Realtek. I have XCP (or something vaguely resembling it) running now, though, so I'm not gonna touch it.

 

[Danger Mouse]

Blacken00100[/url]":3hwj3776]NIC, yeah; it's Realtek. I have XCP (or something vaguely resembling it) running now, though, so I'm not gonna touch it.

Realtek? yah...no go unless you build your own iso.

 

[jshiplett]

There's some support for RTL chipsets in later builds of ESXi. If I'm not mistaken, at least the 8111E works.

 

[Danger Mouse]

euri[/url]":vlbs70p6]There's some support for RTL chipsets in later builds of ESXi. If I'm not mistaken, at least the 8111E works.

http://www.kendrickcoleman.com/index.ph ... ere-5.html

That would support your assertion. I would expect the 8111E would work, but earlier models not so much.

ESXi v5 and later should support it, but no guarantees.

http://communities.vmware.com/message/2067708

Realtek 8168
Realtek 8169
Realtek 8111E

No 8100A/B/C probably

--

Blacken,

if you need an Intel gbit NIC, pay for shipping and I'll gladly shuffle one off to you. I can probably find you a dual port NIC if so desired.