what did you learn today? (part 2)

[M. Jones]

ronelson[/url]":10ltceh8]
For grins:

old-ass-router>sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-IK2S-M), Version 12.XXX, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Tue 03-Sep-02 22:58 by kellythw
Image text-base: 0x80008088, data-base: 0x80E23440

ROM: System Bootstrap, Version 12.XXX, RELEASE SOFTWARE (fc1)

old-ass-router uptime is 7 years, 6 weeks, 2 days, 21 hours, 2 minutes

I'm mildly surprised it's even in the v12 train. I'm hugely surprised we haven't had a power hit in all that time. Kudos to you, kellythw!

I'm quite surprised it's on 12, also. Doesn't anyone follows the rule of 20 anymore?

 

[PaveHawk-]

Rick25[/url]":3iphvpuw]That I really dislike the day after patch day where anything that's wrong with the app is suddenly blamed on the OS....

Any time this happens, I start saying and emailing post hoc ergo propter hoc.

 

[hernias]

This Notepad++ bug wasted three or four hours of my time today. I was not pleased.

 

[ronelson]

I'm quite surprised it's on 12, also. Doesn't anyone follows the rule of 20 anymore?

Which rule of 20 would that be? Can't be an 80/20 rule, because with a 7 year uptime it belongs in the 20-80 portion.

 

[ncrand]

TIL that one of my predecessors decided it would a good idea to zip tie a bunch of Cat5 to the main fibre connection coming from our Service Provider.

Also: Zip Ties should be banned from Server Rooms/Data Centres.

 

[afidel]

ncrand[/url]":23fqraz1]TIL that one of my predecessors decided it would a good idea to zip tie a bunch of Cat5 to the main fibre connection coming from our Service Provider.

Also: Zip Ties should be banned from Server Rooms/Data Centres.

Agreed, you should have seen the look I shot our HP storage guys when they mentioned zip ties. I grabbed them one of our spools of thin high strength velcro.

 

[ronelson]

Agreed, you should have seen the look I shot our HP storage guys when they mentioned zip ties. I grabbed them one of our spools of thin high strength velcro.

Just ask them to make fists and stick their hands out. Zip ties have one really good purpose, and they'll never make that mistake again

 

[SandyTech]

TIL, there are still software developers out there who think its the 90s.

One of our customers is testing a new software to get them off 3 binders for their pricing information in their service department. In the drivers folder for the hardware key, there were drivers for Windows NT PPC and Alpha and windows 9x. Getting the software to talk to the other workstations in the service department? Forget it. If you need to make any change, you have to make it on one workstation, export the config and import it on every other workstation, and pray that nobody has been making changes on another workstation.

With a couple of hours work, I could probably knock out something better in bloody Access.

 

[gblansandrock]

SandyTech[/url]":yg916rx3]TIL, there are still software developers out there who think its the 90s.

Spent far too long yesterday helping a coworker configure an application that is licensed based on MAC address. The software automatically detects the server's active network adapters, and you have to select the correct one whose MAC matches what is specified in the license key. Unfortunately the software cannot detect teamed NIC's, so we had to break the NIC team and go down to a single adapter to get the software to function...

 

[Danger Mouse]

afidel[/url]":2xygf0ks]

ncrand[/url]":2xygf0ks]TIL that one of my predecessors decided it would a good idea to zip tie a bunch of Cat5 to the main fibre connection coming from our Service Provider.

Also: Zip Ties should be banned from Server Rooms/Data Centres.

Agreed, you should have seen the look I shot our HP storage guys when they mentioned zip ties. I grabbed them one of our spools of thin high strength velcro.

There is a place for zip ties in the data center, but bundling either copper or fiber isn't one of them

I would love to have velcro. I finally got my hands on some in the last year or so for the data center. Until then, we just had to be creative

And then it turned out there was a huge roll there all along, but just never actually given to us

----

TIL....that it's dawned on me, that the self-help org/cult that I was required to join (said in not so many words by the recruiter) if I wanted a job at [large national restaurant chain data center], has probably reached into [super mega enterprise platform conglomerate] because the excerpt I saw on ABC's news report in Youtube for [large national restaurant chain's weekend corp "retreats"], exactly matched the format of the vendor presentation that I saw at a vendor meet/greet day.

IOW, inappropriate personal story of triumph over tragedy without seeming relation to the presentation, purely as a way of building a false sense of familiarity with the speaker.

The corp "retreat"? Looked like a modern evangelical church service and was even formatted similarly. Friendly greetings, followed by lots of dancing and singing, then the "presentation" and lots of warm hugs at the end.

That's not to say that any individual person who attends those "courses" is bad in any way, but it's something to be wary of, even in the IT world it seems. And from what I can tell, said self-help org/cult appears to be the explicitly for profit branch of [large modern religious organization that has bought suicide hotlines].

 

[afidel]

gblansandrock[/url]":50wcgyir]

SandyTech[/url]":50wcgyir]TIL, there are still software developers out there who think its the 90s.

Spent far too long yesterday helping a coworker configure an application that is licensed based on MAC address. The software automatically detects the server's active network adapters, and you have to select the correct one whose MAC matches what is specified in the license key. Unfortunately the software cannot detect teamed NIC's, so we had to break the NIC team and go down to a single adapter to get the software to function...

VM, let the hypervisor do the teaming and also makes sure that a hardware failure doesn't blow up your licensing.

 

[Sunner]

SandyTech[/url]":1xbz4vgu]TIL, there are still software developers out there who think its the 90s.

One of our customers is testing a new software to get them off 3 binders for their pricing information in their service department. In the drivers folder for the hardware key, there were drivers for Windows NT PPC and Alpha and windows 9x. Getting the software to talk to the other workstations in the service department? Forget it. If you need to make any change, you have to make it on one workstation, export the config and import it on every other workstation, and pray that nobody has been making changes on another workstation.

With a couple of hours work, I could probably knock out something better in bloody Access.

We have a bunch of developers who have to be dragged kicking and screaming away from things such as Borland Delphi and Paradox. And yeah this is Delphi from the same time as Borland Paradox(which Wikipedia tells me ceased to be Borland Paradox in 1997).

Edit: also, Danger Mouse:
-half baked SAP system implementation
Got it.
-40 Year old codebase in our OpenVMS cluster (HP Superdome servers)
We had a single VMS box that no one quite knew what it did only it's "probably important". I *think* it's gone but I'm not sure. At least we got rid of our Stratus Continuum.
-Windows XP on the desktop (we're finally starting migration! wheeeee)
Some people have that, my department skunkworked Win7 into our workstations. We also have some Windows 98 left.
-.edu shenanigans
Does having ex-.gov shenanigans from four different countries count?
-Xerox MFD contract shoved down our throats, with no end of trouble until the last 6 months of fairly stable drivers
You got me there.

Oh and speaking of OS/workstations, ~18 months ago we were supposed to fall in line with our overlords with regards to hardware/software on our workstations. For me that would have meant moving from a Dell T3500 with a quad core Xeon and 24 gigs of RAM running Linux to some old Dell something or other with 1 GB of RAM and running WinXP. Yeah that would have made my job easy...

 

[PaveHawk-]

Easy now boys and girls. There's enough shitty workplaces in this planet to go around. You'll get your turn, dont need to rush things!

 

[Fulgan]

We have a bunch of developers who have to be dragged kicking and screaming away from things such as Borland Delphi and Paradox. And yeah this is Delphi from the same time as Borland Paradox(which Wikipedia tells me ceased to be Borland Paradox in 1997).

We're still using it. That and a home-grewed desktop "database" that was "extended" with paradox tables.

I'm making progress, though: between the new (but forever late) WCF/WPF app and my efforts to migrate the old tech to the newest version of Delphi and SQL server, I might see something rational sometimes before 2020...

(To be honest, the old tools still work pretty well and the data storage system is well suited to small deployment with no IT support. But still...)

 

[Ardax]

We have a bunch of developers who have to be dragged kicking and screaming away from things such as Borland Delphi and Paradox.

There's nothing wrong developers using Delphi, per se. At least, no more or less than any other dev environment. However, if they sucked at development in 1997 and still code the same way, then that's a whole other ball of wax.

 

[markwo]

Sunner[/url]":3u8sptgs]
HP Superdome servers

Is this what Mad Max runs? Two programs enter, one program leaves!

 

[gblansandrock]

afidel[/url]":2dvntmqt]

gblansandrock[/url]":2dvntmqt]

SandyTech[/url]":2dvntmqt]TIL, there are still software developers out there who think its the 90s.

Spent far too long yesterday helping a coworker configure an application that is licensed based on MAC address. The software automatically detects the server's active network adapters, and you have to select the correct one whose MAC matches what is specified in the license key. Unfortunately the software cannot detect teamed NIC's, so we had to break the NIC team and go down to a single adapter to get the software to function...

VM, let the hypervisor do the teaming and also makes sure that a hardware failure doesn't blow up your licensing.

Would have loved to, but this one had to be a physical server for logistical reasons.

 

[Sunner]

Ardax[/url]":1nulmn4v]

We have a bunch of developers who have to be dragged kicking and screaming away from things such as Borland Delphi and Paradox.

There's nothing wrong developers using Delphi, per se. At least, no more or less than any other dev environment. However, if they sucked at development in 1997 and still code the same way, then that's a whole other ball of wax.

Well yeah if that would be a recent version I'd agree even though it would be a bit out of place given what business we're in. But like I said, this is the same old version so to say it's EOL would be a bit of an understatement. And yeah they suck at development too. Server applications that need to run on an open console so they can open a little window anyone?

 

[kperrier]

ranald[/url]":m4ox1f3h]

wseaton[/url]":m4ox1f3h]I learned that a homeless shelter with no more than 7 concurrent administrative desktop users is a perfectly acceptable location for a Cisco UCS and ESX SANs.

Grant money that needed to be spent, no doubt. Or someone's budget would be cut the following year.

Sounds like that shack of a library in West Virginia that got a honking-huge Cisco router for its internet connection.

 

[Ardax]

Server applications that need to run on an open console so they can open a little window anyone?

You might be better off leaving them to maintain Delphi and Paradox and get developers who can actually do things properly. If you give them access to C# and SQL Server they're going to write the same way.

 

[Demoulous]

TIL, that I'm going to be planning a migration, over the rest of this year, to clustered ontap 8.2. Also COT 8.3 looks like a freaking winner and when netapp start to get caught up with themselves

 

[ramases]

Today I learned of a very head-scratchy problem involving Cisco UCS, vSphere 5.1 and a Nexus 1000v softswitch: If you vmotion a VM from one blade in one chassis to another chassis connected to the same interconnect, that VM will experience a network outage of about 6-7 seconds; if the target chassis is connected to another interconnect, the outage will last about 10-12 seconds instead.

However, this only happens if two additional conditions are met simultaneously:
0) The interconnect has uplinks to both to a VSS consisting of two Cat6.5k and to two Nexus 5k (who in turn are connected to the Catalysts). The problem manifests only if the VLAN the VM is connected to is configured on the Nexus5k uplinks, and goes away as soon as we unconfigure the VLAN from the N5k<->FI links and add it to the Cat6.5k<->FI links.
1) The problem only manifests if the VM's port has a port-security configuration applied.

 

[M. Jones]

ronelson[/url]":15w6vc9i]

I'm quite surprised it's on 12, also. Doesn't anyone follows the rule of 20 anymore?

Which rule of 20 would that be? Can't be an 80/20 rule, because with a 7 year uptime it belongs in the 20-80 portion.

The 'rule' used to be don't put into production any IOS revision where the major, minor and revision numbers don't total 20. 11.2(8) sums 20, so meets the requirement. 12.0(5) doesn't sum 20, and so forth.

 

[jshiplett]

ramases[/url]":208fving]Today I learned of a very head-scratchy problem involving Cisco UCS, vSphere 5.1 and a Nexus 1000v softswitch: If you vmotion a VM from one blade in one chassis to another chassis connected to the same interconnect, that VM will experience a network outage of about 6-7 seconds; if the target chassis is connected to another interconnect, the outage will last about 10-12 seconds instead.

However, this only happens if two additional conditions are met simultaneously:
0) The interconnect has uplinks to both to a VSS consisting of two Cat6.5k and to two Nexus 5k (who in turn are connected to the Catalysts). The problem manifests only if the VLAN the VM is connected to is configured on the Nexus5k uplinks, and goes away as soon as we unconfigure the VLAN from the N5k<->FI links and add it to the Cat6.5k<->FI links.
1) The problem only manifests if the VM's port has a port-security configuration applied.

Do you not have your chassis connected to both FIs?

 

[Rick25]

ESXi 5.1 Update 1 has been released. https://www.vmware.com/support/vsphere5 ... 905f871958

 

[hawkbox]

Rick25[/url]":1vpzyauk]ESXi 5.1 Update 1 has been released. https://www.vmware.com/support/vsphere5 ... 905f871958

So apparently they're canning the VIclient for the web interface. Which while annoying isn't the end of the world, but how the hell do you use the update manager through the web interface? I can't figure out how to add plugins that way.

 

[sryan2k1]

11.2(8) sums 20,

Your 11+2+8 math must be different then mine.

 

[afidel]

Rick25[/url]":32hvwttg]ESXi 5.1 Update 1 has been released. https://www.vmware.com/support/vsphere5 ... 905f871958

Hmm, they're dropping OS/2 in vsphere.next, I wonder if that includes ecommstation which IS still supported.

 

[Danger Mouse]

Rick25[/url]":1zvohvyr]ESXi 5.1 Update 1 has been released. https://www.vmware.com/support/vsphere5 ... 905f871958

Awesome! I love that the "resolved issues" appears to list a lot of fixes for various upgrade from 5.0 issues.

 

[hawkbox]

Holy shit setting up certificates for vcenter is a pain in the ass.

 

[sryan2k1]

We just use the self signed ones.

 

[hawkbox]

sryan2k1[/url]":1y7gaumj]We just use the self signed ones.

I have been but because of the web interface being their new path my boss wants us to sign at least the web portion of it. At which point I might as well do all of them. But holy christ, this is enough to make me look seriously at Hyper-V.

 

[ramases]

euri[/url]":u92xhm6y]Do you not have your chassis connected to both FIs?

Yes; I wrote poorly in my OP by simplifying too much. When I wrote "connect to one interconnect" I meant "connect to one pair of interconnects": We have a total of four FI, two at our main DC and two at our hot "backup"[0] DC, and each chassis is connected to both FIs present at the site.

[0] In quotes because while we still have N:N redundancy for important services we're usually running half our regular workload at each site and regularly fail over completely to one site; makes maintenance windows much easier to get and much less hassle.

 

[ferzerp]

Rick25[/url]":jm234za8]ESXi 5.1 Update 1 has been released. https://www.vmware.com/support/vsphere5 ... 905f871958

Wonderful about the auto-consolidation of redo logs on a failed snapshot. That has been a pain with VADP backups.

 

[DuRaNdAl^]

Doing my first major launch of a new core app tomorrow. Starting at 6 PM, and finishing..... at one point in the night.

I feel much crying and gnashing of teeth tomorrow evening.

 

[daldrich]

That instead of going to TN Monday I am now driving out there with a co-worker tomorrow for a week. Should be fun.

 

[Scotttheking]

TIL that I'm so desperate for help that I'm willing to overlook the fact that this person listed on their resume a project and when I asked about it admitted that they had only proposed a design for it, not implemented it. I feel sick and dirty. This guy has like 10 years of experience with *nix, lists "Lead" and "Senior" in some titles, and I'd consider him a junior person based on the interview.

That said, my company thinks that a 30 minute phone interview for a guy is enough to decide whether to hire an admin or not, and I'm not going to effect any change there.

 

[Rick25]

That DNS is at the root of all thing not working in MS land....even when it shouldn't have been. A happy DNS is a happy admin.

I can't believe VMware won't write a nice GUI for the SSL certs. Point it to an internal PKI, get cert and go...but no has to be in PEM format etc etc.

 

[afidel]

Rick25[/url]":2jjs8hsh]That DNS is at the root of all thing not working in MS land....even when it shouldn't have been. A happy DNS is a happy admin.

I can't believe VMware won't write a nice GUI for the SSL certs. Point it to an internal PKI, get cert and go...but no has to be in PEM format etc etc.

AMEN on both accounts, currently working on setting up DNS scavenging to fix a bunch of fubar stuff with SCCM. The reason it was never enabled is that we imported our zones from the linux BIND server that the previous admin had used and we weren't sure what scavenge would do with the old records. It's now been long enough that with the exception of a few static records it shouldn't matter. I've of course backed everything up and also exported the zone to text in case I need to re-import =)

 

[hawkbox]

God damn it!

2. Update the Single Sign-On SSL Certificate
1. Update the Single Sign-On SSL Certificate
2. Rollback to the previous Single Sign-On SSL Certificate
3. Return to the main menu to update other services
The chosen service is: 1
Enter location to the new Single Sign-On SSL chain (default value is: ccerts):
cinstall_files\certs
Enter location to the new Single Sign-On private key (default value is: ccerts
\sso):cinstall_files\certs\sso
Enter the Single Sign-On master password (will not be echoed):
Do you have a load balancer installed? (yes/no) (default value is: no):
[Fri 04/26/2013 - 15:47:31.26]: Services what will be restarted as part of this
operation are: vCenter Single Sign-On (if it is stopped it won't be started).
[.] ERROR: One or more required parameters are not set or have invalid values:
[.] - I/O error reading the key file cinstall_files\certs\sso: cinstall_fi
les\certs\sso (Access is denied).
[Fri 04/26/2013 - 15:47:33.36]: Last operation update the Single Sign-On SSL cer
tificate failed :
[Fri 04/26/2013 - 15:47:33.39]: The input parameters are missing or incorrect. C
heck the logs for more details.
==================================================================


Fuck it, I'll figure it out Monday, I need a drink now. I'm not even sure I'm selecting the right directories anymore.