what did you learn today? (part 2)

[sporkme]

Someone sent me this, then hours disappeared:

http://devopsreactions.tumblr.com/

 

[afidel]

sporkme[/url]":73vfyay9]Someone sent me this, then hours disappeared:

http://devopsreactions.tumblr.com/

Thanks for that rebit hole ahole =)
Now I'm 20 minutes late leaving on a Friday.

 

[Danger Mouse]

afidel[/url]":2h01ka9n]

Rick25[/url]":2h01ka9n]That DNS is at the root of all thing not working in MS land....even when it shouldn't have been. A happy DNS is a happy admin.

I can't believe VMware won't write a nice GUI for the SSL certs. Point it to an internal PKI, get cert and go...but no has to be in PEM format etc etc.

AMEN on both accounts, currently working on setting up DNS scavenging to fix a bunch of fubar stuff with SCCM. The reason it was never enabled is that we imported our zones from the linux BIND server that the previous admin had used and we weren't sure what scavenge would do with the old records. It's now been long enough that with the exception of a few static records it shouldn't matter. I've of course backed everything up and also exported the zone to text in case I need to re-import =)

Rebuild of our Infoblox config after upgrade to current code fixed ALL THE THINGS.

And if you think VMWare is bad, I'll have to say that Cisco is worse. Some of their server appliances are KNOWN to crash as a "normal" behavior, if the cert doesn't match or expires or, etc.

So much for HA/FT for an appliance pair designed just for that.

Bonus points that even with throttling and bpduguard in place, we were still taking it in the gut at the core router.

After finally getting my boss to agree to redo the connections sensibly a LOT of problems disappeared.

Also, VMWare did write up some kind of SSL Cert maintenance app.

http://blogs.vmware.com/kb/2013/04/intr ... XsNT7SitaQ

From the look of things, a nice text gui.

I also found a few kb articles about using the microsoft cert authority for certs.

One thing to note is that I've used Digicert's utility to check that a cert was correct. It actually helped me find the problem in our old wildcard cert. I used it to confirm the functionality of our current certs (from Comodo).

 

[Danger Mouse]

...that although it's a different version of the same product, clearing out the event history ARCHIVE (not the history itself) will reduce the SQL database size by about 40% on an altiris install. The kb articles refer to Notification Server and not Deployment Server, but it's almost the same thing. It's similar enough in WTF they're doing in the sql db.

Trunc that table and rebuild the index and then shrink the db as needed for best results.

Afterwards, you may need to restart the altiris server (axengine process/server name) service and all should be well. After a minute or two (or more) the various installed agents will report back to the server and start to populate the status information again.

I figure the DB still has some corrupt portions, but that took care of most of it.

 

[Rick25]

sporkme[/url]":1k0bo7sk]Someone sent me this, then hours disappeared:

http://devopsreactions.tumblr.com/

Awesome website


DM - thanks for the linkage for the cert utility from VMware, definately going to dig into it on Monday!

 

[M. Jones]

SandyTech[/url]":29qqhj7g]

With a couple of hours work, I could probably knock out something better in bloody Access.

The more specialised and niche the ISV's software, the more certainty of that software's unrepentant inferiority.

 

[Danger Mouse]

M. Jones[/url]":170ucfiv]

SandyTech[/url]":170ucfiv]

With a couple of hours work, I could probably knock out something better in bloody Access.

The more specialised and niche the ISV's software, the more certainty of that software's unrepentant inferiority.

That describes like all .edu software

 

[SandyTech]

M. Jones[/url]":vvvmts77]

SandyTech[/url]":vvvmts77]

With a couple of hours work, I could probably knock out something better in bloody Access.

The more specialised and niche the ISV's software, the more certainty of that software's unrepentant inferiority.

It wasn't fully done in access, but one of our dev types knocked something together and tossed it on their sharepoint in a couple of hours.

 

[crisavec]

And today I realized...I've been at this job for 3 years. And as of 2 weeks ago I'd finally wrapped up all the projects I identified in my first 2 weeks as Must Do if we wanted to actually keep things operational. Its been three years of ripping off duct tape and baling wire and getting rid of all the shoddy "We'll only be here 7 years before we run out of ore and shut down" bandaids.

I have to admit, its rather satisfying :bigdumbgrin:

Now I just have to go finish all the Other projects that have arisen since then.

 

[KrispyK]

Third party software vendors should not setup servers:

• Server IP address of : 192.188.1.50 !??! WTF!? It's the only server on the network and you chose 50 and also typo'd the private address !
• Software backs itself up to FTP server using a dynamic DNS address and no encryption (probably the guy's home PC). Oh and you're using NetDrive to expose the FTP site as a drive letter? FFS...
• POP collection is being used to pull email data from ISP and to pass it into their perfectly good Exchange server (which is also badly configured)
• Certification Authroity not installed on the only DC

 

[PaveHawk-]

KrispyK[/url]":1red82mr]

• Certification Authroity not installed on the only DC

Wait, are you saying this is a good or a bad thing?

 

[KrispyK]

Wait, are you saying this is a good or a bad thing?

I see your point, in this instance it's an SBS server (installed on a desktop chassis) and they have no other servers to act as a CA.

 

[PaveHawk-]

KrispyK[/url]"45v42ip]it's an SBS server (installed on a desktop chassis)

Yeah, in this case they have other problems

 

[w00key]

sporkme[/url]":3gm0lv5b]Someone sent me this, then hours disappeared:

http://devopsreactions.tumblr.com/

Awesome! I'm seeing myself in like half of the posts, working as "dev + ops + support" does have some advantages


Our "nagios" setup is basically apt-get install plus adding the ips to ping and http urls to check, so every time we take down stuff for maintenance, this happens:


Carefully examining nagios emails

 

[DuRaNdAl^]

Hour number 5 of my deployment. Everything seems to be going well except for a small hitch with the sql replication of one database, seems to be ok now.

Only 3 to 5 hours left.


Plenty of chips left, and the Habs are winning. Woot!

 

[ramases]

w00key[/url]":3vq4gzwu]Our "nagios" setup is basically apt-get install plus adding the ips to ping and http urls to check,

We generate our nagios checks from our NM database, which is usually fairly complete for servers due to it being also used for DNS zone generation (no NM record = no DNS for you); even though the UI to it is quite archaic, being able to add a ssh-check (for example) by just adding an attribute to the object is pretty neat (the IP to be checked will default to the IP attribute assigned to the object)

so every time we take down stuff for maintenance, this happens:

Carefully examining nagios emails

Ahemn

 

[w00key]

I know I know, but I haven't invested any time in it to make Nagios accept external commands, so no scheduled downtime for me.

It really is an apt-get installed package with as little change as possible, and apparently it's not only me that does that

 

[xcrunner529]

sryan2k1[/url]":1hln2jqu]We just use the self signed ones.

Except the braindead idiots at VMware decided to force certificate validation for running resxtop on their VMA appliance meaning you're basically forced to change the certs if you don't want to go and manually enable the shell on each host and get all the scary warnings.

 

[Barmaglot]

Second thing I do when setting up an ESXi host is set UserVars.SuppressShellWarning = 1.

 

[Rick25]

Barmaglot[/url]":3p7ry68m]Second thing I do when setting up an ESXi host is set UserVars.SuppressShellWarning = 1.

Oh handy thanks!

Woke up to a failed SSD in an Array, in the process of getting that ticket open I noticed I've got more emails in my inbox then my BB. Check the BES and it's connecting to two new IPs for the SRP. Couple of FW changes later and all's good, SSD showed up 2hrs after case was opened!

 

[Barmaglot]

TIL that ESXi 5.1.0 does not like configuring USB controllers for PCI passthrough. Back to 5.0.0u2 for this deployment...

Also, TIL, for the third goddamn time, that PowerEdge T310 only takes x8 data width RAM. Not x4, and not x16.

 

[Danger Mouse]

Barmaglot[/url]":9eibp7fe]TIL that ESXi 5.1.0 does not like configuring USB controllers for PCI passthrough. Back to 5.0.0u2 for this deployment...

Also, TIL, for the third goddamn time, that PowerEdge T310 only takes x8 data width RAM. Not x4, and not x16.

Hawkbox posted about that during his move to 5.1. That's one reason I'm glad I have a network USB hub.

It's good to get confirmation though.

 

[afidel]

Barmaglot[/url]":h6tov027]Second thing I do when setting up an ESXi host is set UserVars.SuppressShellWarning = 1.

Our host template sets this, enables the shell, and adds the firewall rule among other things. You can use host templates even without enterprise+, you just have to apply it and then remove it before applying a license since it will use the out of the box demo license =)

 

[ronelson]

this person listed on their resume a project and when I asked about it admitted that they had only proposed a design for it, not implemented it.

Around here, that could still mean they spent 3 years on it

 

[Arbelac]

ronelson[/url]":2wf4fwq7]

this person listed on their resume a project and when I asked about it admitted that they had only proposed a design for it, not implemented it.

Around here, that could still mean they spent 3 years on it

I spent a year on designing an in house data center, only to have it shitcanned a month before go because we bought a company with a DC. I still list it, it was a massive amount of experience gained.

 

[Cookie.Monster]

That Microsoft just released a 314 page best practice guide for securing Active Directory.

 

[macker0407]

That I have a 314 page document to read.

 

[Scotttheking]

Arbelac[/url]":38nkczd0]

ronelson[/url]":38nkczd0]

this person listed on their resume a project and when I asked about it admitted that they had only proposed a design for it, not implemented it.

Around here, that could still mean they spent 3 years on it

I spent a year on designing an in house data center, only to have it shitcanned a month before go because we bought a company with a DC. I still list it, it was a massive amount of experience gained.

Agreed, listing on resume is fine, if it had been "designed and proposed", or something. This wasn't a big project, but the design and proposal happened to be sitting on my desk at the moment, so I asked him to walk me through the implementation and any issues he encountered with it. The first thing he had to say was "I never actually implemented it." There were several instances where I felt he never quite got as far as he made it seem on the resume.

 

[hawkbox]

Danger Mouse[/url]":ccz2msdp]

Barmaglot[/url]":ccz2msdp]TIL that ESXi 5.1.0 does not like configuring USB controllers for PCI passthrough. Back to 5.0.0u2 for this deployment...

Also, TIL, for the third goddamn time, that PowerEdge T310 only takes x8 data width RAM. Not x4, and not x16.

Hawkbox posted about that during his move to 5.1. That's one reason I'm glad I have a network USB hub.

It's good to get confirmation though.

Yeah that bit me hard.

 

[daldrich]

That if you send an e-mail, in English, to someone in China exchange will translate the e-mail for both parties. Which explains some of the strange e-mails we get from our groups in China.

That for how great RPK is for some things, it can be downright infuriating to troubleshoot it.

 

[sryan2k1]

That if you send an e-mail, in English, to someone in China exchange will translate the e-mail for both parties.

Not by default it won't.


We email/lync our Shanghai office just fine in English.

 

[PaveHawk-]

macker0407[/url]":1b8u7row]That I have a 314 page document to read.

Me too.

 

[Technarch]

Cookie.Monster[/url]":1j3qy2ra]That Microsoft just released a 314 page best practice guide for securing Active Directory.

The impossible, in only 314 pages!

 

[ncrand]

That the server I thought was compatible with 2008 R2 isn't (DL320 G5). So this may screw up our AD upgrade plans unless the 2008 drivers work flawlessly.

 

[Darkseid]

afidel[/url]":140qs0gi]

Darkseid[/url]":140qs0gi]10krpm 9gbg LVD scsi drives still hurt like hell when chucklefucks, doing drive swaps, oops butterfingers them rather than handing them down to you.

Once I can forgive, three is 'drop one more and im going to beat you to death with the this backplane'

Do I even want to know why you're wasting power on 9GB drives?

I`ll say three words which should cover it.

Legacy
critical
antidisestablishmentarianism

technically I only need the first two, but I like the third word



Today, Im having 'fun' trying to track down the reason why certain customers of ours arent getting the bandwidtht theyre supposed to be getting. The lines profile at 6-12mb, their dsl circuits are built on fast interleave at up to 17mb, the router shows synch at 6mb, but when they browse or run a speed test theyre getting sub 1mb speeds.

One or two, I can put down to bit-torrent / saturated soho routers, when its a dozen or so, all within a 10 mile radius, yeah that points at point of delivery issues. So I have 300ish customers showing this issue, 2 weeks ago that was 900 but we 'fixed' our DNS servers (killed 2 old proliants, put 2 brand new blades in), which has helped improve matters no end. But now, we're back where we started, I haev 177 line tests, profile checks, circuit checks and WLR3 analysis to grind my way through, then to speak to Eircom/BT Ireland.

Think of the most obstinate, jobs worth, oh we cant possibly do that, kind of person you know, scale that up into a company, then scale it again so that the company has a monopoly on all the laid copper and merely leases it out to other customers.

Trying to get them to _do_ anything is painful, trying to get them to admit theres a problem doubly so.

 

[WingMan]

Darkseid[/url]":3nrdlf45]

antidisestablishmentarianism

technically I only need the first two, but I like the third word

I had to learn to spell that word in elementary school. Damn double negative in a word...

 

[markwo]

w00key[/url]":300enk2f]I know I know, but I haven't invested any time in it to make Nagios accept external commands, so no scheduled downtime for me.

It really is an apt-get installed package with as little change as possible, and apparently it's not only me that does that

Set check_external_commands=0 to 1 in nagios.cfg, restart nagios.

I had my nagios install running for probably 9 months before I invested the 3 minutes it took to figure that out, wish I had done it sooner.

 

[SomeHandleThatStartsWithS]

PaveHawk-[/url]":1azwljn6]

macker0407[/url]":1azwljn6]That I have a 314 page document to read.

Me too.

Make that me three.

 

[Xon]

Darkseid[/url]":w31ybgyy]
One or two, I can put down to bit-torrent / saturated soho routers, when its a dozen or so, all within a 10 mile radius, yeah that points at point of delivery issues. So I have 300ish customers showing this issue, 2 weeks ago that was 900 but we 'fixed' our DNS servers (killed 2 old proliants, put 2 brand new blades in), which has helped improve matters no end. But now, we're back where we started, I haev 177 line tests, profile checks, circuit checks and WLR3 analysis to grind my way through, then to speak to Eircom/BT Ireland.

This speaks to me as of congestion from the exchange(as in the exchange is out of bandwidth) and/or insufficient backhaul to your network's core. If it is congestion of the exchange's backhaul, those clients might not even be the major contibutor and you are getting slammed by shared infrustructure bottlenecks.

 

[Darkseid]

Alfrik[/url]":1j372f86]

PaveHawk-[/url]":1j372f86]

macker0407[/url]":1j372f86]That I have a 314 page document to read.

Me too.

Make that me three.

Dont you mean you have a 300 page document that you`ll read carefully, analysse, update your thought processes ... then have its best practices, at best utterly ignored, at worst, trampled underfoot.


or am I being cynical ?