what did you learn today? (part 2)

[afidel]

TIL (much to my shock) that Cisco cables can actually be cheaper than retail if your rep is hungry for a deal and has applied sufficient discount to the quote. Even more shocking to me is that this was on a <$50k deal not the multi-million dollar deals where you'll usually see that level of discount.

 

[afidel]

Nah, it's a perfectly cromulent rant for the server room.

 

[afidel]

TIL how to use vscsistats to copy a servers IO pattern for testing against a new array. Since the whole process doesn't appear to be documented anywhere on the net I figured I'd put it here so I can find it and in case anyone else finds it useful:

#this needs to be done from tech support mode or through SSH as root
#this gives you the list of machine world ID's
/usr/lib/vmware/bin/vscsiStats -l

#this starts a trace session
/usr/lib/vmware/bin/vscsiStats -s -t -w <world ID from above>

#now you'll have a list of trace sessions, to record you need to do the following
#you probably don't want to use /tmp since it's only 192MB on my esxi 5 install and the trace files for a moderately busy Exchange box totalled over 20MB for 30 minutes

/usr/lib/vmware/bin/logchannellogger <trace session ID> /vmfs/volume/<volume label>/disk.trc &

Now download IOBlazer
On a machine with a disk mounted on the test target mounted do
ioblazer -P <tracefile.trc> -d <path to target disk and filename>

*edit*
And the trace files won't play back, time to figure out why

 

[afidel]

Or just use the VMWare I/O Analyzer:

http://labs.vmware.com/flings/io-analyzer

And now I feel like I wasted an afternoon...
But thank you very much!

 

[afidel]

Based on a quick 5minute look at ioblazer it looks like it is looking for the csv formatted output from vscsiStats not the raw binary data from logchannellogger.

so run: "vscsiStats -e <filename>" on the binary data (output goes to stdout).

This was it, thanks! And the I/O analyzer is very cool, though for playback I think I like ioblazer better, the stupid analyzer only allows you to attach 1 vdisk per target for playback...

 

[afidel]

Also, the VM itself (2cpu/4g) is something like 800/month.

That seems insanely expensive.

Google compute engine/AWS instances of that size are around $170-190 per month, I thought that was insanely expensive compared to what I can provide an instance for internally but apparently they're competing against a whole different set of economics.

 

[afidel]

TIL just how much I need internet. Yesterday, Windows informed me that it would take 2.5 days to download a 254M file. What the...? FIOS is now giving me 200kbps down, and a whopping 30Mbps up. And cuts out intermittently, which means that file kept failing at <10 meg anyway. Apparently this is some issue that may affect more than 1000 customers so they won't do anything about it until 1AM. Fuck you, Verizon! So now I can't work, can't play online games, and can barely even get to Ars. And I have to keep calling them to escalate (they've told me three times now that it will be fixed in a 1AM maint window, so I'm not convinced). What a boring, depressing day.

On the other hand, all of the people *at* Verizon have been pleasant to work with. It's some fuckers in the CO who don't talk to the outside world who I despise right now.

What, no tethering? That's what I do when I have to connect to work and the cable's down (our maintenance window has coincided with my cable companies more than once).

 

[afidel]

WingMan[/url]":3uavkddp]TIL how to figure out how many "micro" RAID arrays that my HP 3PAR V400 manages.

For 45 Virtual Volumes there are 3154 mirco arrays.

For those wondering, you go in and look at the LD's (showld -d) in the CLI and do the following for each row:

RSizeMD/1000/SetSz = Micro array for that LD. The 1000 is the chunklet size for the v400, older models may still be 256MB.

It might even be more than that, at least with AO active a given vvol can have a single region moved into a chunklet that's allocated to that CPG so you can have 4x as many raidlets as that calculation shows in the worst case. Actually I suspect it's always working at the region level internally and that chunklets are just to avoid a high rate of extension growth overhead but I don't have any proof of that since I haven't met an engineer with that level of knowledge.

 

[afidel]

WingMan[/url]":b5nrph3i]

akro[/url]":b5nrph3i]

WingMan[/url]":b5nrph3i]

afidel[/url]":b5nrph3i]

WingMan[/url]":b5nrph3i]TIL how to figure out how many "micro" RAID arrays that my HP 3PAR V400 manages.

For 45 Virtual Volumes there are 3154 mirco arrays.

For those wondering, you go in and look at the LD's (showld -d) in the CLI and do the following for each row:

RSizeMD/1000/SetSz = Micro array for that LD. The 1000 is the chunklet size for the v400, older models may still be 256MB.

It might even be more than that, at least with AO active a given vvol can have a single region moved into a chunklet that's allocated to that CPG so you can have 4x as many raidlets as that calculation shows in the worst case. Actually I suspect it's always working at the region level internally and that chunklets are just to avoid a high rate of extension growth overhead but I don't have any proof of that since I haven't met an engineer with that level of knowledge.

Don't use AO.

Why not? It's work well from what I have seen with customers.

I'll see if I can dig up the internals briefings I have and give you a little more deeper view into the way it's all laying out on disks...

We only have 1 tier of disks, 15k 300GB drives. The other reason, and one I harp management about every opportunity I can, is that we didn't license AO, DO, Thin, or pretty much anything else save Remote Copy and System Reporter.

Wow, what a complete waste of a 3Par, the only reason I can think to do that is that you really, really wanted a 4 controller node to avoid cache issues in a degraded state and didn't like any of the other solutions out there. Either that of don't understand the 3Par advantages.

 

[afidel]

akro[/url]":1imtw3wj]
Nevermind I thought you were saying "Don't use AO" not "We don't use AO:.

I have actually configured a lot of single tiers for customers. If nothing else the wide striping is very nice. Even better that the 7K's come with Thin Suite included. In the past I have heard lots of legacy 3PAR customers do single tier because the wide striping is good enough that they didn't need AO\DO so it's not that uncommon.

Hmm, I guess, though you're probably leaving significant Capex and Opex savings on the table by not allowing stale data to fall down to the much lower $/GB and Watt/GB NL drives. Obviously that's dependant on if you HAVE stale data, but since 80-99% of most data on commercial arrays is stale I think that's a pretty niche use case.

 

[afidel]

If you have core cals and at least 100 desktops there's no reason not to go SCEP. Less than that and the effort of setting up SCCM might be hard to justify.

 

[afidel]

TIL that I can get latency to creep up just a little on the new 3Par 7400 if I hit it with 600+MB/s of svotion traffic while backups are going on =) All in all I'd say I'm very impressed by it, now that I've moved the first production VM's over it will be fun to watch AO moving hot blocks up to SSD and see how effective that is at taking real workload IOPS off the 10k disks.

 

[afidel]

Darkseid[/url]":2bzt1ply]10krpm 9gbg LVD scsi drives still hurt like hell when chucklefucks, doing drive swaps, oops butterfingers them rather than handing them down to you.

Once I can forgive, three is 'drop one more and im going to beat you to death with the this backplane'

Do I even want to know why you're wasting power on 9GB drives?

 

[afidel]

macker0407[/url]":3563j25o]That MS's SQL licensing for VMs is still somewhat irritatingly complex and seems to imply that you have play stupid affinity games with VMs and CPUs if you use the Core model with Standard to remain completely within compliance.

My only hope now is that I can get away with using the CAL model instead, without having to engage in jiggery pokery.

Huh? For 2012 you just pay per vCPU as long as you have SA.

 

[afidel]

ncrand[/url]":23fqraz1]TIL that one of my predecessors decided it would a good idea to zip tie a bunch of Cat5 to the main fibre connection coming from our Service Provider.

Also: Zip Ties should be banned from Server Rooms/Data Centres.

Agreed, you should have seen the look I shot our HP storage guys when they mentioned zip ties. I grabbed them one of our spools of thin high strength velcro.

 

[afidel]

gblansandrock[/url]":50wcgyir]

SandyTech[/url]":50wcgyir]TIL, there are still software developers out there who think its the 90s.

Spent far too long yesterday helping a coworker configure an application that is licensed based on MAC address. The software automatically detects the server's active network adapters, and you have to select the correct one whose MAC matches what is specified in the license key. Unfortunately the software cannot detect teamed NIC's, so we had to break the NIC team and go down to a single adapter to get the software to function...

VM, let the hypervisor do the teaming and also makes sure that a hardware failure doesn't blow up your licensing.

 

[afidel]

Rick25[/url]":32hvwttg]ESXi 5.1 Update 1 has been released. https://www.vmware.com/support/vsphere5 ... 905f871958

Hmm, they're dropping OS/2 in vsphere.next, I wonder if that includes ecommstation which IS still supported.

 

[afidel]

Rick25[/url]":2jjs8hsh]That DNS is at the root of all thing not working in MS land....even when it shouldn't have been. A happy DNS is a happy admin.

I can't believe VMware won't write a nice GUI for the SSL certs. Point it to an internal PKI, get cert and go...but no has to be in PEM format etc etc.

AMEN on both accounts, currently working on setting up DNS scavenging to fix a bunch of fubar stuff with SCCM. The reason it was never enabled is that we imported our zones from the linux BIND server that the previous admin had used and we weren't sure what scavenge would do with the old records. It's now been long enough that with the exception of a few static records it shouldn't matter. I've of course backed everything up and also exported the zone to text in case I need to re-import =)

 

[afidel]

sporkme[/url]":73vfyay9]Someone sent me this, then hours disappeared:

http://devopsreactions.tumblr.com/

Thanks for that rebit hole ahole =)
Now I'm 20 minutes late leaving on a Friday.

 

[afidel]

Barmaglot[/url]":h6tov027]Second thing I do when setting up an ESXi host is set UserVars.SuppressShellWarning = 1.

Our host template sets this, enables the shell, and adds the firewall rule among other things. You can use host templates even without enterprise+, you just have to apply it and then remove it before applying a license since it will use the out of the box demo license =)

 

[afidel]

hawkbox[/url]":1e9iu13x]

Scotttheking[/url]":1e9iu13x]

sryan2k1[/url]":1e9iu13x]My last place was a small engineering company. We'd spend two hours to automate a task that would have taken 45 minutes by hand, and would have been a pain in the ass. Guess how long it takes to do it the 2nd, 3rd, 4th time....

Worse, I'm working on automating the entire OS install process with kickstart and bash (for lack of better tools). When done it'll save something like 2-4 hours per server build, probably, for each person, plus reduce errors.

I got my vmware templates up for exactly this purpose. Plug in an IP address to the config window and walk away, then when it's done the vmdk copy and setup I turn it on and it autojoins the domain, sets GPOs and waits for me to finalize any recent updates or special requirements.

Yep, vmware template + sccm to bring it to current baseline patches means we spend basically no time setting up anything OS relating allowing us to focus on things like correct security design. I went from spending 70+% of my time just racking and configuring boxes during my first year at my current gig to managing 3x as many machines with hardly any effort.

 

[afidel]

euri[/url]":x35hlzei]Anybody taken the VMware vSphere: Optimize and Scale class? Was it worth it?

http://mylearn.vmware.com/mgrreg/course ... ject=39046

Just looking at the course outline it looks like info you can glean fairly easily from places like yellowbricks and scott lowe's blog. The course I took, and think might be more interesting generally was the design workshop, basically get a bunch of smart, experienced folks in a room for a week, give them some information and let them build a design in a small group and then have them present and defend the design in front of the rest of the group.

 

[afidel]

ferzerp[/url]":i6ai7bdh]

Rick25[/url]":i6ai7bdh]That following a vendors recommendation to set IOPs to 0 with a VMware RobinRobin config can lead to 1/2 the performance of when it was set to 1000...so much for a performance boost.

Depends on the workload, the number of paths, very heavily on the array type, etc.

The default settings are abysmal for anything other than light IO, or sequential IO. I know in one of my installs, the throughput of small, random IO in a high load scenario going to the same volume is about 7x with having each request go down a separate path. If you have, say, one VM doing small, random IO with a queue depth of only 1, yes, it will be a detriment, but in every other scenario, it's at worst a minor loss, or at most a major gain. Again, based on array, and health of your storage network, YMMV.

Every array has different queue designs, 3Par says to set it to 100 for instance because their testing has shown that's the best balance of performance and resource utilization on the controllers. Now, if I was seeing 1/2 the performance I'd be a bit weirded out because that's a huge decrease. I know that prior to 4.1U1 there was a bug that would reset any custom value to like 1 million after a reboot.

 

[afidel]

sryan2k1[/url]":3hsl4zoh]I work for a lawfirm, usually at any given point I could trip and hit my head on 4 printers on the way down. If one of those is offline/broken people freak the fuck out. Walking (in some cases) literally < 5 feet to another printer isn't acceptable.

I hate that crap. It makes me miss my time at Cisco, two to four HP 8100's per floor and one color laserjet per floor (we also had a dye-sub and color wax tektronics for the marketing guys to do pre-press work) and that was it. At my current place we have oodles of very high speed MFP's and if it's more than ~30' to a copy room there's usually a medium speed black and white printer but still we have dozens of desktop printers. The cost per page over the life of those small printers has to be at least an order of magnitude higher than either of the other solutions, I can't understand why management lets the idiocy persist.

 

[afidel]

M. Jones[/url]":361ucux5]

afidel[/url]":361ucux5]It makes me miss my time at Cisco, two to four HP 8100's per floor and one color laserjet per floor (we also had a dye-sub and color wax tektronics for the marketing guys to do pre-press work) and that was it.

Cisco famously rationalized their whole printing architecture from a clean-sheet. Everyone should; I've had great success with standardizing on IPP, even using downloadable Microsoft drivers for support back to Windows 98. OS X, *BSD and Linux support IPP natively with CUPS, of course.

CEPS was the bomb, we lost our local print server once and the only thing that happened was that jobs took a bit longer because they had to spool to the server at the larger office we were associated with (DS-3 connection). That server took over the personality of our server including queues and everything and the local traffic manager did the IP redirection. It was bar none the slickest print system I've ever seen. Of course it was born out of necessity, there were two global print admins for a company of 45k employees! Oh, and printing was mission critical because if labels didn't print with MAC addresses and serial numbers stuff couldn't ship.

 

[afidel]

Technarch[/url]":2f02n2er]

KD5MDK[/url]":2f02n2er]Hire a flunky to deliver printouts from the copy room.

Better yet, institute a companywide health improvement campaign that encourages employees to get up and walk.

Then, remove all printers from the building and set up a corporate account with the nearest Kinko's.

I love it!

 

[afidel]

ferzerp[/url]":1fhiilcq]
You are lucky. I'm rather miffed that they killed the ISA server line and force you in to a product that is a sever/cal model for *any* use as a replacement.

Ha, we've got UAG for 'free' and I'm still looking at Kemp with the edge security pack. We used an MS platinum partner to implement UAG and it's still never been right and it seems like every time I touch it it breaks, and since we have DA configured it breaks all connectivity for whatever machines have a working DA config (about 30% break at random, and yes we've had PSS look into everything and they can't figure out why it's breaking either, everything appears to be correct). Since we switched to Cisco for VPN and we're only publishing Activesync through UAG I can't wait to ditch the POS.

 

[afidel]

TIL that there's a company that produces a standalone management server for FEP, this is great for the SMB where getting SCCM up and running is a bit daunting.

 

[afidel]

markwo[/url]":1qfht0v1]

w00key[/url]":1qfht0v1]The front page promoted comments are amusing

For "Facebook aims to knock Cisco down a peg with open network hardware", this one takes the cake

There's a ton of potential here. Cisco actually charges $10,000 for a switch, comparable to a $173.26 "TP-Link" (TL-SG2424) switch you can buy from Wal-Mart... Of course the latter has crap firmware (software) with a crippled and foreign user interface.

Yeah, lets reflash a TP-Link with SDN and use it in racks. You go first

Cisco sells a 10K 24 port L2 switch? Maybe some libraries in WV will buy them.

They sell a lot more expensive L2 switches than that, look at the whole Nexus line =)

 

[afidel]

ferzerp[/url]":hj7mi21r]vCenter 5.1 is pretty much a "People want us to not rely on windows, so, vomit out a rewrite as quickly as possible and shove it out the door, regardless" scenario.

I'm not just talking the appliance either. It's the whole thing.

Agreed 110%, they took a requirement from a few hosting providers and small/midsized businesses and turned it into something that pisses off the other 99% of their customer base. I'm hoping they fix most of the stupid parts in time for vsphere 6 or 6U1, in the meantime I'm upgrading from 5.0U1 to 5.0U2 so I don't have to deal with the idiocy. There's nothing "must have" in 5.1 for my environment so it's easy for me to wait it out.

 

[afidel]

ferzerp[/url]":2unm07rn]

Rick25[/url]":2unm07rn]That despite Western Digital branding their Red drives as the preferred choice for a NAS, the Acronis software that they provide won't install unless you've got a Red (or maybe any WD) drive physically installed on the system you're adding the software to....kinda defeats the purpose of a "NAS" branding for a drive. Either remove the offered software or allow it to work.....

Well, that one is interesting...

I use reds in my home server for media and backup storage.

Yep, I use a red drive in my HTPC because it's silent and doesn't do the micro-pause thing that introduced glitches in recordings with my previous drive when recording two HD streams at the same time. I'm using Crashplan for backups and have never been an Acronis fan so I didn't even take the media out of the package.

 

[afidel]

dredphul[/url]":hft1iuob]that the Cisco UCS C240 M3S is advertised as a 2U product, but actually requires 3U of space as all the air vents are on top of the unit.

Found buried in the install guide:

Do not block the air vents on the top of the server's cover. Do not stack another server directly on top of the C240 server. Doing so blocks the proper airflow, which could result in overheating, higher fan speeds, and higher power consumption.

Glad this was discovered before the units were racked.

Uh, yeah that would be an RMA finding for me. What retard thought that was an ok design decision?

 

[afidel]

ramases[/url]":6zujtlsb]TIL today? Two things:
0) Certain conditions cause a 5.x XenServer to crash after a downed iSCSI path recovered. "Quick, make your bet. Will recovering the path crash the XenServer, yes or no?"
1) Good thing that this environment is soon to be decommissioned, because there are other mines just waiting to explode in that setup. Thin-provisioning at the SAN-level with an overcommit-ratio of 5:1, what could possibly go wrong?

The overcommit wouldn't bother me at all if your growth pattern was fairly predictable. I run up to 4.5:1 at the VMWare level. At the SAN level I'd want warnings at 10-15% depending on how long it took to procure and install hardware, but a 5:1 overcommit wouldn't bother my by itself.

 

[afidel]

M. Jones[/url]":3pfluave]

gblansandrock[/url]":3pfluave]

ferzerp[/url]":3pfluave]
I'm really not sure how cisco has convinced anyone to buy their servers. Actually, after my organization was continually harassed, at all levels, I do know. They keep hammering away until you relent, or find another contact within your org to try to sell to (all the way up to execs). Or at least they try to. We never gave in because we don't buy based on how hard you try to market to us.

Keep it up and they'll flat out give you UCS. That's what happened to us. After turning them down, they eventually gave us a couple chassis, dozen blades and dual interconnects, free of charge.

We don't use UCS, except to run Cisco CallManager/CUCM VMs, although we do control those from the normal vCenter. Now I wonder if we got those gratis -- probably not.

Our people who have gone to UCS training report back that the features are only going to work for you at scale. There's even more lock-in with UCS than with other blade servers.

That's been my assessment, if you have datacenters full of them it's a very sweet system, if you need 2-3 enclosures it's WAY too complicated to ever pay off the time investment. I keep telling the Cisco guys that I spend less than 5% of one FTE on managing hardware so it's impossible that they'll save me enough time, they still keep pitching UCS. The only part I really want from UCS (UCS P81E, aka Palo aka VIC) they don't sell without the servers =(

 

[afidel]

So speaking of VIC/Palo, it looks like the Qlogic 8200 cards with nPAR can do basically the same thing, though they only have 4 buckets. Is anyone using these to assure vmotion and storage bandwidth?

 

[afidel]

M. Jones[/url]":j0caa6sm]

Xon[/url]":j0caa6sm]
At work, the restore process in-case of a event which requires a significant restore is to physically move the backup datastore to be physically next to the VM farm's SANs because getting 1gbit between the two datacentres is prohibitively expensive. Never mind multipule 10gbit uplinks.

This is a reminder that it's important to have less data.

Yes, seriously. Data management is atrocious in most enterprises, and that needs to stop. Store what needs to be stored, store it once, have retention policies, understand clearly what you're keeping for opportunistic reasons (e.g. caches, local copies of ISOs and software installers and Linux repos and WSUS), and understand what's eligible to purge even if you don't purge it now. Data in unstructured systems is the issue.

We have users requesting USB drives because their PSTs are filling their local laptop drives. What should happen is that the data should be properly managed and PST files shouldn't be used at all -- everything within retention policy should be in the live mailbox, and there should be some additional quota within the mailbox for content outside the retention policy, but that content might not have the same availability guarantee.

TL; DR: Useful data needs to be in strongly structured systems, not hundreds of versionless, uncontrolled spreadsheets scattered amongst obsolete package media and multimedia of dubious propriety.

I strongly disagree, it's MUCH more cost effective for my company to pay my small team to purchase and manage storage systems including backup than it is for them to hire enough headcount to wade through the mountains of data and correctly classify whether it needs to be kept or not. A few FTE's and a couple hundred thousand in storage is so much cheaper than doubling headcount (the minimum it would take to manage the data) that it's a complete no-brainer. IT exists to save labor costs, by forcing your users to manage data you're not performing your function IMHO.

 

[afidel]

M. Jones[/url]":2bg9v09g]

daishi5[/url]":2bg9v09g]
There is only one catch, I was reviewing our SQL licensing a little while ago, and I noticed we don't have nearly as many licenses in the Microsoft Volumne licensing site as I know we have purchased over the last 10 years. I suspect that we purchased them through this wonderful vendor, and they were never properly transfered. And, before this last system was put in a year ago, our purchases were all handled on paper. So, I have 10 years of paper copies of orders that I am going to get to sift through, with this wonderful vendor who can never find anything, and try to figure out where our licenses are.

I don't do licenses, but the obvious question presents itself: shouldn't Microsoft be able to discern the owner of record for any given license key? If you chose, I don't see why you couldn't give Microsoft a list of license keys in use and ask if any of them are still registered with the specific vendor. If so, the vendor could be asked to confirm that those licenses were transferred to your firm. Hopefully the vendor hasn't sold the licenses more than once.

SQL Server doesn't use license codes.

 

[afidel]

Darkseid[/url]":vttptn67]That contention on fibre builds is worse than DSL contention, and managers dont understand that if you have 100 customers at 40mbit you need to have 40x100 bandwidth or close to it available. Buying amounts 40% below what youve resold, just leads to some uers getting a shitty connection, and those customers will come a-screaming.

This is why the fibre roll out in Ireland is going to be 'interesting', theyre offering people to sign up on DSL now and switch to fibre when it hits their exchange. Except, the exchanges are already overloaded and under providing dsl on creaky crappy 20-30 year old copper.


by interesting I mean 'colossal goatfuck'

LOL, you obviously know nothing about ISP economics, 1.4:1 is nothing for oversubscription of last mile to backhaul, in fact no ISP could survive on such low oversubscription rates (hint, most enterprise LAN designs are more oversubscribed than that). More typical rates are between 25:1 and 45:1. Canada put the cap at 50:1 for their definition of broadband in their rural broadband initiative.

 

[afidel]

PaveHawk-[/url]":1b87eo3r]Dell can be a bit silly at times. A new sales rep tried to push their big ass FC switch (the PowerConnect B-DCX4S) for a solution I'm putting together. Its only 4 servers with dual FC HBAs.

Thankfully my usual rep came back to me and tore the kid a new one.

Seriously, when would I need 192 8Gb FC ports in a 4 server solution?

for the lulz apparently. Though a director is so big nobody is going to miss it in the pricing, should have gone with a pair of 300's/5100's (btw I love my 5100's, other than an early bug with the web api service eating all ram causing a system reboot they've been rock solid).

 

[afidel]

PaveHawk-[/url]":2hklckcj]Trying to put together an end to end HP solution is enough to drive me mad.

Trying to do a 5 host vSphere environment with a dedicated DC + Backup server (including a backup appliance and dual LTO6 tape library) with networking and SAN, all with HP products. I can do it in my head, but trying to find the right part numbers and put together a BOM for bid pricing is making me curse HP and its fucked up shitty systems.

At least with Dell, I can configure everything I wanted via the partner portal and have the BOM setup nicely.

Still, I want to go the HP option for this since I think the 3PAR is a better offering than an EQL (client requested EQL pricing over Compellant). Hopefully I can get my disti to do the install, since I have 2/10ths of fuckall knowledge about putting together and deploying a 3PAR.

3Par is still HP installation services required at this point AFAIK. The goal is to make it customer installable but at this point it's pretty far from that experience.