what did you learn today? (part 2)

[hawkbox]

RicDavis[/url]":32d8xyel]

hawkbox[/url]":32d8xyel]Has anyone tried or figured out how to install this update on a Dell Tx20 server running vsphere? Dell 12G PH PowerEdge Server backplane expander firmware version 1.07 cause I'm curious what it does and if it might fix my PCI-E Gen1 NIC issues but it only seems to have an installer for red-hat and windows which kind of makes me think it doesn't actually do anything.

Also is Dell just getting worse for explaining it's updates or were they always this bad?

IE:

Fixes & Enhancements
-Iniitial release

Does it show up in the built in firmware update tool - unified server administrator or whatever it is called?

? I don't recall ever seeing something called that.

Edit: Are you talking like OpenManager or whatever? Cause I'm just in the process of building a machine to run that.

 

[dlp]

hawkbox[/url]":yfjhvie2]

RicDavis[/url]":yfjhvie2]

hawkbox[/url]":yfjhvie2]Has anyone tried or figured out how to install this update on a Dell Tx20 server running vsphere? Dell 12G PH PowerEdge Server backplane expander firmware version 1.07 cause I'm curious what it does and if it might fix my PCI-E Gen1 NIC issues but it only seems to have an installer for red-hat and windows which kind of makes me think it doesn't actually do anything.

Also is Dell just getting worse for explaining it's updates or were they always this bad?

IE:

Fixes & Enhancements
-Iniitial release

Does it show up in the built in firmware update tool - unified server administrator or whatever it is called?

? I don't recall ever seeing something called that.

It was called Services in the boot menu before, now I think it's Lifecycle controller. You can boot up off of it, give it a network address, point it to ftp.dell.com and pull updates. They can lag behind the latest updates though.

 

[Brandon Kahler]

Use Dell's Repository Manager to build a boot DVD for your specific machine. Then marvel before the march of hundreds of dots while it does its thing with absolutely no indication of what update is running or how long until it finishes. Of note, the last batch of R710 boxes I did took very nearly one full hour to run all the updates.

Repository Manager boot discs are how I update all the ESXi hosts I work with. I have had horrible luck getting the Life-Cycle Updater to work.

 

[scorp508]

gradster[/url]":12tl4ifk]

sryan2k1[/url]":12tl4ifk]

Can you actually have hosted exchange accounts authenticate using separate domains?

You can have O365 auth against your AD structure. They can do a hybrid approach where some mail is on prem and others are in the cloud.

Hosted Exchange + local AD authentication is a lot more difficult than one would think (at least a year ago when I looked into a project for a company). To get "fully" redundant E365 with AD authentication requires two different sites, and about two servers per site for SMBs. Each site will have a DC and a federation server. The documents that I read through for setting up redundant federation and DCs over two sites was far from a simple deal.

Another recently added and supported option is to host ADFS in an Azure VM. It gets it off of the local network into a service that can make it redundant for you if the local resources aren't adequate.

Office 365 Adapter: Deploying Office 365 Single Sign-On using Windows Azure

 

[hawkbox]

Brandon Kahler[/url]":1ntg6e1l]Use Dell's Repository Manager to build a boot DVD for your specific machine. Then marvel before the march of hundreds of dots while it does its thing with absolutely no indication of what update is running or how long until it finishes. Of note, the last batch of R710 boxes I did took very nearly one full hour to run all the updates.

Repository Manager boot discs are how I update all the ESXi hosts I work with. I have had horrible luck getting the Life-Cycle Updater to work.

Dell has this? God damn it! I know finding IBMs BMOC is a pain in the ass but I didn't even know Dell had one.

 

[Brandon Kahler]

hawkbox[/url]":2amm5ffc]

Brandon Kahler[/url]":2amm5ffc]Use Dell's Repository Manager to build a boot DVD for your specific machine. Then marvel before the march of hundreds of dots while it does its thing with absolutely no indication of what update is running or how long until it finishes. Of note, the last batch of R710 boxes I did took very nearly one full hour to run all the updates.

Repository Manager boot discs are how I update all the ESXi hosts I work with. I have had horrible luck getting the Life-Cycle Updater to work.

Dell has this? God damn it! I know finding IBMs BMOC is a pain in the ass but I didn't even know Dell had one.

Yeah, it was one of those 'Oh Thank Goodness' moments when I learned about it too.

 

[sryan2k1]

Dell update manager is like $50 a year per host, and allows updating of all the firmware from within vCenter.

 

[hawkbox]

sryan2k1[/url]":3m7rlv6j]Dell update manager is like $50 a year per host, and allows updating of all the firmware from within vCenter.

They also have a free 1 host version that will work smashingly for me at home.

 

[afidel]

sryan2k1[/url]":27i9sq89]Dell update manager is like $50 a year per host, and allows updating of all the firmware from within vCenter.

Dell Management Plug-in for VMware vCenter looks to be $33/server/year ($100 per server for a 3 year subscription)

 

[antiwraith]

hawkbox[/url]":2z0e42l6]

Brandon Kahler[/url]":2z0e42l6]Use Dell's Repository Manager to build a boot DVD for your specific machine. Then marvel before the march of hundreds of dots while it does its thing with absolutely no indication of what update is running or how long until it finishes. Of note, the last batch of R710 boxes I did took very nearly one full hour to run all the updates.

Repository Manager boot discs are how I update all the ESXi hosts I work with. I have had horrible luck getting the Life-Cycle Updater to work.

Dell has this? God damn it! I know finding IBMs BMOC is a pain in the ass but I didn't even know Dell had one.

The boot CD is damn annoying and takes forever, but my new ESXi hosts out of the box have a firmware bug where the Lifecycle controller crashes trying to update firmware for anything when my specific 10 gig NICs with a specific NIC firmware are installed in the system. The fix? Take the NICs out and run the lifecycle controller, which doesn't update the NICs, use the boot CD which lags behind and doesn't any newer NIC firmware, or just wait until Q4 when Dell will have an updated version of the LCC that will see and work with the borked NIC firmware. Even better, of my new hosts....some are like this and some have different NIC firmware that works fine with the LCC.

Pro tip, make 1 Boot CD per model of server. Combining all of your updates for all of your models makes a larger load of crap that reports errors when you do boot to the CD.

 

[hawkbox]

antiwraith[/url]":thr1fnmc]

hawkbox[/url]":thr1fnmc]

Brandon Kahler[/url]":thr1fnmc]Use Dell's Repository Manager to build a boot DVD for your specific machine. Then marvel before the march of hundreds of dots while it does its thing with absolutely no indication of what update is running or how long until it finishes. Of note, the last batch of R710 boxes I did took very nearly one full hour to run all the updates.

Repository Manager boot discs are how I update all the ESXi hosts I work with. I have had horrible luck getting the Life-Cycle Updater to work.

Dell has this? God damn it! I know finding IBMs BMOC is a pain in the ass but I didn't even know Dell had one.

The boot CD is damn annoying and takes forever, but my new ESXi hosts out of the box have a firmware bug where the Lifecycle controller crashes trying to update firmware for anything when my specific 10 gig NICs with a specific NIC firmware are installed in the system. The fix? Take the NICs out and run the lifecycle controller, which doesn't update the NICs, use the boot CD which lags behind and doesn't any newer NIC firmware, or just wait until Q4 when Dell will have an updated version of the LCC that will see and work with the borked NIC firmware. Even better, of my new hosts....some are like this and some have different NIC firmware that works fine with the LCC.

Pro tip, make 1 Boot CD per model of server. Combining all of your updates for all of your models makes a larger load of crap that reports errors when you do boot to the CD.

Unfortunately there doesn't seem to be a budnle or anything for the T320 using that DRM. I'm still playing around with setting up the OME so I may get somewhere there but so far this is reminding me of IBM System Director and why I stopped trying to make it work.

 

[Barmaglot]

Just download the CentOS/OMSA LiveDVD and use it to flash the RHEL updates.

 

[Rick25]

That a controller will go on an array over the long weekend. Upside is that there isn't any data on that array yet so I can deal with it on Tuesday.

 

[RicDavis]

dlp[/url]":jjf1szbf]
It was called Services in the boot menu before, now I think it's Lifecycle controller. You can boot up off of it, give it a network address, point it to ftp.dell.com and pull updates. They can lag behind the latest updates though.

That's the thing, sorry, yesterday had been a long day, F10 on boot to enter it.

 

[Xon]

Xerox scanners/photocopiers randomly alter numbers in scanned documents. Apparently JBIG2 compression does lolz-things when faced with simliarish low resolution images.

Just goes to show, poorly configured software can seriously screw with someone's day. And continue todo so for a long time after.

 

[ferzerp]

That I will find a boatload of 1TB volumes with average IO/s under 10 and peaks under 200 IO/s on small 15k drives on a storage array with sorely overtaxed, oversized SATA drives and write cache exhaustion issues. <sigh>

 

[afidel]

Xon[/url]":2ziv57dz]Xerox scanners/photocopiers randomly alter numbers in scanned documents. Apparently JBIG2 compression does lolz-things when faced with simliarish low resolution images.

Just goes to show, poorly configured software can seriously screw with someone's day. And continue todo so for a long time after.

Yeah, that freaked me out last night, luckily I talked to our telecom guy who admins our MFP's and all of ours are defaulted to higher image quality which doesn't use JBIG2 and the profile used to scan to our document repository has it set as a required field so we're not affected. I can only imagine the pure panic for organizations where that isn't the case!

 

[ncrand]

TIL that if you put a discrete graphics card that includes HDMI in some HP desktop systems, it disables on-board audio and requires you to make a BIOS change in order to fix it.

 

[Danger Mouse]

afidel[/url]":1gl64o85]

Xon[/url]":1gl64o85]Xerox scanners/photocopiers randomly alter numbers in scanned documents. Apparently JBIG2 compression does lolz-things when faced with simliarish low resolution images.

Just goes to show, poorly configured software can seriously screw with someone's day. And continue todo so for a long time after.

Yeah, that freaked me out last night, luckily I talked to our telecom guy who admins our MFP's and all of ours are defaulted to higher image quality which doesn't use JBIG2 and the profile used to scan to our document repository has it set as a required field so we're not affected. I can only imagine the pure panic for organizations where that isn't the case!

Guess what brand MFDs we use?

 

[Rick25]

That today (and over the weekend) a whole lot of stuff decided to give up it's magic smoke

Eql controller - Ticket opened, controller onsite in 2.5hrs
HD in HP array - Ticket open, drive dispatched. VM left by courier company that was not possible to understand the call in number or the dispatch number. Only thing clear was that it was being delivered tomorrow....did I mention 4hr onsite on the enclosure. Called back in and it should be getting here about now
Load Balancer - GUI is dead, console is dead but somehow it's still passing traffic. Failed to passive node and powered down. Came back up asking for boot media, RMA in place
Storage array - got error message over the weekend that replication failed. Got there this mornig to find the array turned off and it won't come back on. Looks like PS fried but it's not a field replaceable part so they need to replace the whole chassis.

It's been a bit of a day.

 

[Danger Mouse]

Rick25[/url]":2t0xvl67]
Storage array - got error message over the weekend that replication failed. Got there this mornig to find the array turned off and it won't come back on. Looks like PS fried but it's not a field replaceable part so they need to replace the whole chassis.

It's been a bit of a day.

Our DR array:

-2 failed drives in one enclosure
-datamover failed over
-sp failed over
-replacement drives also failed
-now cannot rebuild that raid group

that was yesterday

Also found out that the data recovery company, can't recover the data, because they were looking for the wrong stuff.

 

[Rick25]

New 6.0.6 firmware is out for Equallogic arrays. Mainly cosmetic stuff so I think I'll stick to 6.0.5 since it's been out for a bit and they haven't issued a patch.

 

[antiwraith]

Rick25[/url]":5z7kxf95]New 6.0.6 firmware is out for Equallogic arrays. Mainly cosmetic stuff so I think I'll stick to 6.0.5 since it's been out for a bit and they haven't issued a patch.

Truth!

Plus I went to 6.0.5 last week, so of course 6.0.6 is out. :bigdumbgrin:

 

[ronelson]

I learned today that one of our appliances has hotswap drives, but you need to reboot to start the rebuild. That's... helpful. Ironically, the drive died during provisioning so there's nothing valuable on it, but still...

 

[SandyTech]

ronelson[/url]":v6e6l56c]I learned today that one of our appliances has hotswap drives, but you need to reboot to start the rebuild. That's... helpful. Ironically, the drive died during provisioning so there's nothing valuable on it, but still...

Talk about pointless... Although I'm not precisely surprised given some of the way some of the PBX appliances I've dealt with are about things like that.

Has anyone tried any of the LenovoEMC products that came out of Lenovo's purchase of Iomega? I need to get some inexpensive bulk storage to get all of our voicemails off of our hosted PBX cluster's local storage and onto something easier to manage (and backup).

 

[Barmaglot]

I've deployed a few of iomega's NAS boxes shortly before the Lenovo acquisition, and they're remarkably dumb. As in, no iSCSI LUN masking dumb.

 

[Danger Mouse]

Barmaglot[/url]":32s59sni]I've deployed a few of iomega's NAS boxes shortly before the Lenovo acquisition, and they're remarkably dumb. As in, no iSCSI LUN masking dumb.

That's good to know

I've got a pair of those waiting deployment. Only 4TB each, I think, but that's certainly 4TB more than nothing.

 

[Danger Mouse]

...that inevitably, when there's an issue with the network setup in a building, it falls back to the contractor having screwed up the config a bit.

In this case it was DHCP options for a WAP setup.

And now, thinking about it, I think I know why some of my phones are screwing up, because they're using the same DHCP pool for cisco VOIP phones as well as WAPs.

I'm thinking there may be some conflicting DHCP options at work here.

EDIT: Yup, option 60 can be set for only one device per dhcp scope/pool/network (depending upon what you're using). Same for option 43, although option 43 can have more than one IP address in hex, it can be an issue still.

Fascinating stuff. And now I see that the previous laborious manual WAP configuration done by the network staff left at the contractor, are due to the incorrect setup of the dhcp pools and respective subnets and vlan assignments for switchports.

 

[BitPoet]

Carpeted floor tiles exist in data centers.

 

[Demoulous]

BitPoet[/url]":23o6pi1u]Carpeted floor tiles exist in data centers.

Sadly, yes they do. In our case due to ancient derp when the DC was commissioned here, and then no bugger giving funding to replace them during the 7 sessions of the M&A waltz. Thankfully we never have cause to lift them, as the air con and cable management are all in the ceiling void. However, it still irritates greatly.

 

[stash]

Carpeted floor tiles are awesome, because you get to use this:

 

[dlp]

That is terrifying.

 

[WingMan]

theevilsharpie[/url]":227ukl1p]

Digitlman[/url]":227ukl1p]HP: That 600G 10k SAS replacement drive is on backorder so we'll get it to you in 3-5 business days.

Me: Bull fucking shit. You find one from somewhere and get it to me tomorrow.

So the RAID5 array for my backups is swinging in the wind down a drive. I have everything spooled to tape, so if another drive went it would not be the end of the world, just a giant PITA.

Plus, I just ordered a refurb'd "spare" from a VAR that will arrive tomorrow, but I still want HP to scramble and do the right thing.

A few years ago, I was working with a small office that had a brand-new HP Proliant ML350 running ESXi. It was intermittently freezing due to an I/O failure, and working with HP over the phone, we determined that the integrated SmartArray controller was failing. This was around 12:00PM.

ME: That's unfortunate. We did purchase the 4-hour response care pack. Can I expect a technician to arrive with a replacement motherboard some time this afternoon?

HP: I'm sorry sir. We don't have any replacement parts available.

ME: Err... what? This server is a common model that's still in production, and this customer is located in one of the largest metro areas in the United States. You're telling me that you don't have any parts available?

HP: We do, but they are reserved for our 6-hour Call-to-Repair customers.

ME: WTF

The only way for me to get same-day repair service was to beg for a temporary discrete SmartArray controller, and the tech to install that arrived at 8:00PM. 4-hour response my ass

4 hour response means that they respond to your ticket within 4 hours, not fix the problem in 4 hours. You would need a 4 hour Call to Repair in order to get that service. They have to keep a certain amount of equipment in stock for those customers who did pay for fast recovery/repair service since if they gave them out to people who didn't pay for it and a customer who did called in right after then they would be in breach of that contract. You get what is available in stock for NBD customers, not what is in stock for their higher obligation contracts. You should really be mad at yourself for not getting the service contract you thought you needed. Our critical infrastructure has 4 hour call to repair on it, the rest has NBD with 4 hour call to response.

 

[sryan2k1]

That sucks. Dell's Pro support is you get the part (and/or tech) in X hours from when the dispatch goes in. So 4 hour pro support means parts in hand in < 4 hours, there's no tricky middle tier.

 

[Rick25]

That whole Call to Repair is new. It never used to be a SKU that they listed. We've always bought the 4hr 24x7 and it was 4hrs for the tech and/or equipment to be onsite.

Our Equallogic's have 4hr response and we get the part in that time...usually less.

 

[skg_loc]

Danger Mouse[/url]":2n5u527j]

afidel[/url]":2n5u527j]

Xon[/url]":2n5u527j]Xerox scanners/photocopiers randomly alter numbers in scanned documents. Apparently JBIG2 compression does lolz-things when faced with simliarish low resolution images.

Just goes to show, poorly configured software can seriously screw with someone's day. And continue todo so for a long time after.

Yeah, that freaked me out last night, luckily I talked to our telecom guy who admins our MFP's and all of ours are defaulted to higher image quality which doesn't use JBIG2 and the profile used to scan to our document repository has it set as a required field so we're not affected. I can only imagine the pure panic for organizations where that isn't the case!

Guess what brand MFDs we use?

ThisMorningIL after reading this article before leaving the office yesterday, that this is not enabled for the Workflow scanning configuration, however is for the more commonly used email scanning feature. We recently did a refresh on all of our MFP's and used the cloning feature to deploy said config across 70+ devices. Let the panic ensue...

 

[afidel]

Yeah, we get 6 hour call to repair on all our HP gear, doesn't cost that much for the first 3 years of uplift. Be aware that there's a clause that gives them an out during the first 90 days of availability of a new model due to normal parts constraints. With hypervisors and live migration I'm not sure it's really necessary anymore but it's a cheap insurance policy in case some freak event should cause us to lose a significant number of hosts at the same time, when downtime is north of $30k an hour you can buy a LOT of warranty uplifts before coming close to even a partial day event.

 

[sryan2k1]

There are two Dell / EQL / Cisco depots within about 25 minutes of here (one at DTW and the other in Troy). We've had EQL drives show up in ~45 minutes from when the dispatch goes in.


I think we had a Sup720 show up in about an hour and a half. It's also the only 720 I've ever seen fail.

 

[ronelson]

There are two Dell / EQL / Cisco depots within about 25 minutes of here (one at DTW and the other in Troy). We've had EQL drives show up in ~45 minutes from when the dispatch goes in.

Sadly, no-one ever looks at dispatch/depot info when locating DCs. We had to wait 3 days for that hard drive in that appliance with the hotswap array issue, though admittedly we did not drive it hard because it wasn't live and we switched to deploying the standby VM version as the primary unit.

 

[Barmaglot]

Goddammit, I just spent two hours trying to figure out why an IIS6 website was working fine in some parts, and throwing 404s on everything in others, before I realized that some idiot used a mapped drive for virtual directory paths and set the start menu to Windows 95 mode, which meant that whenever I accessed that drive, I simply went to start -> run, due to the lack of My Computer shortcut in start menu. It only clicked when I rebooted the server and suddenly the virtual directories became inaccessible in IIS manager due to mapped drive letter disappearing.