what did you learn today? (part 2)

Danger Mouse

Ars Legatus Legionis
38,791
Subscriptor
TIL, the key to smooth transition from windows vCenter to VCSA is making sure the windows vCenter is solid and simple. Back before the upgrade to 6.0 (U3), I went through and cleaned up any/all extensions/plugins/WTF that were giving off error alerts in the GUI.

That included a non removable Nexus 1000v install.

While the migration took some time, everything went by the book. Woulda been faster on faster storage, but that's always how it goes.


Do you have links to a doc on how to purge 1000v if it's been left as artifacts because I have one.

Same old extension removal through the MOB interface on the vcenter's webgui was enough for me after doing all the uninstalls, luckily.

Then when I did the individual host upgrades, I actually saved the host profiles and did clean installs.

That was to confirm any file bits were gone from the hosts.

----

EDIT: So, it looks like even though I told it to only install a PSC, the PSC somehow started acting like the vCenter and grabbed the licenses and other bits. So when the vcenter was reinstalled on the old windows VM, it acted as though the licenses were already in use and so forth.

Shutting down and deleting the old VMs seems to be the only way forward, since I can't restore easily since commvault was reliant upon vcenter in the vm backup/restore scenario.

The DR time lagged copy did fire up, but turns out the local admin password isn't what I thought it was and that was going to be a different fight to get it to boot from an external ISO.

So, forget it.

Nuke and pave on vcenter and go straight to VCSA plus PSC in preparation for 6.5, which will have to wait until the commvault is upgraded to v11.

Some fiddling will be required to get things back where I like them, but most of the settings appear to be preserved on the hosts, so that's more than plenty.

The alternative was waiting for 6+ hours for the backup to restore, assuming I got that working.
 

Danger Mouse

Ars Legatus Legionis
38,791
Subscriptor
TIL not only did I long ago go over the max number of domains (spam) that you can block in O365's EOP, that making transport rules to do effectively the same thing (redirect to the quarantine), that we are now at the max limit for text in the transport rules. :/

I have to return to making intermediate blocks at ye olde Edge Transport.
 

Danger Mouse

Ars Legatus Legionis
38,791
Subscriptor
TIL, the key to smooth transition from windows vCenter to VCSA is making sure the windows vCenter is solid and simple. Back before the upgrade to 6.0 (U3), I went through and cleaned up any/all extensions/plugins/WTF that were giving off error alerts in the GUI.

That included a non removable Nexus 1000v install.

While the migration took some time, everything went by the book. Woulda been faster on faster storage, but that's always how it goes.


Do you have links to a doc on how to purge 1000v if it's been left as artifacts because I have one.

Our 6.0u3 VCSA keeps giving password errors when we try to upgrade to 6.5u1?2? and we've had a ticket open with VMware for almost a month. It's been about as productive as you would expect. We can log in with the password we have, we can manage everything, but we can't run the upgrade.

See if there is an interim upgrade?

Check NTP of course and it sounds like it could actually be a certificate error instead of NTP.
 

hawkbox

Ars Legatus Legionis
13,914
Subscriptor
TIL, the key to smooth transition from windows vCenter to VCSA is making sure the windows vCenter is solid and simple. Back before the upgrade to 6.0 (U3), I went through and cleaned up any/all extensions/plugins/WTF that were giving off error alerts in the GUI.

That included a non removable Nexus 1000v install.

While the migration took some time, everything went by the book. Woulda been faster on faster storage, but that's always how it goes.


Do you have links to a doc on how to purge 1000v if it's been left as artifacts because I have one.

Our 6.0u3 VCSA keeps giving password errors when we try to upgrade to 6.5u1?2? and we've had a ticket open with VMware for almost a month. It's been about as productive as you would expect. We can log in with the password we have, we can manage everything, but we can't run the upgrade.

See if there is an interim upgrade?

Check NTP of course and it sounds like it could actually be a certificate error instead of NTP.

I'm not handling it right now, I'm letting one of my minions run with it. But I would hope to god after 3 weeks that VMware support would have at least checked those things.
 

Barmaglot

Ars Legatus Legionis
11,783
Subscriptor
Okay, it's been a while since I deployed Windows on HP bare metal, but... what the fuck HPE? What the actual fuck?! Whose fucking bright idea was it to make a 'support' website that gives you hundreds of downloads in a randomly-ordered list without functional filters, that can be loaded TEN GODDAMN RANDOM ITEMS AT A TIME, and where every click sends you to the top of the page? Is HPE web design run some sort of hentacle space alien monsters these days? I thought that the old support site that would eat your serial number or model name and tell you 'nuh huh, this doesn't actually exist' was the bottomless pit of web design, but compared to the new 'responsive' one, it was the fucking pinnacle of usability.
 
Okay, it's been a while since I deployed Windows on HP bare metal, but... what the fuck HPE? What the actual fuck?! Whose fucking bright idea was it to make a 'support' website that gives you hundreds of downloads in a randomly-ordered list without functional filters, that can be loaded TEN GODDAMN RANDOM ITEMS AT A TIME, and where every click sends you to the top of the page? Is HPE web design run some sort of hentacle space alien monsters these days? I thought that the old support site that would eat your serial number or model name and tell you 'nuh huh, this doesn't actually exist' was the bottomless pit of web design, but compared to the new 'responsive' one, it was the fucking pinnacle of usability.
Not something I deal with, being a Dell customer (though their new site layout is also garbage), and I'm sorry for your pain, but thanks for the laugh that I passed around our NOC.
 
TIL, the key to smooth transition from windows vCenter to VCSA is making sure the windows vCenter is solid and simple. Back before the upgrade to 6.0 (U3), I went through and cleaned up any/all extensions/plugins/WTF that were giving off error alerts in the GUI.

That included a non removable Nexus 1000v install.

While the migration took some time, everything went by the book. Woulda been faster on faster storage, but that's always how it goes.


Do you have links to a doc on how to purge 1000v if it's been left as artifacts because I have one.

Our 6.0u3 VCSA keeps giving password errors when we try to upgrade to 6.5u1?2? and we've had a ticket open with VMware for almost a month. It's been about as productive as you would expect. We can log in with the password we have, we can manage everything, but we can't run the upgrade.

See if there is an interim upgrade?

Check NTP of course and it sounds like it could actually be a certificate error instead of NTP.

I'm not handling it right now, I'm letting one of my minions run with it. But I would hope to god after 3 weeks that VMware support would have at least checked those things.

6.0U3 is newer than 6.5 IIRC

Edit: 6.0U3 is FEB 2017, whereas 6.5 is NOV 2016 and the release date is what matters so verify the update you are going to in 6.5 is later than what you have for 6.0UX

Edit 2: take a look at vdcrepadmin to check sso nodes and partner status, etc. - KB2127057, it might help point you to where the issue is?
 

afidel

Ars Legatus Legionis
18,165
Subscriptor
Okay, it's been a while since I deployed Windows on HP bare metal, but... what the fuck HPE? What the actual fuck?! Whose fucking bright idea was it to make a 'support' website that gives you hundreds of downloads in a randomly-ordered list without functional filters, that can be loaded TEN GODDAMN RANDOM ITEMS AT A TIME, and where every click sends you to the top of the page? Is HPE web design run some sort of hentacle space alien monsters these days? I thought that the old support site that would eat your serial number or model name and tell you 'nuh huh, this doesn't actually exist' was the bottomless pit of web design, but compared to the new 'responsive' one, it was the fucking pinnacle of usability.
Is there a reason you're hunting for individual drivers instead of using the lifecycle controller to do a guided install? From Gen8 onwards we found it was actually easier to do baremetal than it was with the old SmartStart CD\DVD.
 

hawkbox

Ars Legatus Legionis
13,914
Subscriptor
TIL, the key to smooth transition from windows vCenter to VCSA is making sure the windows vCenter is solid and simple. Back before the upgrade to 6.0 (U3), I went through and cleaned up any/all extensions/plugins/WTF that were giving off error alerts in the GUI.

That included a non removable Nexus 1000v install.

While the migration took some time, everything went by the book. Woulda been faster on faster storage, but that's always how it goes.


Do you have links to a doc on how to purge 1000v if it's been left as artifacts because I have one.

Our 6.0u3 VCSA keeps giving password errors when we try to upgrade to 6.5u1?2? and we've had a ticket open with VMware for almost a month. It's been about as productive as you would expect. We can log in with the password we have, we can manage everything, but we can't run the upgrade.

See if there is an interim upgrade?

Check NTP of course and it sounds like it could actually be a certificate error instead of NTP.

I'm not handling it right now, I'm letting one of my minions run with it. But I would hope to god after 3 weeks that VMware support would have at least checked those things.

6.0U3 is newer than 6.5 IIRC

Edit: 6.0U3 is FEB 2017, whereas 6.5 is NOV 2016 and the release date is what matters so verify the update you are going to in 6.5 is later than what you have for 6.0UX

Edit 2: take a look at vdcrepadmin to check sso nodes and partner status, etc. - KB2127057, it might help point you to where the issue is?

I'm aware of the 6.0u3 and 6.5 issue, that isn't the problem. The VCSA is a single integrated setup with the PSC built in. I'll get them to look a the repadmin info but I'll be surprised if that is relevant.
 

Barmaglot

Ars Legatus Legionis
11,783
Subscriptor
Is there a reason you're hunting for individual drivers instead of using the lifecycle controller to do a guided install? From Gen8 onwards we found it was actually easier to do baremetal than it was with the old SmartStart CD\DVD.

Because I didn't want to wait for the hojillion jigglybytes of ProLiant 'Service' Pack to come through the acoustic coupler that HP is using for their CDN? Ha ha, joke's on me.

Freudian slip or incredible typo?

Neither.
 

afidel

Ars Legatus Legionis
18,165
Subscriptor
TIL that moving some files off of one virtual filer can apparently bog things down so much that a different virtual on the same box is severely affected. Started getting reports a little after 10am that users were having trouble logging into our financials application and at the same time a NOC alert went out that users couldn't print from said app. Went into the servers that host that app and check the logs and sure enough there are logs starting a little after 9am on most of them talking about slow logins, not failures mind you just slow. But around 10am our monitoring started recording a bunch of failed login attempts.

At first I thought it might be a bad AV DAT because a few of the reports online about the event ID mentioned it and these servers still run McAfee (ugh) but a check of the DAT date shows yesterday afternoon so I'm pretty sure it's not that. Then my coworker tells me that their print process goes to a folder on a certain network share, turns out the profiles for this app are on the same server. Finally ping that server and ping times are 4-7ms with spikes to 70+, thing is it's in the same datacenter and my servers at least are on 10gig so I expect <1ms response times. Starting poking around and most of the filers are as expected <1ms but a total of 3 are reporting these high times. At that point I turn it over to the server and storage team. Turns out the root cause was they were archiving some old cases to free up space because the highly restricted drive on another virtual on the same cluster was low on space and they're out of ability to add more storage. Not sure if there's any kind of QoS that can be done on the NetApps to keep chatty neighbors from taking the stuff we rely on down but man was that an interesting troubleshooting session.
 

Danger Mouse

Ars Legatus Legionis
38,791
Subscriptor
I installed some Ubiquiti cameras today to try them out. Pretty impressive system for the price. It's my first experience with their equipment, and now I'm considering a couple of their switches for the camera system when it's all built out.

Cisco is trying to sell us on their cloud managed camera kit from Meraki.

We'll see how that goes....
 

akro

Ars Scholae Palatinae
1,309
come through the acoustic coupler that HP is using for their CDN

I actually laughed. Bravo.


Smart Start doesn't work on Gen 8 or later... intelligent provisioning is built into the firmware (nand flash partition) which is what you would use...

I think it's f9 or f10 during bootup...

I don't remember anymore...
 

Danger Mouse

Ars Legatus Legionis
38,791
Subscriptor
Have seen HP intelligent Provisioning to just Not Work, unless the firmware was updated first, which was good for a laugh and a few wasted hours.

Yup, but it is pretty nice when it works.

Nice to have it register itself for you on HPE's website.

Of course, it took a year or so before the part where I have to acknowledge the registration was visible to me.

Then blam, all of the equipment magically was registered and allowing access.


I've been fighting with the EMC/DELL site since the buyout/merger. Only this week did they finally fix it so I can get in and register support cases and download stuff that I had access to for years.

They also reset my damn password which I didn't need reset. Oh well.
 

Klockwerk

Ars Praefectus
3,757
Subscriptor
~stuff

EDIT: So, it looks like even though I told it to only install a PSC, the PSC somehow started acting like the vCenter and grabbed the licenses and other bits. So when the vcenter was reinstalled on the old windows VM, it acted as though the licenses were already in use and so forth.

Shutting down and deleting the old VMs seems to be the only way forward, since I can't restore easily since commvault was reliant upon vcenter in the vm backup/restore scenario.

The DR time lagged copy did fire up, but turns out the local admin password isn't what I thought it was and that was going to be a different fight to get it to boot from an external ISO.

So, forget it.

Nuke and pave on vcenter and go straight to VCSA plus PSC in preparation for 6.5, which will have to wait until the commvault is upgraded to v11.

Some fiddling will be required to get things back where I like them, but most of the settings appear to be preserved on the hosts, so that's more than plenty.

The alternative was waiting for 6+ hours for the backup to restore, assuming I got that working.

You've probably just clean installed by now, but I'm pretty sure commvault can restore vms to individual hosts as well as a cluster. Not sure if the restore is any good if you're using distributed switches though
 

Danger Mouse

Ars Legatus Legionis
38,791
Subscriptor
~stuff

EDIT: So, it looks like even though I told it to only install a PSC, the PSC somehow started acting like the vCenter and grabbed the licenses and other bits. So when the vcenter was reinstalled on the old windows VM, it acted as though the licenses were already in use and so forth.

Shutting down and deleting the old VMs seems to be the only way forward, since I can't restore easily since commvault was reliant upon vcenter in the vm backup/restore scenario.

The DR time lagged copy did fire up, but turns out the local admin password isn't what I thought it was and that was going to be a different fight to get it to boot from an external ISO.

So, forget it.

Nuke and pave on vcenter and go straight to VCSA plus PSC in preparation for 6.5, which will have to wait until the commvault is upgraded to v11.

Some fiddling will be required to get things back where I like them, but most of the settings appear to be preserved on the hosts, so that's more than plenty.

The alternative was waiting for 6+ hours for the backup to restore, assuming I got that working.

You've probably just clean installed by now, but I'm pretty sure commvault can restore vms to individual hosts as well as a cluster. Not sure if the restore is any good if you're using distributed switches though

Standard vSwitches.

I think I figured out what happened. Somehow I must have triggered "vcsa with external psc" instead of just "external psc". I may have blinked when I clicked or someone walked into the room.

Either way, clean install.

Standard vSwitches.

I needed to stay on ESXi 6.0 either way, since I only have Commvault 10. Commvault 11 is another upgrade that has to be done before ESXi 6.5 can go in.
 

hawkbox

Ars Legatus Legionis
13,914
Subscriptor
Have seen HP intelligent Provisioning to just Not Work, unless the firmware was updated first, which was good for a laugh and a few wasted hours.

Yup, but it is pretty nice when it works.

Nice to have it register itself for you on HPE's website.

Of course, it took a year or so before the part where I have to acknowledge the registration was visible to me.

Then blam, all of the equipment magically was registered and allowing access.


I've been fighting with the EMC/DELL site since the buyout/merger. Only this week did they finally fix it so I can get in and register support cases and download stuff that I had access to for years.

They also reset my damn password which I didn't need reset. Oh well.

The server build/order page is complete shit now unfortunately, makes it damned hard for me to eyeball a server for a site.
 

afidel

Ars Legatus Legionis
18,165
Subscriptor
Have seen HP intelligent Provisioning to just Not Work, unless the firmware was updated first, which was good for a laugh and a few wasted hours.

Yup, but it is pretty nice when it works.

Nice to have it register itself for you on HPE's website.

Of course, it took a year or so before the part where I have to acknowledge the registration was visible to me.

Then blam, all of the equipment magically was registered and allowing access.


I've been fighting with the EMC/DELL site since the buyout/merger. Only this week did they finally fix it so I can get in and register support cases and download stuff that I had access to for years.

They also reset my damn password which I didn't need reset. Oh well.

The server build/order page is complete shit now unfortunately, makes it damned hard for me to eyeball a server for a site.
Holy hell yes it is. I had to spec a server for my dads company a few months ago and it was ungodly slow. I thought it was the site not liking Firefox Mobile so decided I would try it on the desktop later. Tried all the major browsers and they were all crap. Tried over 3 days too so it wasn't a temporary blip either. I have zero idea how anyone could find that slowness acceptable, in any company I've worked for our customer facing presence being that slow would have been a Sev 1 all hands emergency.
 

Paladin

Ars Legatus Legionis
33,531
Subscriptor
Well, over the last week, I have learned that it is a massive pain in the butt when the boss man decides it is worth it to save a few bucks and buy drives that were pulled from a 3Par or something to use with a Cisco UCS C Series rack server. My desk is now host to a couple of Dell r410 servers with an SSD to boot up on their SATA port (with an external power for the drive from a USB to SATA adapter kit, cause the rack server has no extra molex or sata power cables inside it) because it is the only thing I could find sitting around with a RAID card I could re-flash to be a dumb HBA to expose these HP 2TB nearline SAS drives to an operating system so I can reformat them from 520 Byte allocation sectors to 512 Byte sectors so they will work with the LSI 9271-8i RAID cards in the Cisco UCS servers. :eek: :facepalm:

Yeah, that was a lot of fun. It was a rush thing to get the UCS servers integrated, which I did accomplish by the crazy deadline of 2 days to get them on the system and setup with a profile to integrate with OpenStack for use as Ceph storage nodes, at which point I realized that only about 12 of the 40 drives that were put in the servers would come online. It took a couple days of research to catch that they were formatted for 520 bytes and a couple more trying various methods to make them work anyway, talk to them on a desktop machine, then finally the re-flash procedure for a Dell H200 to make it an LSI 9211-8i in IT mode (passthrough HBA instead of RAID card with bootable BIOS).

Now that I have that procedure completed, it just takes about 8 hours per drive to reformat. 8 drives at a time in the 2 servers and about 24 drives to go... :( :facepalm:

Thank goodness for big, fat, noise isolating headphones and catching up on The Expanse while I sit at my desk watching blinking green lights.
I've got firewalls I could be working on but no room to put them anywhere with all this other junk all over the place.

This is why we can't have nice things (like a quickly growing business). We could buy proper products but instead we save money on the purchase and make up the difference in time wasted and projects delayed until no one wants to work on them anymore.
 

CPX

Ars Legatus Legionis
27,044
Subscriptor++
TIL that Ars jumped the shark.

Well, it's not the first time, but .. just .. can't comprehend.

I guess he forgot just how whack...

Seriousness, though, I've never relied on Ars alone for mobile equipment. Especially phones. I don't do Apple devices, but Google can apparently do no wrong in the UX department despite my utter and enduring hatred of flat design.
 
TIL that Ars jumped the shark.

Well, it's not the first time, but .. just .. can't comprehend.

You just learned this? Continual click bait headlines, “mesh” WiFi reviews with a single node and internet limited tests, etc didn’t show you that long ago?

Edit: Also, this isn’t the first completely stupid article they’ve recently posted. There was one recently by a guy who was apparently proud of his abuse of retail workers. I have a strong suspicion that this type of junk is on purpose because quality isn’t what is important to them anymore. Page views are. It is obvious that now that is their metric of choice, and it doesn’t matter what the quality is, as long as the clicks keep coming. Angering people gets traffic, so if you are judging your success on traffic only, you’re incentivized to be an asshat. It’s like all those gawker stories from a while back Gave them bad ideas.
 

hawkbox

Ars Legatus Legionis
13,914
Subscriptor
Well, over the last week, I have learned that it is a massive pain in the butt when the boss man decides it is worth it to save a few bucks and buy drives that were pulled from a 3Par or something to use with a Cisco UCS C Series rack server. My desk is now host to a couple of Dell r410 servers with an SSD to boot up on their SATA port (with an external power for the drive from a USB to SATA adapter kit, cause the rack server has no extra molex or sata power cables inside it) because it is the only thing I could find sitting around with a RAID card I could re-flash to be a dumb HBA to expose these HP 2TB nearline SAS drives to an operating system so I can reformat them from 520 Byte allocation sectors to 512 Byte sectors so they will work with the LSI 9271-8i RAID cards in the Cisco UCS servers. :eek: :facepalm:

Yeah, that was a lot of fun. It was a rush thing to get the UCS servers integrated, which I did accomplish by the crazy deadline of 2 days to get them on the system and setup with a profile to integrate with OpenStack for use as Ceph storage nodes, at which point I realized that only about 12 of the 40 drives that were put in the servers would come online. It took a couple days of research to catch that they were formatted for 520 bytes and a couple more trying various methods to make them work anyway, talk to them on a desktop machine, then finally the re-flash procedure for a Dell H200 to make it an LSI 9211-8i in IT mode (passthrough HBA instead of RAID card with bootable BIOS).

Now that I have that procedure completed, it just takes about 8 hours per drive to reformat. 8 drives at a time in the 2 servers and about 24 drives to go... :( :facepalm:

Thank goodness for big, fat, noise isolating headphones and catching up on The Expanse while I sit at my desk watching blinking green lights.
I've got firewalls I could be working on but no room to put them anywhere with all this other junk all over the place.

This is why we can't have nice things (like a quickly growing business). We could buy proper products but instead we save money on the purchase and make up the difference in time wasted and projects delayed until no one wants to work on them anymore.

But hard drives cost money and your time is free right? :facepalm:
 

Paladin

Ars Legatus Legionis
33,531
Subscriptor
Well, over the last week, I have learned that it is a massive pain in the butt when the boss man decides it is worth it to save a few bucks and buy drives that were pulled from a 3Par or something to use with a Cisco UCS C Series rack server. My desk is now host to a couple of Dell r410 servers with an SSD to boot up on their SATA port (with an external power for the drive from a USB to SATA adapter kit, cause the rack server has no extra molex or sata power cables inside it) because it is the only thing I could find sitting around with a RAID card I could re-flash to be a dumb HBA to expose these HP 2TB nearline SAS drives to an operating system so I can reformat them from 520 Byte allocation sectors to 512 Byte sectors so they will work with the LSI 9271-8i RAID cards in the Cisco UCS servers. :eek: :facepalm:

Yeah, that was a lot of fun. It was a rush thing to get the UCS servers integrated, which I did accomplish by the crazy deadline of 2 days to get them on the system and setup with a profile to integrate with OpenStack for use as Ceph storage nodes, at which point I realized that only about 12 of the 40 drives that were put in the servers would come online. It took a couple days of research to catch that they were formatted for 520 bytes and a couple more trying various methods to make them work anyway, talk to them on a desktop machine, then finally the re-flash procedure for a Dell H200 to make it an LSI 9211-8i in IT mode (passthrough HBA instead of RAID card with bootable BIOS).

Now that I have that procedure completed, it just takes about 8 hours per drive to reformat. 8 drives at a time in the 2 servers and about 24 drives to go... :( :facepalm:

Thank goodness for big, fat, noise isolating headphones and catching up on The Expanse while I sit at my desk watching blinking green lights.
I've got firewalls I could be working on but no room to put them anywhere with all this other junk all over the place.

This is why we can't have nice things (like a quickly growing business). We could buy proper products but instead we save money on the purchase and make up the difference in time wasted and projects delayed until no one wants to work on them anymore.

But hard drives cost money and your time is free right? :facepalm:
Yeah exactly, but he should realize that it is not, and his is not either when I pester him about the frustrating process.
 

Danger Mouse

Ars Legatus Legionis
38,791
Subscriptor
TIL I learned that some times the only way to delete a file is using 8.3 syntax in the command line.

Or a different OS like Linux or BSD. Or I suppose if I spent enough time digging out the alternate keyboard strokes using the alt+ etc, it would work that way.

I wonder if those linux\unix utility imports to Windows would work better than the standard windows commands...hmm