what did you learn today?

[abenn]

That Hyper-V can't do SCSI pass through!

Just starting down the virtualization path and decided to move the tape portion of my backup process to a VM. Nope, can't be done. Hyper-V can't pass a SCSI tape drive through to the VM.

So, I have to maintain an extra server just so I can perform a weekly dump to tape. That's really efficient MS!

 

[sryan2k1]

ESX can, <3 VMWare.

 

[bigmikebrooklyn]

we currently use public folders on exchange to have a shared calendar and a group inbox for each department in our company.

Today I learned that Microsoft's cloud email really doesn't have a good solution to public folders.
they say "oh, just use a share point folder", but you cannot use a mail enabled sharepoint folder, so you can't have mail to the "public folders".
you can use a mail enabled sharepoint folder if you have a local sharepoint server and you are not using their cloud sharepoint offering, but you have to have an exchange server present on your local network to mail enable the local sharepoint folder, thus defeating most of the purpose in moving email to the cloud.

the do have a license sku for (if i remember correctly) a "deskless worker" account that is a mail folder and shared calendar, but it is $1.99 a month.

i edited because i had initially called it a "desktop" account instead of a deskless worker account. because sometimes those nuero-transmitters take a while to wander across the ol' synaptical gap...

 

[moobycow]

When the electrician comes in to replace the timer on the lights for the your floor, it would be best if you reminded him to not accidentally put the IT room circuits on said timer as well.

 

[Tsun]

Originally posted by moobycow:
When the electrician comes in to replace the timer on the lights for the your floor, it would be best if you reminded him to not accidentally put the IT room circuits on said timer as well.

hahaha.. sorry to laugh at your misfortune but hehe..

 

[moobycow]

quote:
hahaha.. sorry to laugh at your misfortune but hehe..

I actually laughed as well. It is just one in a long line of screw-ups by the only 'approved' vendor for our building.

 

[bigmikebrooklyn]

Today I learned several things:
the vendor with the lowest quoted hardware price will not necessarily be the vendor with the lowest quoted bottom line.

when the ups guy decides to be a lazy ass and doesn't call the numbers prominently posted on the door next to the freight elevator and instead makes 3 check marks on an info notice and scans the barcode, that 13 packages get delayed delivery until the next day, and unless you can walk your ass across the 59th street bridge to long island city and get to the ups central office between exactly 5:40 and 6:00pm, the parts and manuals you have been waiting on will be delayed yet another day.

I also learned that every time you try to verify a new domain on the microsoft cloud email system, that it issues a new hostname to add as a C-record to that domains DNS entry and the old one no longer works, and you have to rush your vendor to make the change in time for the domain to be verified.

I learned that running 44 xp based print server VM's for a specialized SAP app, and 2 2k3 vm's on an ESXi 3.5 host with a grand total of approx. 32,000mhz of cpu cycles available is probably pushing the envelope a little.

Finally i learned that i should probably be making about twice what i am being paid according to the research i am doing. Damn -- --

 

[sryan2k1]

Today I learned with a MD3000i IAF (Instant availability init/format) works great on a new disk group, however if you are adding a drive to an existing group to add more storage into a virtual disk, it wont let you assign the space until the array has finished adding the drive to the array and calculating/adding the parity to it.


Looks like this VM is going to have to wait until Monday to get it's iSCSI storage.

 

[Rick25]

That making Cat 6 patch cables is a PITA. (Trying for a very neat datacenter)

Most Cat 6 RJ-45 ends come with a loader to help stagger the cables

Some vendors don't include loaders for the RJ45 ends....how you're supposed to get them in and offset without one is beyond me

 

[BajanDude]

That I really should pay attention to my rule that says 'No upgrades on a Friday'. However, had we obeyed that, and encountered the problems I ran in to, we'd have lost 3 business days of time trying to solve the problem. Then again, we'd have had support too (cursed 12x5 support contract).

What was I doing? Upgrading from Yosemite 8.7 to Yosemite 8.8. I have spent 30 hours or more on this upgrade since Friday 6 PM. It is now Sunday 9 PM.

On the plus side, at least I can use iptables + the 'downgrade clients' functionality of YB to stagger my rollback to 8.7.

 

[scorp508]

Originally posted by Rick25:
That making Cat 6 patch cables is a PITA. (Trying for a very neat datacenter)

Most Cat 6 RJ-45 ends come with a loader to help stagger the cables

Some vendors don't include loaders for the RJ45 ends....how you're supposed to get them in and offset without one is beyond me

I love watching the network guys at work who make these. They bang em' out so fast it a mind boggling and I struggle to make one in 10 minutes.

 

[Rick25]

Scorp, can you check and see if they use a 1 piece CAT 6 RJ45 end or a 2 piece (with loader).

 

[khoaled]

I learned that evolution pbx can't be upgraded from version 2 to 3 without a complete wipe/reinstall. Also the version 2 backup isn't compatible with version 3. Made for a long day.

 

[scorp508]

Originally posted by Rick25:
Scorp, can you check and see if they use a 1 piece CAT 6 RJ45 end or a 2 piece (with loader).

I don't know what a loader looks like, but they just strip the insulation off of the end of the CAT6, cut the center conduit back just enough, de-twist the pairs in their hand, arrange them, shove em' into the cable end, crimp then test with a Fluke device.

 

[pokrface]

I've learned that some vendors lie. And when you call them on it, they backpedal and lie to cover their lies. And when you call them on THAT, they say you don't know what you're talking about. Sigh.

 

[Heresiarch]

That a major update for a business critical application will be released 3 days before we need to do our most business-critical process with it. And yes, we need the update for the process to complete.

 

[the maddman]

Originally posted by Pokrface:
I've learned that some vendors lie. And when you call them on it, they backpedal and lie to cover their lies. And when you call them on THAT, they say you don't know what you're talking about. Sigh.

Rule #1 here is "Vendors lie." We once ran an Oracle server for 2 and a half years for an application we bought that didn't exist. F'ing vendors.

 

[finni]

Originally posted by the maddman:
We once ran an Oracle server for 2 and a half years for an application we bought that didn't exist. F'ing vendors.

Story please!

 

[Whittey]

Originally posted by the maddman:

Originally posted by Pokrface:
I've learned that some vendors lie. And when you call them on it, they backpedal and lie to cover their lies. And when you call them on THAT, they say you don't know what you're talking about. Sigh.

Rule #1 here is "Vendors lie." We once ran an Oracle server for 2 and a half years for an application we bought that didn't exist. F'ing vendors.

See, you're looking at it the wrong way. You ran an application without a single outage (scheduled or nonscheduled) for 2.5 years. Bravo to you and your team!


-=Whittey=-

 

[the maddman]

Originally posted by finni:

Originally posted by the maddman:
We once ran an Oracle server for 2 and a half years for an application we bought that didn't exist. F'ing vendors.

Story please!

Alright:

I work in local government, so we get to deal with Public Safety. This is it's own whole world of screwy-ness.

Back around 2001, we were involved in a Computer aided dispatch system upgrade. We moved from a program that was written by a local cowboy coder (There's more horror stories there) to a shared system running on an AS/400 in another city. This was a six figure cost, even without us having to purchase the iSeries from IBM.

Part of this upgrade included a software package to track the patrol cars via GPS. Let me count the ways this project failed:

Order server from HP. Server is delivered. Company then informs us they don't do installs on-site, the server must she shipped from CA to FL and back for the install.

UPS gives the server it's own special care on the cross country trip. Huge holes in both boxes it was packed in. I end up replacing 90% of the hardware in the server getting it to POST again.

Vendor setup the server for "maximum disk space" so everything is installed on a 6 disk RAID 0 array. My boss is pissed.

We pay to fly someone in to install the rest of the system, and when we complain about the RAID 0 array, he whips out a CD-R and does the super secret install again on a RAID5 volume.

We equipped the mobile computers with Sierra Wireless CDPD modems with embedded GPS. Specifically recommended by the Vendor. We fire up the first unit, install the software and it asks for the serial port the GPS device is on. Sierra Wireless GPS units communicate over UDP. We spend 12 months explaining this to the developers. What really got me going was that everything we bought was right from their recommended hardware list. We paid extra for the fancy modems on the vendors recommendation, yet they had NEVER EVEN SEEN ONE.

We install a 42" flat panel in dispatch to display the moving maps. Software will be delivered "real soon now."


The fail just kept going and going. We didn't get AVL working until 3 generations of modem replacements, and only then because the Sierra Wireless EDGE modems had a USB connection and would let us spit the GPS data out via serial. By the time the vendor got it working, the application migrated from Oracle on windows to MSSQL, so we never used the Oracle box for anything, but politically, since so much money was spent on this, we dutifully kept the machine up and running so when people asked about it, we had something flashy to point to.

 

[Arbelac]

Geez. Your tax dollars at work.

 

[finni]

Originally posted by the maddman:
Ouch ouch ouch ouch fuckery ouch ouch

I'm kinda sorry that I asked. It's like slowing down to look at a car accident.

 

[Darthkim]

Originally posted by Pokrface:
I've learned that some vendors lie. And when you call them on it, they backpedal and lie to cover their lies. And when you call them on THAT, they say you don't know what you're talking about. Sigh.

I can't decide whether its worse for the VAR or the Vendor to lie. For now, i think it's the Vendor, since they should know all the answers or have direct connection to those that can provide the answers..

I cut my VARs a little slack, since they are a third party to the vendor, but when they bill themselves to be an expert....

sigh.

 

[ronelson]

CDPD

This was your problem, right here! I have never seen CDPD anything work well...though unlike you, I have seen them work... -- --

 

[bigmikebrooklyn]

Originally posted by goForth:

Talk about "pffft". Doesn't this pale in comparison to moronic Easterners waking someone on the West coast at 5:00am in the morning?

Try dealing with folks in Hawaii or Guam.

try dealing with Japan. try getting someone on the line to do a critical process at their datacenter at 2pm eastern time. try having a video conference with the bigwigs at 2 pm Japan time.

 

[the maddman]

Originally posted by ronelson:

CDPD

This was your problem, right here! I have never seen CDPD anything work well...though unlike you, I have seen them work... -- --

Well, we got to cheat there.... CDPD had a specific priority for public safety customers. So we'd kick people off the cell towers if they were full. -- -- I mean, it was 19.2kbit per sec, but it was rock solid.

Switching to GPRS/EDGE/HSDPA sucked tons. We're on a "private" apn, so all our mobiles are bottle-necked by a leased line that has to run half way across the country, and due to cost considerations it's only 512k shared between 70+ vehicles.

 

[the maddman]

Originally posted by Arbelac:
Geez. Your tax dollars at work.

No Comment.

Originally posted by finni:

Originally posted by the maddman:
Ouch ouch ouch ouch fuckery ouch ouch

I'm kinda sorry that I asked. It's like slowing down to look at a car accident.

I had to quote that again, your summary makes me laugh.

 

[Incarnate]

Originally posted by Rick25:
That making Cat 6 patch cables is a PITA. (Trying for a very neat datacenter)

Most Cat 6 RJ-45 ends come with a loader to help stagger the cables

Some vendors don't include loaders for the RJ45 ends....how you're supposed to get them in and offset without one is beyond me

I learned this a while ago, but you can order cables in pretty much any length you want, and its usually cheaper than paying a system engineer $25-$50/hour wasting time doing that. -- --

 

[Paulie]

That when CA sends a bad signature update for eTrust, it makes quite a mess. sigh

edit: Make that bad engine update.

 

[jaericho]

Originally posted by Incarnate:

Originally posted by Rick25:
That making Cat 6 patch cables is a PITA. (Trying for a very neat datacenter)

Most Cat 6 RJ-45 ends come with a loader to help stagger the cables

Some vendors don't include loaders for the RJ45 ends....how you're supposed to get them in and offset without one is beyond me

I learned this a while ago, but you can order cables in pretty much any length you want, and its usually cheaper than paying a system engineer $25-$50/hour wasting time doing that. -- --

I remember taking a simple networking class when I first started college. One day we were taught how to make networking cable by making our own cable. At the end of the day, the teacher asked the class: "What did we learn today?" He then answered his question by replying: "We learned why we buy network cable. The machines do it faster and better than we do. Class dismissed."

I still remember that very clearly. -- --

 

[janus]

except....

Don't buy CDW branded Belkin cable to use as patch cable. The boots are too wide to plug in multiple connections right next to each other on a switch.

 

[bigmikebrooklyn]

today i learned that some microsoft online cloud email tech guys are better than others.
the deskless worker license will not work as a public folder replacement because you can only add delegates via AD or via Outlook client, and deskless workers are only able to access OWA.

I also learned that if on the 10th of the month the powers above command you to "let there be SAN" and "...VMware in production at least as a proxy" but it has to be up by the end of the month, you better know your netapp options and trust the vendors a lot... and buy an air mattress for the server room, not that you'll be sleeping much.

 

[Spatula]

Don't buy CDW branded Belkin cable to use as patch cable

A boss of mine was very adamant about this. I could understand after seeing the stacks of KVMs that had been pulled out of the server room in a rage.

 

[Rick25]

I know but it's the only way to ensure that we've got cables of the right length on hand when we need it. Just need to find the best CAT 6 end to use. The cables test fine in everything but NEXT.

 

[BajanDude]

That when CA sends a bad signature update for eTrust, it makes quite a mess. sigh

We learnt that we were lucky. Our servers appear untouched, and for whatever reason, the majority of our clients aren't updating their signatures/engines and never got the update.

 

[Spatula]

That you will still have to use CDs for deployment every once in a while.

That a person can try to make a file server available in all of these ways, at the same time:
-One logon script "net use" style, configured in each user's AD object
-One group policy preference, Replace, in GPO#1
-One group policy preference, Update in GPO#2
-And let's not forget to Loopback Process both GPO#1 and GPO#2, even though the Users and Computers and still in the Users and Computers containers respectively, and they're applied at the domain level.

Oh well, that person hasn't realized that he put me in the Builtin\Administrators group (I assume because he wanted to make sure I was a local admin on each computer), so at least I can remove the logon script and block inheritance to most.

And don't get me started on applying wsus configuration at the domain level, enforcing the GPO, and not understanding why people get pissed when servers choke.

 

[gregthomas]

That no matter how many times you warn your customers against it, Facilities will always put sprinkler heads in customer server rooms.

Had a sprinkler head go off in a server room.....of a hospital....where they consolidated all hospital IT services into that room....under that sprinkler head.


Today I also learned that if you ask nicely enough, you can get a hold of the guy in charge of production of Dell servers at 6pm on a Friday to have your servers built on a Saturday for delivery Monday. Yay Dell.

 

[pokrface]

I learned once again that poor planning on someone else's part really does constitute an emergency for me, if the person who did the poor planning can scream loudly enough to the right people. And now I'm going to spend all of next week designing and implementing a DR replication system that has to be fully tested and in production by COB 31 August, so that the customer can meet his deadline.

The DR requirement then goes away on 15 September, and the entire solution will then be torn down and the customer's data migrated off of storage my group controls and onto a VMware application server.

Who the fuck is driving this crazy bus? And can we at least stop for some ice cream?

 

[ronelson]

I learned that when your customer agrees to purchase their own hardware/software support instead of letting us buy it and include it in their monthly costs, they do not understand that support includes the $20k in licenses they need to connect to our management platform.

I also learned that they are not happy to figure that out on the Friday before a planned Monday migration that will not be happening. Oh well, not our problem. We are Operations - not like we have any money to spend on it anyway!

 

[meglet]

Originally posted by gregthomas:
That no matter how many times you warn your customers against it, Facilities will always put sprinkler heads in customer server rooms.

Had a sprinkler head go off in a server room.....of a hospital....where they consolidated all hospital IT services into that room....under that sprinkler head.


Today I also learned that if you ask nicely enough, you can get a hold of the guy in charge of production of Dell servers at 6pm on a Friday to have your servers built on a Saturday for delivery Monday. Yay Dell.

As someone who was forced by building management to have sprinklers in our server room. . . . OUCH.

When we built our server room, we were in a leased space, and building management flatly refused to leave the sprinklers out of the room, although they were "nice enough" to set the trigger temp way over what the halon temp was. Not that a higher trigger temp would have helped if one of those sprinkler heads had malfunctioned.

I'm so glad we moved into a hosted DC last year. Not that that doesn't have it's own set of issues.

Oh, and something I learned recently, not from personal experience, fortunately: having UPS and generator power for your DC doesn't do you any good if the panel to switch from grid power is one of the things that caught fire and killed grid power.