what did you learn today?

[Mittens T. Cat]

Even though you've planned things out, stuff will go wrong. Leave a respectable margin for error in project planning.

Also, sometimes you just have a perfect storm of bad luck and bad timing, and nothing can be done about that. Just keep pluggin'!

 

[BajanDude]

Instead of futzing with drivers and pebuilder, just use ultimate boot CD.

 

[Fink_Ployd]

Originally posted by Mittens T. Cat:
Even though you've planned things out, stuff will go wrong. Leave a respectable margin for error in project planning.

Also, sometimes you just have a perfect storm of bad luck and bad timing, and nothing can be done about that. Just keep pluggin'!

That's the normal order of things in sysadmin work. The real secret is to always plan for complete failure, especially when it's something new you're implementing. Always, always, always leave yourself an out.

 

[BajanDude]

++ planning for failure. Pessimistic people make good sysadmins :> (Personally, I've taken to calling it pragmatic.)

pgfouine is wonderful software.

 

[Soko]

Always remember Murphy was an optimist.

 

[Zaphod]

Originally posted by Fink_Ployd:

Originally posted by Mittens T. Cat:
Even though you've planned things out, stuff will go wrong. Leave a respectable margin for error in project planning.

Also, sometimes you just have a perfect storm of bad luck and bad timing, and nothing can be done about that. Just keep pluggin'!

That's the normal order of things in sysadmin work. The real secret is to always plan for complete failure, especially when it's something new you're implementing. Always, always, always leave yourself an out.

A number of years ago I had a data migration that could have taken as little as 10 hours, but my schedule permitted a half dozen 'sub-optimal' contingencies. I hit every one of them and the result was just shy of 100 hours...but I was still on target because I had planned through those possibilities.

...good times...

 

[akro]

I have to rules when working an upgrade etc....

1. Always have a backout plan!
2. Always stick to the plan!

Rarely has a rush decision made at midnight ever worked out well...

 

[shizakapayou]

Originally posted by Mittens T. Cat:
I learned that you must check on:
- Support hours/availability
- Support documentation availability

...before you buy something.

We just purchased an Enhance Tech RS-16 IP-4 and filled it with 2TB Hitachi disks. It works great for Windows iSCSI connectivity, but VMware ESX 4.0 can't attach the LUNs.

Any resolution to this? I've been eying one of those units to setup some VMWare here, honestly because we can afford it. I can deal with the support times but it'd be nice to know it actually works.

As for today, I already knew but had it reinforced that I'll always be the bad guy, even if I'm saving the company's ass.

 

[Heresiarch]

Originally posted by akro:
I have to rules when working an upgrade etc....

1. Always have a backout plan!
2. Always stick to the plan!

Rarely has a rush decision made at midnight ever worked out well...

3. Ensure you have number of $good_takeout_place_that_delivers and/or a minion to collect it.

 

[Danger Mouse]

The good news is, our manpower shortage for the infrastructure rebuilding has been solved.

The bad news is, the contractors will be doing all the "good" stuff that I need to know for my career.

-installing and provisioning a 50tb san
-replacing all of our domain controllers
-replacing all of our dns servers
-replacing our pseudo digital phone system with a Cisco VOIP solution
-provisioning an organization-wide digital document system
-planning, implementing a new campus active directory infrastructure

Yes, that would all be useful to have on my resume instead of "became caretaker to vendor installed solution(s)".

So, how did this state of affairs come about? We have the skillset to do the work internally. Hell, *I* have the skillset to do the work even if most of my coworkers don't, except I was never allowed to do it by being assigned crappy helpdesk workticket after workticket after workticket.

Why was I assigned so much crapwork? Because I'm better/faster at it than my coworkers as well.

FML, I need a new job.

EDIT: I cannot describe my rage adequately. I've been effectively outsourced by my boss. For the amount of money they're paying one contractor for a few weeks, they could have paid my promotional salary for the next few years.

Yes. FML.

Bonus points for me winding up having to fix/rebuild/etc all of the stuff without training/documentation/etc in a year or two, after all of the bond money is gone. Would have been nice to have the experience/knowledge that went into building things in the first place

 

[finni]

If you have the skillset for all that stuff today, why do you need to do it again for your career?

Or are you saying that you've never done it, but could be an effective team-member under someone more experienced? That's a valid gripe - if they're getting a consultant to do normal-but-high-level stuff and not also including knowledge-transfer to the in-house staff, then screw your employer, that's awful.

 

[Danger Mouse]

Originally posted by finni:
If you have the skillset for all that stuff today, why do you need to do it again for your career?

Or are you saying that you've never done it, but could be an effective team-member under someone more experienced? That's a valid gripe - if they're getting a consultant to do normal-but-high-level stuff and not also including knowledge-transfer to the in-house staff, then screw your employer, that's awful.

Yes, it's in my skillset. It's stuff I've done before without training/forenotice.

HOWEVER, it's nice to do it on all the brand new/latest models of stuff to have on your resume instead of model x from y many years ago.

Also, I would much rather do that then "yes, you find your file in your my documents folder" on the phone.

Theoretically, there will be knowledge transfer, but that means we still wind up effectively being IT custodians instead of executing/innovating/etc.

And it's not my employer, it's my boss.

Theoretically, this is done to shield the in-house staff from any screwups and there have been quite a few. However, it's still all stuff that we should be doing in-house.

That and there's the whole salary thing flying around.

 

[meglet]

Originally posted by Heresiarch:

Originally posted by akro:
I have to rules when working an upgrade etc....

1. Always have a backout plan!
2. Always stick to the plan!

Rarely has a rush decision made at midnight ever worked out well...

3. Ensure you have number of $good_takeout_place_that_delivers and/or a minion to collect it.

Sigh. By the time I need dinner tonight, the minions will be off shift, and the delivery places won't deliver.

I hate year end. (Yes, my accounting department is JUST about to start closing last year, a 6-10 hour process, at 2pm PST. And they haven't read the instructions yet.)

Corollary to this experience, re-learned - or reminded - over the last couple days: when people think you have all the answers they need, they won't bother to think for themselves even for the simplest questions, because it's easier to just ask you. See: my accounting department not reading THEIR year-end close procedures.

Maybe I better just order the pizza now and reheat it later.

 

[jdsmith575]

That when management sets the priorities... everything's a priority!

 

[llib]

When renovating a data center, check the specs yourself. Very carefully. All ten million of them. The devil is in the details.

Our new chillers have an operating range of 0-125º. That's fine, except it's going to be -2 on Fri/Sat morning!

Now we're scrambling for portable heaters to put heat on the chillers to keep them from getting too cold. How bizarre is that?!?

My kingdom for water-side economizers! Or just a big-ass outdoor coil and fan! Bad planning. Bad!

 

[akro]

Duct the server room exhaust to the chillers!!!!!


=-D I visit a building routinely that heats it entire building by ducting the heat from the server rooms to the office areas during the winter.

Originally posted by llib:
When renovating a data center, check the specs yourself. Very carefully. All ten million of them. The devil is in the details.

Our new chillers have an operating range of 0-125º. That's fine, except it's going to be -2 on Fri/Sat morning!

Now we're scrambling for portable heaters to put heat on the chillers to keep them from getting too cold. How bizarre is that?!?

My kingdom for water-side economizers! Or just a big-ass outdoor coil and fan! Bad planning. Bad!

 

[MKG]

2 lessons so far this week.

Lesson 1:
39 straight hours is not wise. My advice, try to avoid it.

Lesson 2:
When making a major change, expect everyone to automatically assume you and your shitty work are responsible for everything that went wrong that day. Don't bother to argue facts. They are irrelevant:

It didn't just install itself on your busted ass DL360 G2.

It doesn't affect a vendor's datacenter in California.

It didn't create a super virus.

It didn't change all the traffic lights to green.

The virtual infrastructure doesn't run the shredder.

It doesn't control the night cleaning crew.

It doesn't make coffee in the morning or flush the toilet when you've done your business.

It's not SkyNet. It's sole purpose in this existence is not to hunt you down and kill you.

What it does do, is run very well. Regardless if you don't.

Yes, I'm venting.

Did I mention 39 hours?
<me curls up in fetal position and cries for hours>

 

[Metzen]

HP DL380G6 will not PXE boot into Windows Vista or Windows 7 from a boot-chained PXE-Linux. Not sure if it's just the consumer OS's (doesn't make sense to me), and supposedly Windows 2008 should work (will attempt tomorrow). I'm just trying to setup some performance baselines on a pristine system (namely HDD and RAM performance).

As well, the HP DL380G6 is the highest HDD density server you can get 2U with the expansion bay. 16 2.5" SAS HDD's.

 

[PaveHawk-]

Originally posted by Metzen:
As well, the HP DL380G6 is the highest HDD density server you can get 2U with the expansion bay. 16 2.5" SAS HDD's.

Didnt the Sun x4250 have that? The x4150 sitting next to me has 8 drives and its a 1u. Pretty sure the 4250 has 16 drives as well...

 

[Barmaglot]

Originally posted by Metzen:
As well, the HP DL380G6 is the highest HDD density server you can get 2U with the expansion bay. 16 2.5" SAS HDD's.

http://www.chenbro.com/corpora...s_detail.php?sku=149

 

[montegard]

Sorry for the double quotes but it isn't an Airstack chiller is it? Ours has the same issues, in our instance though the chiller is just too large for our heatload after we virtualized. I'm looking into adding a freecooling module to it. I wonder how hard it would be for us to have facilities add ducting and fans to our bullpen area....

Originally posted by akro:
Duct the server room exhaust to the chillers!!!!!


=-D I visit a building routinely that heats it entire building by ducting the heat from the server rooms to the office areas during the winter.

Originally posted by llib:
When renovating a data center, check the specs yourself. Very carefully. All ten million of them. The devil is in the details.

Our new chillers have an operating range of 0-125º. That's fine, except it's going to be -2 on Fri/Sat morning!

Now we're scrambling for portable heaters to put heat on the chillers to keep them from getting too cold. How bizarre is that?!?

My kingdom for water-side economizers! Or just a big-ass outdoor coil and fan! Bad planning. Bad!

 

[jarends]

People really believed some of these things?

Originally posted by mkg:
2 lessons so far this week.

Lesson 1:
39 straight hours is not wise. My advice, try to avoid it.

Lesson 2:
When making a major change, expect everyone to automatically assume you and your shitty work are responsible for everything that went wrong that day. Don't bother to argue facts. They are irrelevant:

It didn't just install itself on your busted ass DL360 G2.

It doesn't affect a vendor's datacenter in California.

It didn't create a super virus.

It didn't change all the traffic lights to green.

The virtual infrastructure doesn't run the shredder.

It doesn't control the night cleaning crew.

It doesn't make coffee in the morning or flush the toilet when you've done your business.

It's not SkyNet. It's sole purpose in this existence is not to hunt you down and kill you.

What it does do, is run very well. Regardless if you don't.

Yes, I'm venting.

Did I mention 39 hours?
<me curls up in fetal position and cries for hours>

 

[Metzen]

Originally posted by Barmaglot:

Originally posted by Metzen:
As well, the HP DL380G6 is the highest HDD density server you can get 2U with the expansion bay. 16 2.5" SAS HDD's.

http://www.chenbro.com/corpora...s_detail.php?sku=149

I should rephrase...

Among the "Big 3" I was allowed to look at for servers (HP, Dell, IBM), HP was the only one with the appropriate solution.

 

[Metzen]

Originally posted by jarends:
People really believed some of these things?

Originally posted by mkg:
2 lessons so far this week.

Lesson 1:
39 straight hours is not wise. My advice, try to avoid it.

Lesson 2:
When making a major change, expect everyone to automatically assume you and your shitty work are responsible for everything that went wrong that day. Don't bother to argue facts. They are irrelevant:

It didn't just install itself on your busted ass DL360 G2.

It doesn't affect a vendor's datacenter in California.

It didn't create a super virus.

It didn't change all the traffic lights to green.

The virtual infrastructure doesn't run the shredder.

It doesn't control the night cleaning crew.

It doesn't make coffee in the morning or flush the toilet when you've done your business.

It's not SkyNet. It's sole purpose in this existence is not to hunt you down and kill you.

What it does do, is run very well. Regardless if you don't.

Yes, I'm venting.

Did I mention 39 hours?
<me curls up in fetal position and cries for hours>

Haha, I've been in similar situations where, after upgrading a application, someone's mouse dies, keyboard dies, computer won't turn on, etc. and they blame it on the upgrade. They'll be convinced that some hardware issue is a direct result of some application update within Windows!

"My mouse worked just fine with the old version, I want it back!"

 

[nathaniel]

Originally posted by Metzen:

Originally posted by jarends:
People really believed some of these things?

Originally posted by mkg:
2 lessons so far this week.

Lesson 1:
39 straight hours is not wise. My advice, try to avoid it.

Lesson 2:
When making a major change, expect everyone to automatically assume you and your shitty work are responsible for everything that went wrong that day. Don't bother to argue facts. They are irrelevant:

It didn't just install itself on your busted ass DL360 G2.

It doesn't affect a vendor's datacenter in California.

It didn't create a super virus.

It didn't change all the traffic lights to green.

The virtual infrastructure doesn't run the shredder.

It doesn't control the night cleaning crew.

It doesn't make coffee in the morning or flush the toilet when you've done your business.

It's not SkyNet. It's sole purpose in this existence is not to hunt you down and kill you.

What it does do, is run very well. Regardless if you don't.

Yes, I'm venting.

Did I mention 39 hours?
<me curls up in fetal position and cries for hours>

Haha, I've been in similar situations where, after upgrading a application, someone's mouse dies, keyboard dies, computer won't turn on, etc. and they blame it on the upgrade. They'll be convinced that some hardware issue is a direct result of some application update within Windows!

"My mouse worked just fine with the old version, I want it back!"

Saw some of ninkendo's computer duck images in the lounge and decided to look through the collection: http://ninkendo.org/~ken/images/computerduck half of those are my father. I did have a user that kept their bookmarks in a notebook and not the little links I sent out in emails but the redirected ones that are 3+ lines long with variables in them.

 

[nathaniel]

Originally posted by shizakapayou:

Originally posted by Mittens T. Cat:
I learned that you must check on:
- Support hours/availability
- Support documentation availability

...before you buy something.

We just purchased an Enhance Tech RS-16 IP-4 and filled it with 2TB Hitachi disks. It works great for Windows iSCSI connectivity, but VMware ESX 4.0 can't attach the LUNs.

Any resolution to this? I've been eying one of those units to setup some VMWare here, honestly because we can afford it. I can deal with the support times but it'd be nice to know it actually works.

As for today, I already knew but had it reinforced that I'll always be the bad guy, even if I'm saving the company's ass.

Can you make your disk arrays and Logical drives under the 2TB mark then expand the logical drives with vmware's extents? I'm running 12x400GB Cheetahs in one disk array split into two RAID10 1.22TB LUNs which in sphere I use extents to make the two LUNS appear as one disk: http://virtualgeek.typepad.com...and-counter-fud.html
vmware's maximum configurations: http://www.vmware.com/pdf/vsph...sp_40_config_max.pdf

 

[Danger Mouse]

Pro Tip:

If you're a vendor onsite for a multimillion dollar contract, and you decide to graciously buy pizza for your onsite IT crew:

-spend a few extra $20s and buy some for the client's small onsite IT staff
-don't leave that 1 slice of pizza and the pizza box and other trash in the data center for the onsite IT staff to clean up

We know that your crew is getting paid 3x to 4x what we are per hour. We know that you're making a ridiculous profit on all of this (who charges $2000 to install one workgroup switch? and then sends 4 people to do it, where 3 people watch one guy..).

Want to build some good vendor/client relations? Spend another $20 or $40 to buy more of those $5 Dominos special pizzas. No, really. There would be less blazing hatred and more simmering resentment.

Oh and I don't think your crew was too happy either. 2 medium, 1 topping pizzas for 8 guys?

That's good morale building, right there.

 

[finni]

Originally posted by Danger Mouse:
who charges $2000 to install one workgroup switch?

who pays $2000 to install one workgroup switch?

 

[Danger Mouse]

Originally posted by finni:

Originally posted by Danger Mouse:
who charges $2000 to install one workgroup switch?

who pays $2000 to install one workgroup switch?

Remember, I work in .edu

Don't get me started.

 

[bigmikebrooklyn]

Originally posted by finni:

Originally posted by Danger Mouse:
who charges $2000 to install one workgroup switch?

who pays $2000 to install one workgroup switch?

govt/edu would be where i laid my money.
if i were a gamblin man,

 

[chris]

Today I learned that Sonicwall IPS must have been designed by the same guy that did User Account Control in vista.

 

[Metzen]

My first experience with ILO 2 today. Just amazing. Would be absolutely crack-fabulous amazing if the "Virtual Media" was a weeee bit faster than 3Mbps. But, overall... Knock your socks off good.

 

[bigmikebrooklyn]

Originally posted by Metzen:
My first experience with ILO 2 today. Just amazing. Would be absolutely crack-fabulous amazing if the "Virtual Media" was a weeee bit faster than 3Mbps. But, overall... Knock your socks off good.

today i learned that getting a certain french telecom company named after a secondary color and/or a citrus fruit, to do anything at all within the scope of their contracts or even job descriptions, is like getting kicked in the balls hundreds of times while trying to disarm a bomb with a fast timer.

 

[Paulie]

Replacing a dead Celerra control station is not too bad. Couple hours once parts are on site. SAN keeps chugging away, but sure do miss being able to manage it.

 

[scorp508]

Originally posted by Metzen:
My first experience with ILO 2 today. Just amazing. Would be absolutely crack-fabulous amazing if the "Virtual Media" was a weeee bit faster than 3Mbps. But, overall... Knock your socks off good.

That has been our experience as well, everything is great UNTIL you try to do anything with the virtual media. It's impossible to load an OS with it unless you're counting on being paid by the hour.

 

[Metzen]

Originally posted by scorp508:

Originally posted by Metzen:
My first experience with ILO 2 today. Just amazing. Would be absolutely crack-fabulous amazing if the "Virtual Media" was a weeee bit faster than 3Mbps. But, overall... Knock your socks off good.

That has been our experience as well, everything is great UNTIL you try to do anything with the virtual media. It's impossible to load an OS with it unless you're counting on being paid by the hour.

Wasn't too bad. It took about 90 minutes to install Server 2008 R2 Standard over virtual media.

 

[scorp508]

Originally posted by Metzen:
Wasn't too bad. It took about 90 minutes to install Server 2008 R2 Standard over virtual media.

I guess installing from USB key has me a little spoiled, I hate waiting.

 

[ronelson]

I guess installing from USB key has me a little spoiled, I hate waiting.

Gah, stop whining, you have it good! Our iLO is used across 5 continents. Virtual Media is out for most of it. If you can get it to mount, it will never stay mounted the whole time. And when our normal management can be done over a partial T1, no-one wants to pay for something larger on the off chance that we actually need to use the virtual media functions.

 

[akro]

I know i have said it before but please please please please do network builds for loading boxes. If your using ILO put your iso out on a share and just boot smart start to get the install started from the network. There just isn't much you can do via an emulated USB1.1 port on iLO.

Also if you have iLO advanced you can mount an ISO via HTTP. Not sure if it is any faster not...

Originally posted by ronelson:

I guess installing from USB key has me a little spoiled, I hate waiting.

Gah, stop whining, you have it good! Our iLO is used across 5 continents. Virtual Media is out for most of it. If you can get it to mount, it will never stay mounted the whole time. And when our normal management can be done over a partial T1, no-one wants to pay for something larger on the off chance that we actually need to use the virtual media functions.

 

[scorp508]

Originally posted by akro:
I know i have said it before but please please please please do network builds for loading boxes. If your using ILO put your iso out on a share and just boot smart start to get the install started from the network. There just isn't much you can do via an emulated USB1.1 port on iLO.

Also if you have iLO advanced you can mount an ISO via HTTP. Not sure if it is any faster not...

How about HP just improves the feature to be at least as adequate as their competitors? Performing the same task on an IBM Bladecenter for example is much quicker. Honestly, very happy with the ProLiants this is just like a nasty whitehead on the face of a supermodel.

We're not always able to install over the network from a share for security reasons or limitations of the physical site the box is at.