what did you learn today?

[PaveHawk-]

Doing the server rebuild now. One card is 2.xx firmware, the other is 3.xx firmware.

What were the chances...

 

[PaveHawk-]

HP SUM is starting to piss me off. It keeps failing on the scan for any number of invalid reasons (usually failing saying that it cant download a file, then prevents me from moving on to install the files it could find.

Just fucking scan you piece of shit, that file is not coming down so leave it the fuck alone. ARGH.

 

[PaveHawk-]

HP SUM is starting to piss me off. It keeps failing on the scan for any number of invalid reasons (usually failing saying that it cant download a file, then prevents me from moving on to install the files it could find.

Just fucking scan you piece of shit, that file is not coming down so leave it the fuck alone. ARGH.

Can you grab the necessary file from HP's ftp site?

Tried all that Turns out, it cant recognise that it has a valid file sitting in the directory. Still wants to complain.

Eh, I went and installed files manually to make it work.

 

[PaveHawk-]

Cos this server is PCI-E 1.0, I've only got half the bandwidth going to the card

Means its topping out at 10k IOPs @ 670MB/s on 64KB 100% random (100GB file).

I want my 1.1GB/s damnit!

Still, this is pretty good - gonna test a couple of queries against EPICOR, hopefully we can move some of the problematic ones down from 9mins to sub 2mins.

Straight away though, this server has become immensely more responsive under WS2012 from WS2008 (non-R2). The next step will be to see about virtualising this environment on a pair of BL6xx blades.

Also, since I spent ages hunting for this, here is a link to the guide from HP on configuring Fusion-IO with SQL. Its basically 4k sector sizes, with 64k NTFS cluster sizes.

HP Manuals and reference guides for Fusion-IO (admittedly, this is for a C-class blade system, but the general reference guides apply regardless).

 

[PaveHawk-]

I haven't posted in here for a while, because after a while it starts to become unbelievable.

Today, however, my boss has directed me to analyze previous historical data to predict when the next DDoS attack will occur, and where it will be coming from. Buddy, if I could that, I would be making a lot more than you're paying me, and I wouldn't be working for you.

No luck finding a new role at all?

 

[PaveHawk-]

I think there's a CSU campus out there? What of Harvey Mudd college? Too far away? (assumes openings)

If there's a community college there, try your hand at part time/adjunct prof if necessary?

Wait, you paint such a horrific picture of .edu and you're suggesting he gets a job in there? Did Josh hurt you in some past life? :bigdumbgrin:

 

[PaveHawk-]

I'm finding I'm making a ridiculous number of mistakes recently. I've made about 7-9 in the last 3 weeks.

I'm drastically slowing things down so I can avoid it, but I'm still making them (the last was a collation setting for the re-install of an SQL server).

Sigh.

In other news, I'm publicly announcing that I'm taking the ICND2 exam (took and passed ICND1 in December after I got my 4Q12 objective of CCNA certification split between quarters) later this week so someone will hold my feet to the fire, 'cause I'm sure as hell not doing it myself.

Nice, congrats. I need to do the same, preferably before the summer update to the CCNA exam

It will more or less guarantee me a promotion along with a raise of approximately 15%

Summer update?!

Hm, time to restart my CCNA study again.

 

[PaveHawk-]

TIL that our Exchange 2010 server:

a) really hates it when it gets less than 3GB free on C:, even though mailboxes and logs are on separate drives.
b) will silently enter backpressure mode
c) won't log any events to that effect
d) Postini won't complain about the constant 400 errors. We just won't get any mail.

Fun.

Changing between Backpressure levels is always logged in the Event Log. I think they are in the 10000 to 10005 Event ID area, but that may be off.

If you did hit medium or high backpressure it is probably because the drive that holds your mail.que database or transaction logs (Install Drive by default) went under the free space threshold.

Default thresholds are listed here.

Well, maybe it wasn't technically backpressure because there definitely are no events (in Application Log right?). But increasing the free space on C: (deleting OWA access logs) instantly made the 400 errors go away. :shrug"

Verify where the mail queue is. If its C, then you have a problem.

If this is a virtual server, create another thick provisioned disk and follow MS's steps to move the queue database

Bam, problem solved!

Though maybe institute a clean up process for your logs

 

[PaveHawk-]

The Trainsignal course has been awesome and a steal at $49/month for all their courses (VMware, Cisco, Microsoft, Citrix, etc.).

Holy shit. Thats actually pretty awesome.

I've got the Cisco books, a C2851, C878, 3750, 3750E and a 3560G as my environment - that thing might be better for me.

 

[PaveHawk-]

Geez, that's massive overkill for the CCNA... you could do the CCNP with that amount of equipment, and it's new enough/powerful enough to get you most of a CCIE R&S lab...

euri, have you looked at their practice exams? Are they any good? I'm looking for some additional practice tests for the CCNP SWITCH before I go and take it for real...

hah, yeah. I used to run a hosting company. Killed it cos it was too much of a time sink for very little return. Admittedly my return was poor because I'd buy high end gear so that everything was setup properly. Thus the 3750E etc.

I kept 1 of most of the comms gear that I thought might come in handy.

 

[PaveHawk-]

I'm finding I'm making a ridiculous number of mistakes recently. I've made about 7-9 in the last 3 weeks.

I've become very single-threaded lately. Most of my errors stem from multi-tasking - not necessarily doing the wrong thing because of it, but skipping steps or wasting time trying to figure out what step I was on before interruption #72 intruded.

How are you finding the change? Not getting interrupted is a massive problem I'm experiencing, which tends to hurt my prep when doing tasks.

I think I'll go back to basics for a little, 1 task at a time only. It may piss some people off, but I need to eliminate these mistakes.

 

[PaveHawk-]

...that Digicert has a great tool to do the certificate request powershell statement for Exchange 2007 as well as 2010.

Pity that the Comodo interface is so dang antiquidated And, their support portal has just one account per customer, which is fine, except that means I need to involve my manager each time I have to reset his password and get in after it gets locked

Digicert are awesome. I've got >500 odd certs out there with various clients, and I'm slowly moving them all to Digicert.

One of our client's Exchange servers had their cert renewed, but the guy doing the update didnt understand about the root certificate issues that may arise. They periodically check for all the certs they have registered, picked up an issue and called us. We're in AU. They waited till our business hours, called and explained the problem - then sent all the relevant links on the issue and how to fix it.

Seriously, thats awesome customer service.

 

[PaveHawk-]

Gotta provide a quote for a client for 3PB of storage, BDM ran straight to our reseller for EMC who said VMAX!!!

Its video and lots of jpgs (security and video camera feeds). Surely Isilon makes more sense?

[edit]

Oh, and the guy literally replied back with "VMAX!!!". I'd screenshot the email if I could.

 

[PaveHawk-]

For 3PB you'll be doing at least one round of competitive RFPs, and likely at least one Proof of Concept.

For IP surveillance, I'd certainly lean toward NAS over SAN, because NAS is simpler and should have less server and network/fabric overhead, depending on architecture. If the 3PB accumulates over time, a scale-out solution probably works best because you can add storage incrementally without incurring the Capex and Opex of spinning a bunch of empty storage.

For an Isilon solution, twenty-one 4u NL400s with one 10GBASE interface each will do 3PB now that 4TB SATA drives are available for each 36-drive node.

It seems likely that VMAX generates bigger commissions or spiffs than Isilon right now, which would account for the reaction.

Yeah, this is gonna be a 18 month long process from what I'm starting to see.

A lot of the infrastructure that will rely on the storage is getting changed as well. So, gotta schedule meetings and sort out what the various vendor's requirements are and see what we get.

One vendor is already adamant that they cannot have SAN or NAS storage for their portion of the environment. It absolutely has to be bare metal local storage boxes for their systems, else it will never work. Their storage requirements are already sitting at 1.2 PB of that 3 PB.

Yeah, I'm really provisioning you the 14 servers you want with 1.2PB of "high performance, fast RAID 5". You're writing a ton of data, you want local storage in RAID 5. Are you an idiot?

 

[PaveHawk-]

Bonus, I have to deal with that idiot from above. And I need to get 2 reports out today that both require ~8 hours of work each.

I've got 4 out of 8 hours left in the day (well, 9 since I came in early) and "the business will be extremely disappointed" if they're not done.

 

[PaveHawk-]

If it makes you feel any better, I have been given 160 hours of billable work that is supposed to be finished between now and the end of the month.

Well, lets not start comparing work required that is due at the end of the month. The business is expecting me to complete significantly more than 160 hours before month's end.

But, enough of that.

I'm asking all my EMC people to give me indicative quotes for things, surprisingly they're nowhere to be found. Curious.

I'm hungry.

 

[PaveHawk-]

-a very nice private sector senior systems admin position is open again, almost like it's been waiting for me. Time to put in the application...

Get out, for the love of god get out!

I need to get serious, I need to get in touch with a few CIOs that I know and get some career advice on how to jumpstart things and get my sorry ass off my current employer payroll. The shit they pulled today is unbelievable, and I'm scared I'm going to lose it when I have a meeting with the director and my manager (that I asked for) to discuss why they think its acceptable I push so hard to get the work done that I cant take a lunch break.

 

[PaveHawk-]

I don't know the specifics you're talking about but I rather doubt they are using XenApp 6.5 then: so far, we're up to 5 private patches applied to all servers to fix "rare" issues that more or less killed all servicing for several hours each time several workaround to disable functions that work fine but only up until they fail miserably. That's after 3 weeks in production for the new system (we rebuilt it completely, AD and all, and migrated users and data from a XenApp 4.5 farm)

And I can tell you that after fixing this morning's mess (and still not having found any reason why random STAs prevents the whole setup for working after reboot), I'm really not inclined to schedule weekly reboots.

Historically, at least in PS4.x and XA5 days, Citrix recommended scheduling a weekly reboot to reduce the risk of overly contended resources on a server. Thats what DM is talking about.

 

[PaveHawk-]

When did Cisco release the 3850 series of switches? About the only genuinely "big" thing about them is that they have a WLC onboard.

 

[PaveHawk-]

A few months ago.

It's not a WLC onboard. The switching silicon has the functionality built in, meaning it's not a switch with a WLC bolted on like that old 3750 option.

The stacking is also a lot faster.

Would you use it? Serious question.

[edit]

Wait, could you stack a 3850 with a 3750X without any significant issues?

[edit2]

Nope. Besides, it'd drop from 480Gb to 64Gb on the backplane.

 

[PaveHawk-]

and most important of all
Ordered my new PC for home. I must be getting old, that seems awful cheap for a bad ass box.

Got any specific games in mind to play? Or getting ready for battlefield 4?

 

[PaveHawk-]

Awesome. Gotta prep for presenting a $20 million proposal. I have 4 days to write my content and prepare.

 

[PaveHawk-]

Cheers Rick25 I think I can do this easily, just gotta work out the overall theme I'm presenting with. If I work that out, the rest should go well.

Kittens? I bet it was kittens.

Why not puppies? Puppies are cooler.

 

[PaveHawk-]

Office 2013 appears to use more resources, specifically CPU, than Office 2010, according to a post on the Citrix blogs.

I'll have to see how much this will impact densities on XenApp, as opposed to the VDI tests they performed.

Interesting nonetheless.

 

[PaveHawk-]

Yet another lesson in client stupidity: a client deleted half the log files on an active/mounted Exchange database because they were running out of space.

The weird bit was that he didnt simply grab the last lot of files. He grabbed a 5 hour span right smack bang in the middle of the logs.

 

[PaveHawk-]

Exchange logs are automatically truncated when a full (exchange aware) db backup is performed. My guess is the backups weren't running right.

National public holiday on Friday meant that the client wasnt in to change the tapes, leading to the backups not running. I should also point out that their DB grew by 40GB (approx 20% of the original size) overnight as they pumped some PSTs back into the DB after being told that they were risking corruption by running PSTs over the network.

Basically the client pays us to support their network and starts randomly doing shit without consulting us. Then they complain that it should be covered under their fixed price support agreement, ignoring the rather large item that excludes any screw ups created by their staff.

Admittedly, the bigger issue I see is that the logs havent been moved off to their own separate drive. When the server was provisioned, their DB only had around 25GB of data, so it didnt make sense to have a dedicated volume for logs at the time.

[edit]

Just to address/reinforce techix' point, this client doesnt actually believe in backups. They view them as a largely unnecessary expense since they havent yet had the need to rely on them.

 

[PaveHawk-]

Hope they at least cut the logs to another file location rather then an actual delete....

Cut, paste, select all, shift-delete.

Thats the exact order, as relayed by the individual who did it. What could possibly go wrong?

Apparently they confused copy and cut or something. I dunno.

Stupid people are stupid.