what did you learn today?

Status
You're currently viewing only PaveHawk-'s posts. Click here to go back to viewing the entire thread.
Not open for further replies.
Danger Mouse":14cstb0t said:
Even, *I* don't do that. No, wait, I do that, but less and less as time goes on. Your infrastructure can't be that bad. My coworkers are all slowly coming along, some faster than others, into the modern systems admin way of doing things.

Something someone pointed out to me is quite true: They won't miss you, if you aren't gone

HTF will they appreciate you, if you do stuff like that. Trust me. If it's critical and they can't handle it, they'll call you. And if they do F things up because you weren't around, they'll appreciate you more when you come back and fix things up, because whatever it is likely can wait.

And if it can't wait, they'll call you.

If however, you work in some kind of insane work environment where you'll get blamed for not being there to rescue them from their stupidity, get your resume out and about. It's time to go.

I think part of my problem is that for the last 3 years or so, I've been working so hard I forgot to have a life. This year I made a promise to myself that I would reduce my OT, go out and do more stuff thats not IT related and generally not do work related things. That bit is going well, but clearly I need to remove my work email off my iPhone so that I can not check it.

In fact, I'm going to remove it entirely - no more work related stuff on my personal phone.

Its also never been about making sure things are operating or even just to know whats going on - I think its a case of "its there, so it'd be cool to know" sort of thing. I need to get a life.

I'm already job hunting, but thats cos I've been here 3 years, complained about the way things occur and then hit a wall with what I can learn in the role from both a technical and management perspective.

Oh, as for how fucked up things might get - since reading all your posts about your environment, my view of things has mellowed out drastically :) It may suck for you, but its been a learning experience for me. I'm gunning for team lead and eventually management roles in the next 5-10 years, I've damn well learnt what sort of TL or manager I dont want to be.
 
Frennzy":3l2ibpnh said:
Strooth. Where I work, they are encouraging users to hook up their personal phones to our infrastructure via a third party platform. In return, you can collect a monthly stipend for using the phone for work purposes. This platform gives the company control and monitoring of what communications and company data go to/from their phone.

It also gives us the power to wipe their phones remotely. Their personal phones.

I also get paid for use of my personal mobile; but the problem for my work is that if it came time to do a remote wipe, I'd know about it long before they could try it. I'm basically the one with the indepth knowledge about all the products we use and how they're setup. This applies for our internal infrastructure as well as our client's infrastructure. I've said "document, document, document" for the internal network, they've said "yes!! On your own time though, because we cant afford to give you internal time for it".
 
euri":31r7tye6 said:
One of my blog posts got linked on Planet V12N (VMware community blogs)!!!

I've just started with this whole blogosphere thing, so it's more than doubled my total hits. :cool:

You could have at least linked to the article, I want to read it! (I searched and found it myself)

[edit]

Not to commit the same crime as euri...

Here is the link: http://www.thatsmyview.net/2011/03/30/r ... urity-7-5/
 
Danger Mouse":3ocfbz9r said:
The segregated accounts thing? Yeah, my manager ordered that. Hell, he ordered it because I recommended it and implemented it for myself. One of my coworkers responded by putting in a local admin account on every system he touched. Um. No. Just, no.

My management is unusually paranoid. Company is a managed services provider/integrator with 50 staff and whilst its not massive, its not a tiny little thing either.

The MD is so paranoid that he, and only he, has the passwords for the Administrator account, Backup Exec service account, Enterprise Vault service accounts and a couple of other key accounts that have access to everything or access to specific things. So when it came time for me to upgrade from BE11 to BE12, BE12 to BE12.5 (etc etc), I had to get him to enter in the service account every single time.

At some point, an employee (fine, me) cottoned on to how a lot of the security permissions were setup. Basically there are 2 other domain admin accounts, of which 1 belongs to the Infrastructure team and the other belongs to the Software Development team. The Software Dev account is unrestricted, but the Infrastructure team's account is given a deny permission on the OU that contains the directors accounts, administrator and other key accounts. I stupidly reported how weak this was (since you could create another account with the restricted infrastructure account, then grant it domain admin to circumvent the deny ACL), and I got lumped with redoing it all and restricting it.

So I worked out their requirements:
- Everyone, who needs it, to have domain admin accounts
- All of these domain admin accounts to be restricted
- Restrictions include to not change other domain admin account passwords, not to change management team (and other designated) account passwords, not to have access to management team (and other designated) mailboxes, not to be able to easily remove restrictions.

After a little playing around I did the following, which is my own notes to work out how to back out of what I did if I needed to:

Code:
Security Group created, called "Restricted Domain Admins":
-          Create central container
-          Change ownership of the container to Enterprise Admins
-          Change permissions of group to enable read, deny write, deny add/remove self as member to the group (“Restricted Domain Admins”) itself

Administrative Users:
-          Create/Move admin users to a central container
-          Add admin users to the security group (“Restricted Domain Admins”)
-          Change ownership of the admin accounts to enterprise admins
-          Change permissions of the admin container to deny write access to the security group

Builtin Folder:
-          Change ownership of folder to enterprise admins
-          Change ownership of following accounts to enterprise admins: administrators, server operators, backup operators, account operators

Objects with specific permissions for the security group:
-          AdminSDHolder (via ADSIEdit.msc) has deny write to "Restricted Domain Admins", is owned by enterprise admins
-          Management and Management_Security groups have deny write to "Restricted Domain Admins"
-          Director group has deny write to "Restricted Domain Admins"
-          Schema Admins / Domain Admins / Enterprise Admins are all owned by Enterprise Admins to "Restricted Domain Admins"
-          Administrator account is owned by Enterprise Admins to "Restricted Domain Admins"
-          Management Team OU is deny read/write for the to "Restricted Domain Admins"

Now, it appeared to do what I wanted to do (except for the mail, since I needed E2010 and RBAC). I was pretty fucking proud of myself for working it out and making it hard to break out of...till I realised that I really only prevented myself since this was simply one way of doing things. I asked another engineer to break out of it, and just as he started the Infrastructure manager got wind of it, watched for literally 45 seconds and promptly told me "Implement it!!!!!!!" (that is a literal copy and paste from the email, you can never have too many exclamation marks apparently).

I had a point in all of this long winded explanation, but I've since forgotten it. Suffice to say, I'm not happy and this is a goddamn boneheaded decision because I never really got to test the impact of this on an Active Directory domain - I don't know if anything is broken as a result of all of these shenanigans :( Oh and also, I've somehow convinced management in all of this to give me an account that is unrestricted in any way, shape or form - so why not just give me the damn administrator password and all the others so I don't need to wait 2 weeks (cos you're on holiday) to do upgrades that you wanted 2 days ago.

[edit]

I thought I had the point again, but no it was just this: the other problem in all of this, is that the directors don't want to let me remove the domain admin access on their accounts even though this is an IT company and they understand the risks of them operating in that fashion. WHAT. THE. FUCK.

[edit2]

I remember part of my point: I need to keep my big mouth shut.
 
Danger Mouse":a8j8d9lm said:
I actually liken my career to the protagonist's in Kentaro Miura's Berserk. Horrible beginnings, middling middle times, finally the horror and then fighting against a hellish fate for eternity.

I'm not a fan of much Japanese manga, but I am a fan of Berserk. Thats probably the most apt description of your environment, based on what you've told us.
 
scorp508":1wg04lo4 said:
[non-IT, but still geeky]

My parallel port based Pocket Programmer 2 from Xtronics won't work in Windows 7. :( Time to buy a new one so I can tune the car. For now the Moates.net Ostrich EEPROM emulator will have to do. :D

Wow, reading this at 6am in the morning didnt help. I swear I read that as "time to buy a new car".

Aaaaanyway.

I just finished an Exchange 2003 -> Exchange 2010 migration for a small client (~70 users, 70GB mail store, single server) and ran into a problem I've never experienced before. Basically a whole raft of user accounts (around 40 or so) were losing their security inheritance for the AD object and I was sitting there scratching my head wondering why. The issue cropped up because it kills ActiveSync completely.

So I've looked around for 5 minutes wondering what could possibly be causing this; then I stumbled on to the issue:

The previous IT vendor, instead of diagnosing an issue, put all the users who needed access to a specific share into the domain administrators group. Apparently some of the users also needed local administrative access to their desktop PCs and this was also the fix for it.

Honest to god, who does this?

In a PM that I wrote to scorp508:

I have a client who has a server that I will call server02. Server02 is/was a Exchange 2003 Standard server running on a Domain Controller. The previous IT group dcpromo'ed the server out before uninstalling Exchange 2003. Right now I cant log on to the server because its just not responding to AD credentials and it almost looks like it was removed through a metadata cleanup.

Same group did that as well.
 
Barmaglot":rnupu40l said:
PaveHawk-":rnupu40l said:
I just finished an Exchange 2003 -> Exchange 2010 migration for a small client (~70 users, 70GB mail store, single server) and ran into a problem I've never experienced before.

What a coincidence, I'm also migrating Exchange 2003 to 2010 right now, 55 users, ~60GB store (kept there by draconian measures), except my problem is different - it's been 3 days since I issued a "move all replicas", and exchangeV1 still appears under public folder instances on the 2003 side.

Actually, I have another problem but I havent looked into it yet. I did move all replicas too, and 1 folder hasnt moved off. I may end up deleting that folder anyway, since it hasnt been in use since 2009, was mail enabled and contains around 2k messages in spam. Apparently they spam filtered all the users, but not PFs.
 
Soko":32m6kar0 said:
I swear to FSM that if I meet any of the coders or QA people responsible for BackupExec 2010 next week @ Symantec Vision, I will beat them sensless, disrobe them with fire, paint thier ass bright red and throw them in the Baboon pit of the nearest zoo. :mad:

Take photos. Lots of them. The cathartic release for an untold number of support personnel would have a profound impact.
 
Apparently we should be selling EMC VNX/VNXe over HP P4000/Dell MD32xx because EMC is a more recognised name in storage and their products must be inherently better in terms of quality and performance even though they're providing xTB of storage via mix of SAS/MDLSAS vs the P4000 with 16 SAS disks or the MD3220 with 24 SAS disks.

The back story is that my company is trying to define its strategy in SAN offerings rather than the adhoc stuff thats been in place. I'm advocating that we use the Dell MD3220 offering, HP P4000 and the VNX/VNXe offerings where they are appropriate. Management and the one other engineer who I've less respect for think offering EMC's line only is a better solution since some of our companies have revolving doors for the role of CIO and the new CIOs would be impressed if they saw we deployed EMC over HP or Dell.

Basically its buying the offering based on name, not on effectiveness as a solution.
 
smaug9":2oei96fb said:
Although, some VARS seem to be put off right now with the 'lack of direction' vibe that HP storage, especially their high-end stuff, is emanating. It's put some in a weird position of saying, "I don't know what's going to happen with the HP line, but I know EMC is solid right now. If you have to buy right now, buy EMC."

I buy a P4300 starter kit for circa $16k AUD, thats 7.2TB raw of SAS. I can also buy a a Dell MD3220 DAS with 3.5TB raw for circa $10k. The EMC VNXe in AU will be $33k for 16TB raw with 12TB of that being MDLSAS.

The example in this case was a client who needed 1TB formatted, projected to grow to 1.7TB in 2 years based on historical usage patterns.

I'd rather recommend a cost effective solution that gives them performance over something that has an EMC badge on it. Thats just me.
 
some organisations don't quite "get" IT. No documentation on business processes or the applications that relate to them, no concept of DR or BC in general. One of the BAs can deal with this, so that gives me a reprieve.

Oh, the CEO has a 21GB mailbox, on Kerio mail server. Mail is their #1 or #2 most relied on system, but it has no redundancy, no support contracts for the hardware.

One of their systems relies on, wait for it, Compaq 1600R and 1800R machines since the software is hard-coded for the motherboards, array controllers and the 9GB SCSI drives in use. The system is so tied down, they're sourcing 9GB drives for these machines around the world from anywhere they can get them (ebay, swap meets etc).
 
finni":2rsjjfjz said:
PaveHawk-":2rsjjfjz said:
Compaq 1600R and 1800R machines since the software is hard-coded for the motherboards, array controllers and the 9GB SCSI drives in use.
Wait. WTF? Do they have business-process software written in assembler? Why would that stuff matter otherwise?

Well, its some core software that runs the entire company written in god only knows what. I havent even touched those yet, but they're moving offices in a few months so I want to see what the attrition rate is when the servers are unracked and racked.

It will be comedy gold, so long as I am not directly or indirectly involved. If I am, it will be hell.
 
RRLSi":30ba0b07 said:
Everyday I am convinced that "developers" have no business working on systems.

The phrase "chmod the world" comes to mind.

Not all devs are bad, the manager (former team-lead) for the dev team at my work is pretty knowledgeable about infrastructure. The added bonus is that if he doesnt know, he'll say so in his typical saffa blunt manner.
 
Matt Wallis":2vo82zzj said:
dredphul":2vo82zzj said:
I know that I'm getting old when I pause and have to marvel how much CPU and network bandwidth I have on my goddamn phone compared to to the Apple II, TRS-80, and Atari 400 desktop computers I started on.

I have a DPT SmartCache IV SCSI controller sitting around at home, mostly just for the fact that it has a M68020 CPU on board, and can take up to 32MB of 72pin memory, which makes it significantly more powerful than the Apple Lisa with an M68000 and 1MB ram I started my computing on.

My first modem was a 2400bps I picked up from somewhere, then a USR14k4, and several cheap broken 28k8 modems before picking up a Swann 33k6 that lasted me for ages. Had cable for a couple of years, then in a new estate with ancient phone lines and no cable, had ISDN until someone freed up a slot on the local exchange for ADSL.

Telstra Australia have much to answer for in terms of poor service and expensive data.

I remember when Microplex offered unlimited hours/data accounts, but had 6 hour cutoffs. Then Optusnet bought them out and later on killed the plans. DingoBlue also was there reselling Optusnet services with unlimited. I went to Primus in the end, they had 4 hour cutoffs, though if you scheduled your dial after midnight to 1am, you could circumvent it and stay on for as long as you wanted.

I still have my old netcomm 56k modem! Wonder if it works...

[edit]

Oh, and I've been working on a client machine that is a Compaq Proliant 800, I've actually forgotten how to use NT4 and its embarrassing.
 
bigmikebrooklyn":2q80wsrq said:
PaveHawk-":2q80wsrq said:
Basic maths skills are still too hard for some:

User: This copy of Office is very old, I think it’s about 17 years old.

Service Desk Team Lead: Wow, what version is it?

User: 2003.

well, you guys ARE on the metric system...
#:sudo flamesuit engage

Look buddy, dont start. :)
 
A desktop PC (Dell Precision T1500) was bought to run our Software Dev's TFS environment, using low quality MLC SSDs and a 3rd party RAID controller - against my recommendations.

If our devs go down, its circa $20-30k/day that we're losing in revenue.

The box died, as I advised management it would. I got told "maybe we should have listened to you".

No shit, hey...
 
ronelson":12rzhwfa said:
The box died, as I advised management it would. I got told "maybe we should have listened to you".
You should be happy. Most people never hear that, even if it is blindingly obvious!

Just exhausting though. There'll be an incident report and some how I'll get the blame for it, its happened that many times before.
 
Fulgan":238wt8ri said:
PaveHawk-":238wt8ri said:
Awesome; a client's server some how reset its permissions on an entire drive (predictably the one holding the shared folders; weirdly the home directories/tsprofiles folders are fine) and I have no idea what happened.

Let me guess: the user swore that "he didn't do nothin'", right ?

The client demanded the administrator password (fine), on the Friday. On the Monday they started having problems. The password was handed to a senior staff member who logged on to try and fix something.

The claim is that nothing was changed.

But apparently it's a coincidence.
 
Rick25":1pgg66ly said:
That the world is conspiring against me ever finishing our Exchange 2010 build/migration. So much for the quiet week to get stuff done.

You too? The internal Exchange 2010 install for my company has been on hold since December due to 3rd party application issues which I cant fix cos its not billable time.

[edit]

In other news, I might have a new job lined up :D
 
Barmaglot":fejgtoqn said:
Depends on what server you have and what you want to do. There's IBM Director Server, but it's a huge unwieldy beast. On the last couple generations, hardware monitoring and email alerting is built into the management processor and works kinda halfway well.

Its an x3650 M2, though the previous provider spec'ed the server without RSA. Just want to avoid going to the client site to bring the server down so I can simply look at its memory config (amongst other things).
 
Barmaglot":1hjd6y5c said:
Pretty sure that M2 comes with an integrated management module that will let you view the memory config; there should be a utility that will let you configure its IP address from Windows without shutting down the server. You'll need an upgrade key (similar to HP Advanced iLO) to get virtual KVM and media, but basic functions are built-in.

I havent found it yet, but glad you're around to provide some useful information. I dont know IBMs all that well, only really deployed HP/Dell in the last 12 months.

So, thank you :)
 
scorp508":17ysijeu said:
I recently took that test as well. I did much better on the top half and just barely squeaked by on the bottom half. I also had sweaty palms waiting for the results to pop up.

Really? I would have expected you to blitz it.

I took 649, didnt find it at all hard - wonder how much of a difference there is between the exams.
 
scorp508":12x8tw74 said:
PaveHawk-":12x8tw74 said:
scorp508":12x8tw74 said:
I recently took that test as well. I did much better on the top half and just barely squeaked by on the bottom half. I also had sweaty palms waiting for the results to pop up.

Really? I would have expected you to blitz it.

ADFS, ADRMS, IPv6, NAP and RRAS aren't things I work with on a daily basis (if at all), so of course Murphy's Law kicked into full throttle and I got hammered with questions on all of those sections.

Ah, Murphy loved me it seems - I got mostly questions I knew the answer to. I didnt get much ADFS/ADRMS...
 
afidel":1ldmksvc said:
It's now been 26 hours since my second case regarding a production down case has been open and I've had a total conversation time of maybe 5 minutes with one recomendation that didn't resolve the problem. This is what we get for a six figure support contract?!? I wouldn't expect this kind of crap from a tier 2 or even a tier 3 vendor let alone an enterprise vendor.

Wow, thats pretty fucked up. I'm curious to know how this ends for HP.
 
stash":grdxcba3 said:
I wouldn't call my experiences horror stories, but the management leaves a lot to be desired. It might be better with SCVMM 2012, but I haven't had a chance to play with it yet. But the current version requires far too many interfaces to setup and manage a host. Failover manager, Hyper-V manager, SCVMM, server manager, etc.

Lots of stupid limitations compared to VMware around modifying VMs. For example, I discovered today that there apparently isn't a way to rename a VM without powering it off. It seems the only hardware you can add to a running VM is a disk. No NICs or additional RAM.

Networking with Hyper-V is...interesting. Once you get used to it, it isn't too bad, but having to rely on third-parties for teaming sucks, especially on a Core server.

Dont forget the snapshotting with Hyper-V; for it to merge back in, you need to power down the VM. That is silly.
 
Status
You're currently viewing only PaveHawk-'s posts. Click here to go back to viewing the entire thread.
Not open for further replies.