what did you learn today?

Status
Not open for further replies.

Devin

Ars Tribunus Angusticlavius
8,422
I learned that forwarding all desk calls to my phone is a great idea. Before I did that, people could sense which phone I was not near and leave five messages on that, and when I called them back they would be out of the office.

Now if my company would emerge into the 1990s and let me get email on my phone, all will be right with the world. Well, not all, but some.
 
Soko":1623kpw9 said:
scorp508":1623kpw9 said:
Agreed, but in the end who's fault is that? The admin busting their hump to do what they can with what they have, or the CTO/CIO who didn't plan (or didn't want to!) accordingly for technology updates and then fund the projects? Maybe it's a shared rsponsibility as the admin is the trusted advisor who can show the CxO the potential monetary benefits of upgrading.

Dude - I'm not that lame. Neither is my CIO. We put in 3 fully justified proposals over 4 years to replace our current Exchange server, all of which got shot down by the board.

Same exact analogy, just shot down by a different level of people.
 

Danger Mouse

Ars Legatus Legionis
38,881
Subscriptor
That a certain big copier/printer/services company will sell you a 2 generations old, single HD/single PSU BIG NAMED SERVER COMPANY server and pretend that everything will be fine and dandy forever.

That's right, not even RAID 1. This is on a $2.5 million contract spanning 3 years where the system is supposed to have 5 "9"s for uptime. Yeah, right.

You'd think they could give us something decently fitted out. 2 minutes of blank screen POST plus another 5 minutes to get to the "launching desktop" in the console tells me otherwise.

Oh and the shipping box looked like pandas had played soccer with it.

Did I mention the miniscule power button, which easily got stuck behind the facade preventing boot? I had to dig in with my finger nail clipper to get the little button loose enough to start the system.

Well, at least the rackmount rails were nice. Nice magnetic bits on the ends to help keep the rails steady while you snapped them into place. Better than the HP stuff, but not as versatile as the newer Dell rails.

---

EDIT: oh and recent threads about HP 380G6s fitted with 460 watt PSUs seemed oh so appropriate, when I was behind our "mission critical" VMware HA cluster. Yup, 460 watt PSUs. I haven't officially or unofficially been allowed to touch that system yet, so I have no idea if the systems will stay lit with 1 PSU.
 
That even a conservative firm like Microsoft are prone to levels of stupidity of epic proportions.

I'm at MS TechEd in the Gold Coast, QLD, AU and in the time leading up to there was a big, nay massive, promotion about the whole 'Women in IT' concept. So imagine my surprise when I walk around the expo area and they have this remote control car off-road track with the Halo warthog vehicles "manned" by staff from a group called "Meter maids" (they're pretty well known around here, but I imagine most of the people on this forum dont know about them).

For reference: http://www.metermaids.com/history.htm

Half naked women, pandering to the age old stereotype about IT nerds. During a time when MS is promoting Women in IT.

Dont get me wrong, I love looking at hot scantily clad women, but its really hard to justify that this is a professional development event when things like this occur. Especially given the fight I had trying to convince management that this is a useful event.

Seems like others have similar opinions:
- http://katec.posterous.com/the-auteched ... othes-plot
- http://www.itnews.com.au/News/229858,te ... in-it.aspx

Would have been awkward for the women that were here though, there are surprisingly a larger amount of women at this event than I've seen in the past; probably between 3 to 5% of the attendees.

Also, 2 out of 3 Exchange sessions have been good. 1 was the worst lot of marketing for Hyper V I've seen in a while; its more of a shame since it was sposed to be about E2010 virtualisation.
 

Cossix

Smack-Fu Master, in training
77
Paulie":3w10bknl said:
Anyone have a recommendation on a serial/usb converter that "just works" on both PC and Mac, without funky drivers? I could then retire my old "designed for windows 2000" lappy with a serial port. :) :) :)

The best ones I've ever used are the National Instruments ones, but they're both expensive and require a large large software install.

Your best bet is just to avoid converters that use the FTDI chipset. It doesn't seem to be very stable in Win7. One of the electrical guys had eight bluescreens in a single day with five of those hooked up to his system. Problems went away when we switched to a different chipset.
 

afidel

Ars Legatus Legionis
18,224
Subscriptor
Pokrface":1ak4gbks said:
PaveHawk-":1ak4gbks said:
Half naked women, pandering to the age old stereotype about IT nerds. During a time when MS is promoting Women in IT.
Enterprise IT is still on the whole very much a boy's club. It's still not terribly uncommon for vendors to take customers out to strip joints.

Hmm, my workplace must be very odd then, we're about 50/50 with 2 of 3 director positions filled with women. Of course in my group which is the tech side it's all male so I guess we're a bit stereotypical.
 

sryan2k1

Ars Legatus Legionis
46,568
Subscriptor++
Cossix":2ik36et9 said:
Paulie":2ik36et9 said:
Anyone have a recommendation on a serial/usb converter that "just works" on both PC and Mac, without funky drivers? I could then retire my old "designed for windows 2000" lappy with a serial port. :) :) :)

The best ones I've ever used are the National Instruments ones, but they're both expensive and require a large large software install.

Your best bet is just to avoid converters that use the FTDI chipset. It doesn't seem to be very stable in Win7. One of the electrical guys had eight bluescreens in a single day with five of those hooked up to his system. Problems went away when we switched to a different chipset.


We have a bunch of IOGear GUC-232A's that work wonders, I beleive the chip is Aten and their part number is some variation of that (UC-232A maybe)


But they have 32/64 bit drivers, and works great.


Also, ++ on avoiding FTDI, we use a TTL-USB chip in a few of our products from FTDI and they are nothing but a hassle.
 

Spatula

Ars Tribunus Angusticlavius
9,314
Subscriptor
User accounts have been getting locked out for the last 24+ hours. Domain admin turned off account lockout, but that didn't solve anything, people were still getting locked out. At this moment, 90% of our user accounts are locked out, including service accounts etc.

I was sick of my own account locking out (and I even have the tools to fix it), so I spent 5 minutes looking through Event Logs on the DC (not sure why I have those permissions), found all the attempts were coming from a single computer, and went and unplugged it to bring it back to my office to figure out what is wrong.

If I'm going to have to keep doing the domain admin's job, can I just have it?
 
Spatula":mrc65r5x said:
User accounts have been getting locked out for the last 24+ hours. Domain admin turned off account lockout, but that didn't solve anything, people were still getting locked out. At this moment, 90% of our user accounts are locked out, including service accounts etc.

I was sick of my own account locking out (and I even have the tools to fix it), so I spent 5 minutes looking through Event Logs on the DC (not sure why I have those permissions), found all the attempts were coming from a single computer, and went and unplugged it to bring it back to my office to figure out what is wrong.

If I'm going to have to keep doing the domain admin's job, can I just have it?


Any chance it is an authentication mis-match issue--NTLM v2 being required on the servers but not on the workstations for example.

But, yeah, you can have the job--just tell them some jackass on the net said it was cool.
 

Spatula

Ars Tribunus Angusticlavius
9,314
Subscriptor
Accounts were being locked out one by one as a result of bursts of network logon attempts with incorrect passwords.

I started netstat and plugged the machine into a "gated community" network, and it's going crazy throwing microsoft-ds all over the world.

My guess? XP SP3 machines compromised before they got updated. Hopefully this will be enough fodder to convince them that running a WSUS downstream of our central IT WSUS so that I can tie it into SCCM for OSD is worthwhile.
 

Spatula

Ars Tribunus Angusticlavius
9,314
Subscriptor
There's this wonderful place, called academia...

My position is being reviewed in late September. I have been told to submit what I feel my job description should be. If the money or responsibilities are not close to what I want, it's new job time.

In the meantime, it's fun to hear the domain admin say (32 hours after the problem started, and after I have removed the cause) "I've got the monitoring tools in place now".
 

Heresiarch

Ars Scholae Palatinae
909
I re-learned that our power company couldn't find their arse with both hands. 3 times in the last 5 days we've had multiple-hour valley-wide undervoltage during the working day.

So rather than getting projects done, I've been fucking about babysitting UPS and ensuring clean shutdowns while getting "is the power back on yet?" calls from random fuckwits whose can clearly see their striplights are barely glowing.
 

Spatula

Ars Tribunus Angusticlavius
9,314
Subscriptor
ronelson":amr6gtxg said:
In the meantime, it's fun to hear the domain admin say (32 hours after the problem started, and after I have removed the cause) "I've got the monitoring tools in place now".
Actually, that is fairly impressive. Setting up monitoring after a problem occurs is rare, but important.

A good point. Very good point. But what I used as a "monitoring tool" was looking at the DC event logs, applying one filter, then dumping to Excel and spending 5 minutes playing with the data.

The monitoring tools that are needed here are the more basic - automated alerts based on event log criteria, for example. Which would also have caught an issue I commented on a while back in this thread: last time I bothered to look at the DC event logs (still not sure why I have those permissions), 70% of the entries were red GC errors. Stretching back months.

And 90% of our user accounts are still locked out as I type this. Tomorrow morning is going to be hilarious, I bet they get unlocked one by one via ADUC as people call in or e-mail from coworkers computers.
 

bigmikebrooklyn

Ars Centurion
345
Subscriptor
okay, this is going to take a minute.
friday night, 8:30 pm, me, working hard on finishing up a deployment of new riverbeds.
i don't want to post up an org chart or nothing, but i got the manager of my department who's my boss. the 2 AGM's who are my boss, and a GM who's THE boss. 1 AGM and the GM like to play techie, because in a former life before middle management castrated them with budgetary guidelines and policy concerns they were a developer and a closet computer nerd. GM and AGM have been troubleshooting an outlook issue coming from just 2 of our 22 locations. 32 tcp/IP sessions eventually stack up at the exchange server on users that are on RPCoHTTPS and get cut off from the server until those sessions expire (about 2 hours later). usually when someone tries to send an attachment over 1mb. I was working on it, some lackey helpdesk dude at one of the offices starts getting an attitude and i say, sure, let me wave my wand over the easy button and presto, magicfix. that email gets forwarded to all kinds of management types, my GM says, lay off, i'll troubleshoot this one. 3 months ago.

Que friday evening. THE boss sees that if he turns an option off on the riverbed, the emails suddenly go through. AGM looks at another office's riverbed configuration, see's that the settings of that same option are different. Demands we provide documentation of every setting on all 22 riverbeds. Mind you, i'm balls deep in this deployment and go over there to tell him that the layout and options change with every release of RiOS, and that what he's asking for is a herculean task, but that i can give him a copy of the "sh run" command and he can see what configs have changed from defaults. before i can finish my sentance dude is yelling at me (in front of coworkers, at 8:30 on a friday night) that I can't manage the system and rah rah rah.
i start yelling back, catch my self, go for a walk around the hall and punch a bathroom door. come back, tell him he's jumping to conclusions and i know why the setting is different on that unit, because it was interfering with a very secure app used for commodities trading only in that office and that the setting was made 2 years ago. Also, that he was the one who demanded me implement the offending setting on every unit 3 years ago. he doesn't want to hear it.

He is out on biz trip all week this week. monday morning, GM emails that issue is back, even with setting changed on riverbed (oh, they changed network settings without consulting my manager or me. some real fucking cowboy bullshit) so, the issue is not at all related to what they thought.

have team meeting today, GM shows me what they are looking at. it's one of the freshly deployed Riverbeds with nothing but default config (I explain that since we have no test environment, and i knew that the only possible effect of doing this was that someone might hit a slow web page, complain and be fixed immediately, i wanted to test which in-path rules were totally necessary by going with teh default config, then writing a centralized policy and pushing it to all the new units with the CMC with the minimum of rules and options activated once i had tested everything, but since the "deployment" was on a deadline, I was getting all the unit's out to the branches first.). So the reason the options were set differently was because they were looking at one of the 5 new units so far that had been installed that were rocking the defaults only config. He agreed that my plan was decent.

Issue is now down to a sane level of hysterics and i am now, again, capable of managing the system (at leasst in the eyes of the GM).

what I learned today, control your reactions, take a walk, don't put anyone's head through a 6th floor plateglass window, and things tend to work themselves out over the weekend.
vindication.

I also learned that if they are used to micromanaging, and you just get things done, that they will complain about it if you didn't do it their way, even though the end results are identical. Hopefully this little episode earned me some trust. seems to be that way.


also still in the office, working with the GM to trouble shoot the issue. ISA is passing almost no traffic to the exchange proxy on the internet from 2am til 11 am, then the traffic spikes and is normal until 2 am.
time to break out wire shark or netmon on that segment and see if we can't clear the good name of the network before we set into ISA.

wish they didn't turn the A/C off at 5pm though...
 

ronelson

Ars Legatus Legionis
21,399
Subscriptor
Que friday evening. THE boss sees that if he turns an option off on the riverbed, the emails suddenly go through.

...

GM emails that issue is back, even with setting changed on riverbed (oh, they changed network settings without consulting my manager or me. some real fucking cowboy bullshit) so, the issue is not at all related to what they thought.
It always baffles me that people stop after they "fix" the issue. It is *very* import to re-enable the supposed problem option to see if the problem re-appears. Yet, time and time again, people fail to do this very thing. It would save a lot of time and finger pointing.
 

Fulgan

Ars Tribunus Angusticlavius
7,534
Subscriptor
what I learned today, control your reactions, take a walk, don't put anyone's head through a 6th floor plateglass window, and things tend to work themselves out over the weekend.

You know, I go to the gym every day instead of going to lunch just because of this. Funny thing: many, many people I meet there at IT admins and techs who go there for the same reason (actually, nearly half are cops because the central station is just on the other side and about 80% of all the others are ITs)

The good point is that, after a few years at that regime, your shoulders should be getting (literally) wide enough that people starts thinking twice before trying to scream in your face. When emotions are fully engaged, we usually fall back to very primitive mechanism, it seems.

I also learned that if they are used to micromanaging, and you just get things done, that they will complain about it if you didn't do it their way, even though the end results are identical.

I get that a lot, unfortunately :( Not really from my boss but from several customers.
 

afidel

Ars Legatus Legionis
18,224
Subscriptor
Today I learned that AT&T can provide a very realistic disaster recovery scenario, for free!

We moved our DR site from a nearby location to an AT&T managed colo facility. As part of that move we changed our agreement structure. Well they finally got around to actually deprovisioning the internet DS3 from the site we moved more than a year ago. Last week we got a notification about the intent to terminate and our network engineer noticed that the circuit ID was that of our HQ circuit so he calls in ASAP and has them cancel the disconnect. Well apparently only part of the order actually got stopped and a week later without any notification going out they turned off the local loop. It's been 19 hours and the last word we had was that the reconnect order was in the system and should be processed in 24 hours, WTF? How do you incorrectly term a line after being told not to and then take almost 48 hours to reprovision! We're strongly considering pulling all business from AT&T, about $1M/year worth between managed internet, colo, wireless, and local and long distance. Last night we swung outbound internet over to a cable connection that we had for guest access and rerouted mail through the DR mail relay but remote access for 40% of our employees is down until the circuit and our IP block come back up.
 

Incarnate

Ars Tribunus Angusticlavius
9,004
Subscriptor++
afidel":3ss5gmqg said:
We're strongly considering pulling all business from AT&T, about $1M/year worth between managed internet, colo, wireless, and local and long distance.
Honestly, you could have a similar issue with another provider as well. They ALL suck. You'd just waste your time and effort to switch over just to find that the next telco is just as bureaucratic and stupid. :)
 

afidel

Ars Legatus Legionis
18,224
Subscriptor
ronelson":25ic62tk said:
If you are not getting assistance, rn7284@att.com. I work till "about" 4pm eastern so hurry.
Thanks but at this point it's in the hands of the ilec which is not SBC so there's not much that can be done, they are "expediting" the reprovision and have told AT&T that it's a miracle it will be done in 24 hours since it's normally a 30-45 day process. My comment to that was BS, all the lines are physically in place all you have to do is put the freaking config commands back into the router(s)/switch(s). Our problem with it is we escalated things when the original mistaken order came through and had everyone look through to make sure there were no traces of the disconnect order. If your process is so broken that nobody can figure out what's actually going on I don't want to do business with you. Oh and the fact that the ball didn't get rolling yesterday because there was no business office open to issue the provisioning order on the AT&T side, why don't they have at least *someone* available for such issues, what does a 24x7 operation do when they have a problem?
 

afidel

Ars Legatus Legionis
18,224
Subscriptor
Incarnate":gwixn3tb said:
afidel":gwixn3tb said:
We're strongly considering pulling all business from AT&T, about $1M/year worth between managed internet, colo, wireless, and local and long distance.
Honestly, you could have a similar issue with another provider as well. They ALL suck. You'd just waste your time and effort to switch over just to find that the next telco is just as bureaucratic and stupid. :)

I know they all suck but the stupidity and bureaucracy at AT&T has just reached our limit and the only way these big companies seem to take notice is if it affects their pocketbook, perhaps if enough companies leave them they will eventually take notice and try to fix things.
 

Incarnate

Ars Tribunus Angusticlavius
9,004
Subscriptor++
afidel":2u6gr3pd said:
Incarnate":2u6gr3pd said:
afidel":2u6gr3pd said:
We're strongly considering pulling all business from AT&T, about $1M/year worth between managed internet, colo, wireless, and local and long distance.
Honestly, you could have a similar issue with another provider as well. They ALL suck. You'd just waste your time and effort to switch over just to find that the next telco is just as bureaucratic and stupid. :)

I know they all suck but the stupidity and bureaucracy at AT&T has just reached our limit and the only way these big companies seem to take notice is if it affects their pocketbook, perhaps if enough companies leave them they will eventually take notice and try to fix things.
Naaa...It'll never change. While you're switching from AT&T to another provider, someone else is busy switching to AT&T because their old provider fscked things up. :D

The grass always looks greener, but once you walk up to it you find out the grass is really some left over artificial mini-golf turf from 1972 with green spray paint covering up the bare spots.
 

akro

Ars Scholae Palatinae
1,309
afidel":1q2vwoig said:
Incarnate":1q2vwoig said:
afidel":1q2vwoig said:
We're strongly considering pulling all business from AT&T, about $1M/year worth between managed internet, colo, wireless, and local and long distance.
Honestly, you could have a similar issue with another provider as well. They ALL suck. You'd just waste your time and effort to switch over just to find that the next telco is just as bureaucratic and stupid. :)

I know they all suck but the stupidity and bureaucracy at AT&T has just reached our limit and the only way these big companies seem to take notice is if it affects their pocketbook, perhaps if enough companies leave them they will eventually take notice and try to fix things.

It's like that for any large company I am afraid.... boy I could tell you some stories about my company....
 

Arbelac

Ars Tribunus Angusticlavius
7,684
I had Verizon Business do something similar with a DS3 line on a MANAGED private MPLS.

All installed and working, leave the office... 3 days later, I get a call from that office, hey we can't get anywhere.

What do I find? The DS3 is down.
Call Verizon - "Oh, the project manager didn't check off a box, so it auto deleted itself." (WTF?!?!).
Me: "Okay fine, put the commands back in and bring it back up"
V: "We can't, the IP block has already be re-provisioned. It'll take 2-3 weeks for it to be re-setup." (WTF!?!?! HOW??)
Me: *swears a blue streak and slams the phone down*

At least we had a 2nd connection there, but Jesus H. Christ....
 

Frennzy

Ars Legatus Legionis
85,841
I used to regularly beat up AT&T with a $12m budgetary bat. They were...awful...about so many things.

I was in a meeting with a Sr. VP once, discussing why they couldn't seem to figure out how to give me a consistent billing rate for data. We had no metered usage, but our data costs would fluctuate several hundred thousand dollars per month.

He finally told me (off record, so I won't substantiate this if cornered) that AT&T had something like 60 different provisioning systems, and no one person or team in the company understood how they call interconnected...and the method by which they connected to billing was a complete fucking mystery to just about everyone. This was around the year 2000. They were supposedly in the middle of replacing all of those labyrinthine intertwined hydra with a single System To Rule Them All (can't remember the name...Voyager? something like that).

Sounds like they never completed that project.
 

ronelson

Ars Legatus Legionis
21,399
Subscriptor
He finally told me (off record, so I won't substantiate this if cornered) that AT&T had something like 60 different provisioning systems, and no one person or team in the company understood how they call interconnected...and the method by which they connected to billing was a complete fucking mystery to just about everyone. This was around the year 2000. They were supposedly in the middle of replacing all of those labyrinthine intertwined hydra with a single System To Rule Them All (can't remember the name...Voyager? something like that).
I do not think Voyager was the name, but it did happen. Then they replaced it with something else, then there was the SBC "merger", then Bell South, Cingular, etc. all brought their own billing systems which are actually getting consolidated at a fairly rapid pace. And if you think it is difficult to be a customer dealing with that, try it as an employee! Takes a senior VP to approve post-it notes, I believe.
 

Graeme K

Ars Legatus Legionis
14,776
Subscriptor++
Frennzy":syxhe0zh said:
He finally told me (off record, so I won't substantiate this if cornered) that AT&T had something like 60 different provisioning systems, and no one person or team in the company understood how they call interconnected...and the method by which they connected to billing was a complete fucking mystery to just about everyone. This was around the year 2000. They were supposedly in the middle of replacing all of those labyrinthine intertwined hydra with a single System To Rule Them All (can't remember the name...Voyager? something like that).

Sounds like they never completed that project.

Things are no different at Verizon Business. Three billing/dispatching systems that don't communicate at all. To this day.

As for providers...ours is awesome, and I don't want to tell you who it is, lest you people migrate to them and make them suck somehow. :scared: :altscared:
 

akro

Ars Scholae Palatinae
1,309
Ostracus":b3d7et9z said:
akro":b3d7et9z said:
It's like that for any large company I am afraid.... boy I could tell you some stories about my company....

This forum lives for stories like that. It's like looking at a bad car accident. You know you're not suppose to look...but.

Except I still work for said company/vendor.... ;-)
 
Status
Not open for further replies.