what did you learn today?

Status
Not open for further replies.
Originally posted by Whittey:
ISA 2k6 on W2k3. No PC's (using rpc over https or owa) had an issue, and only the older phones (well, and a couple newer ones). That's what was throwing us off. Verisign said it was pretty common, as the intermediate they used for our cert was authorized starting March of this year and they just had us reapply for a new one. Simple fix, but really annoying.

Yeah it can be annoying. Our EAS devices suddenly didn't work when we went to ISA 2006, but all testing was fine. Turns out the ISA admin hadn't installed the new intermediate on the 2006 boxes. Doh. That fixed the EAS devices once I saw it was missing. EAS is kind of weird, it'll basically allow ISA to say "yeah this SSL is ok" if an intermediat is missing on the phone itself.
 
Originally posted by scorp508:
Originally posted by Rick25:
The May 09 Root Certificate update from MS included all the new intermediate issueing servers so you're PC would have gotten caught during the regular update cycle.

Really? Usually the root updates just include the roots and no intermediates.
 

afidel

Ars Legatus Legionis
18,224
Subscriptor
Originally posted by tinyMan:
I think a lesson that has been learned in the Ukraine and even perhaps at Microsoft management this week is that you should always have a full backup of your data.

This story is so absolutely and completely full of *FAIL*. I hope that more information about the incident is released. There is a lot to learn about best practices, and the costs and risks of not following them from this incident.

What I learned is that most people are whiny idiots, T-Mobile had an optional backup application available for less than the cost of a month of service. For all the people whining about how horrible it is that the cloud failed to backup their data all I have to say is trusting ANY company to be the sole backup for your "critical" data is just stupid. I had a LOT more sympathy when I though the device was a completely closed platform with no provision for user self service, now not so much. There was one guy talking about how he was in the entertainment industry and had lost thousands of contacts worth tens of thousands of dollars to him yet he didn't buy a $45 backup application? Really? My dad who still hunts and pecks knows enough to demand multiple backups at multiple sites because it's risk prevention 101.
 
Originally posted by afidel:
What I learned is that most people are whiny idiots, T-Mobile had an optional backup application available for less than the cost of a month of service.

Um, where would that backup be located? On the server-side? Where the data was already until they lost it?

In fact, are you sure that TMo sells (sold?) a backup app for the Sidekick? That was the whole point of the device, it was all server-side and the phone was mostly a terminal, from a data point of view.

I have the BackupNow app for my Verizon phone, which is great. My phone holds everything in NVRAM, and it gets backed up once a day to VZW's servers. My phone explodes, still got my contacts. But for the Sidekick, it was never permanent on the phone.
 

afidel

Ars Legatus Legionis
18,224
Subscriptor
Originally posted by finni:
Originally posted by afidel:
What I learned is that most people are whiny idiots, T-Mobile had an optional backup application available for less than the cost of a month of service.

Um, where would that backup be located? On the server-side? Where the data was already until they lost it?

In fact, are you sure that TMo sells (sold?) a backup app for the Sidekick? That was the whole point of the device, it was all server-side and the phone was mostly a terminal, from a data point of view.

I have the BackupNow app for my Verizon phone, which is great. My phone holds everything in NVRAM, and it gets backed up once a day to VZW's servers. My phone explodes, still got my contacts. But for the Sidekick, it was never permanent on the phone.

It was/is a Windows app that backs the device up locally. My personal method for backing up my phone is to store to nvram+SIM, paper copy, gmail contacts, Thunderbird contacts which also gets backed up to Mozy, and I don't really consider them important, just annoying to recreate. If I thought my contacts were worth tens of thousands of dollars you bet your bottom dollar I would have that and more. People whine about how valuable their data is and then spend zero time or money on backups, that just tells me the real value the hold for that data.

Bah, and now I find out the device could save contacts to SIM, wow that really was valuable data....

Oh and the app was Intellisync, not exactly some no-name app.
 

afidel

Ars Legatus Legionis
18,224
Subscriptor
That shutdown from an RDP session can be dangerous, even two months later.

So apparently my boss issued a shutdown from an RDP session back on 8/20 without the -f switch. The shutdown halted because the admin user was logged into console and had processes running. Well today another admin logged into that server and noticed the errors and asked me to investigate. I check the error messages and acknowledge them, and once I acknowledge the last one I see "Windows is shutting down", wtf? So I run to the server room and sure enough the power button is orange, ruh roh Shaggy. I turn it back on and let the helpdesk know it should be back up soon.

Ten minutes go by and it finally boots up (normal reboot = 2 minutes). Things look good, two minutes later my jr admin tells me he's not getting drive mappings and not able to login to Citrix. I check and sure enough the box is out of kernel memory (damn x86 OS). I shutdown everything I can but it's not freeing up memory fast enough. So I reboot again after disabling everything not critical to that box being a file server (WMI, HP tools, etc). It comes back up after another long reboot and still no dice, three minutes of normal operation and then out of kernel memory.

So I shut it down unmap half the LUN's and map them to a dev x64 server and update DFS, leaving only the LUNS that are called by server name instead of through DFS referrals. Finally 2.2 hours later I have stability back. I update backup policies so that I don't have to run fulls in the middle of the week and grab a cup of coffee to substitute for the adrenalin that's starting to wear off. Then I come here to share my story of woe.
 

Metzen

Ars Scholae Palatinae
1,042
Originally posted by akro:
Paged Pool and NonPaged pool memory are B!TCH to deal with!!!!!

Much to my surprise and Microsoft's when we tried to restore or copy a 220GB Information Store on NT4.

Yet another reason PAE is bad. I had a server with 64GB of RAM partitioned into a 32GB RAM drive and 32GB for the OS. The OS was 32bit 2003. The RAM drive stored lots of files (twas a file server) but after an day or two, the machine would "Run out of resources". A reboot would fix the problem but that is unacceptable. Turns out that the 32bit server OS's only allow something like 200MB of Non/Paged Pool memory. Using the RAM as a filesystem consumed all that space causing the message. The simple solution? Upgrade to 64bit. Now you can have 2GB of Non/Paged Pool space (IIRC).

We used 32bit at the time as one of our custom apps wasn't tested on 64bits. When we ran into this problem, we tried the program on the 64bit OS and it worked like a charm. We then set in motion a plan to ensure this app would be certified on both OS's.
 

akro

Ars Scholae Palatinae
1,309
It's a long story about an organization that didn't believe in mailbox quotas.... did you know on Exch 5.5 you could actually roll over a mailbox to say all zeros I think it was when the mailbox hit 4 GB. Needless to PSS said I was either the bravest or the dumbest sysadmin when I made them confirm the paged poll limit was getting hit (we were the largest production IS they had seen) In fact we had 8 Servers all pushing 220+ GB. The funny part is we were a huge NT on Alpha shop before this and it wasn't an issue then. When MS dumped AXP support we moved to x86 and wammo now we have a problem. Trust me I had been begging since I got there to implement a quota and since it was the government and everyone was afraid to go to Jail over records management it fell of deaf ears...

Finally security forced us to implement quotas because it took too long scrub the information store of a bad email message....

Originally posted by finni:
Originally posted by akro:
Paged Pool and NonPaged pool memory are B!TCH to deal with!!!!!

Much to my surprise and Microsoft's when we tried to restore or copy a 220GB Information Store on NT4.

I'm sure you had a very good reason to do this.... <cringe>
 

jshiplett

Ars Tribunus Angusticlavius
6,710
Subscriptor
Originally posted by Widger:
The VCP 410 test is fucking hard.


How bad compared to the 310 test? How did you do? I'm planning on taking it in a couple of weeks.


Failed by a couple of questions. -- :( --

I'd say it was much, much harder than the VCP 310 test. I passed the VCP 310 test on my first try and that was two years ago (meaning two fewer years of experience).
 

afidel

Ars Legatus Legionis
18,224
Subscriptor
Originally posted by the maddman:
Today I learned our 25 year old building needs a new roof no one wants to pay for. So far it's "only" a trickle of water running into the UPS transfer switch that's in the underground basement.

Call your local fire marshal, he won't take any nonsense from anyone when it comes to such things.
 
Originally posted by akro:
It's a long story about an organization that didn't believe in mailbox quotas.... did you know on Exch 5.5 you could actually roll over a mailbox to say all zeros I think it was when the mailbox hit 4 GB. Needless to PSS said I was either the bravest or the dumbest sysadmin when I made them confirm the paged poll limit was getting hit (we were the largest production IS they had seen) In fact we had 8 Servers all pushing 220+ GB. The funny part is we were a huge NT on Alpha shop before this and it wasn't an issue then. When MS dumped AXP support we moved to x86 and wammo now we have a problem. Trust me I had been begging since I got there to implement a quota and since it was the government and everyone was afraid to go to Jail over records management it fell of deaf ears...

Finally security forced us to implement quotas because it took too long scrub the information store of a bad email message....

Originally posted by finni:
Originally posted by akro:
Paged Pool and NonPaged pool memory are B!TCH to deal with!!!!!

Much to my surprise and Microsoft's when we tried to restore or copy a 220GB Information Store on NT4.

I'm sure you had a very good reason to do this.... <cringe>
Woah! Another NT on Alpha veteran! Exchange on Alpha as well? I was that soldier back around 2000-2002/3...
 

ronelson

Ars Legatus Legionis
21,399
Subscriptor
Everybody wants to be Dr. House--swoop in on a troubled project and miracle it into health. Guess what? Repeated radical changes with no follow-through don't help anyone. They just push the plan further to the right.
Hopefully my new mgmt is aware of this. Today we have a call to discuss our "new design" - you know, the design that was never mentioned further in the calendar invite? I have no idea what this call will be about, but it promises to revolutionize everything!

Side note: New boss has been the new boss for less than 8 full days. His old job was routers; the new one is firewalls, proxy, and other security devices. I hope he does not try to treat servers like routers.
 

Fulgan

Ars Tribunus Angusticlavius
7,534
Subscriptor
Side note: New boss has been the new boss for less than 8 full days. His old job was routers; the new one is firewalls, proxy, and other security devices. I hope he does not try to treat servers like routers.

Sounds like you're going to be adding a lot to this thread in the next few month. Good luck to you but I hope you don't mind if I grab the popcorn.
 
Originally posted by afidel:
Originally posted by the maddman:
Today I learned our 25 year old building needs a new roof no one wants to pay for. So far it's "only" a trickle of water running into the UPS transfer switch that's in the underground basement.

Call your local fire marshal, he won't take any nonsense from anyone when it comes to such things.

It's ok, a rolled up towel on the wall fixed it! Sadly I work in Government, so there are politics involved in everything.
 

ronelson

Ars Legatus Legionis
21,399
Subscriptor
Sounds like you're going to be adding a lot to this thread in the next few month. Good luck to you but I hope you don't mind if I grab the popcorn.
Could be. I started on something at 8a today - deployment plan for a new server that is desperately needed - but could not get back to it after 8:30a. All of my afternoon was taken up in a single task from this meeting.

Oh yeah, boss-man could not make the meeting today. We had the same meeting last Friday and he could not make that one either. Whee! I am okay with pie-in-the-sky ideas, I just want a little direction, that is all...
 

Rick25

Ars Tribunus Militum
2,511
Subscriptor
If your users suddenly get Access Denied to a network drive but you can get to it with \\server\share then you've got a problem

autorun.inf making it seem like a drive which then launched a malware that's on a folder in the drive, but somehow the folder spoofs the users recycle bin if you browse into it through the GUI

Thanksfully autoplay was disabled on 99% of our machines...now 100% thanks to GP.
 

Whittey

Ars Tribunus Militum
2,006
Originally posted by Rick25:
If your users suddenly get Access Denied to a network drive but you can get to it with \\server\share then you've got a problem

autorun.inf making it seem like a drive which then launched a malware that's on a folder in the drive, but somehow the folder spoofs the users recycle bin if you browse into it through the GUI

Thanksfully autoplay was disabled on 99% of our machines...now 100% thanks to GP.
And the desktop guys here still think it's a stupid policy to disable remote creation of inf's within McAfee...


-=Whittey=-
 

gradster

Ars Scholae Palatinae
942
In his first week as the Director of our security group, he left his computer unlocked and his desktop got Hasselhoffed. Today, our entire group was emailed, "What do you guys want for lunch? It's on me!" This just gets better and better -- :) --


Wow, this forum is a bit darker than I thought! Your poor poor Security Group Director!

http://www.youtube.com/watch?v...=player_profilepage#

Sorry...when I saw the term hasselhoffed I had to Google it.
 

Paulie

Ars Tribunus Militum
1,521
Subscriptor++
Originally posted by mkg:
When eTrust decides that your core, tier-0 application is now a threat and takes measures, it's time you stop trusting eTrust.


Yeah, details please. Couple months ago with the bad engine update took me hours to clean up. This something I can prevent before I go to bed or does eTrust just hate your app?

Thanks in advance!!! -- :D -- -- :D -- -- :D --
 
Originally posted by gradster:
In his first week as the Director of our security group, he left his computer unlocked and his desktop got Hasselhoffed. Today, our entire group was emailed, "What do you guys want for lunch? It's on me!" This just gets better and better -- :) --


Wow, this forum is a bit darker than I thought! Your poor poor Security Group Director!


Our is even worse. Due to a security glitch on our Exchange server, if you don't lock your desktop it's you that's sending a lunch invite by name of our CEO. Yep -- :) --
 

ronelson

Ars Legatus Legionis
21,399
Subscriptor
Sorry...when I saw the term hasselhoffed I had to Google it.
Ha, close but not quite. It is more like finding a 1024x768 version of puppies and the hoff and making that the background, then opening all the windows so he does not find out until a few hours later.

Not really a background image, but this would be awesome to install at the office - http://gallery.krunk4ever.com/.../hasselhoff_soap.jpg - in fact, I hear our new Director is a big fan of the Hoff, maybe he can buy it! -- :) --
 
Learnt over the past few days that P2V isn't always straightforward. VM'd our ancient BES server, and told VSphere to do the tool install one time. Booted the (successful) image, got a GUI, logged in, watched the tools install and reboot.

INACCESSIBLE_BOOT_DEVICE.

Fiddled with fixvmscsi. No dice.

Re-imaged, without the tools.

Booted cleanly, logged in.

Added the tools. Rebooted. Golden.
 
I learned when EMC tells you to update the Firmware and Boot code on Emulex cards, upgrade any host with PowerPath under 5.1, and make sure Storport.sys is new that....


1. Storport.sys upgrades will go fine.

2. When you go to the Emulex site and download the 'EMC' approved latest FW/Boot Code that it can be 'too' new for your SAN infrastucture if you're SAN infrastructure itself isn't up to date and cause your server to not be able to boot from its boot LUN. The fix was too down-rev the HBA to something slightly older. Ugh.

3. That for some reason PowerPath 5.3 will nearly guarantee one of two different BSODs resulting in an endless reboot loop if you have the auto-restart turned on. And yes we uninstalled 4.6.x, rebooted, installed 5.3, rebooted. This was on many 2003 32-bit SP2 Enterprise servers. Even safe mode was f'd. The fix was to try Last Known Good config and if that didn't work to boot to something like knoppix, role back the system state manually, and give LKG another try.
 
Status
Not open for further replies.