Amazon stuck with months of repairs after drone strikes on data centers

Hard to believe I'd ever rally for Iran... but I want that bratty orange bully humiliated! Yes, that means siding against my own country - but if that will finally cut the Republican fascists down - it's well worth it to me. Sigh...

Your country very, very much deserves being sided against right now. Time to drop the nationalism and choose between right and wrong. Iran is by far the lesser evil in this shit show.
 
Upvote
55 (82 / -27)
6f6.jpg
 
Upvote
20 (34 / -14)
Hard to believe I'd ever rally for Iran... but I want that bratty orange bully humiliated! Yes, that means siding against my own country - but if that will finally cut the Republican fascists down - it's well worth it to me. Sigh...
Two things can be true and bad.

1) Iran’s autocratic government is a repressive theocracy that sucks shit.

2) Invading sovereign nations you are actively negotiating with, for no genuine political objective or end goal, is how you kick off global instability and who knows, maybe a fucking world war?!

Bonus laughs: all the allied repressive theocracies that are goading the US into destroying Iran for being a regressive theocracy!
 
Upvote
194 (197 / -3)

msawzall

Ars Tribunus Angusticlavius
7,395
Hard to believe I'd ever rally for Iran... but I want that bratty orange bully humiliated! Yes, that means siding against my own country - but if that will finally cut the Republican fascists down - it's well worth it to me. Sigh...
You're not siding against your country. You're siding against a fascist authoritarian who happens to run your country right now. Big difference. Your conscience should be clear.

They try this verbiage all the time... "You're against the troops." "You're not a true patriot." It's all BS to get you to just go along with the fascism.
 
Upvote
139 (144 / -5)

Bongle

Ars Praefectus
4,477
Subscriptor++
Isn't this rather a bad design? If your sprinklers stop the fire but destroy all the computers, what's the point?
Yeah I always thought datacenters used halon or other gases to get rid of the oxygen without dousing everything inside. I guess if it was a big enough fire or if there was a hole in the building (likely, given a drone strike), that wouldn't work.
 
Upvote
26 (27 / -1)
What happens if Iran takes a lesson from the military doctrine being used against them and sends a fresh wave of drones to "double tap" the data centers while repairs are underway? It's hard to imagine your average rack-n-stack guy being willing, at any price, to install gear under those conditions. It could be a whole lot longer than several months before these facilities are restored.
 
Upvote
76 (78 / -2)

Cthel

Ars Praefectus
10,027
Subscriptor
Yeah I always thought datacenters used halon or other gases to get rid of the oxygen without dousing everything inside. I guess if it was a big enough fire or if there was a hole in the building (likely, given a drone strike), that wouldn't work.
IIRC, the gas-based suppression systems have issues with the extremely loud noise of their activation causing head crashes in HDDs
 
Upvote
17 (17 / 0)

Andrewcw

Ars Legatus Legionis
19,023
Subscriptor
Isn't this rather a bad design? If your sprinklers stop the fire but destroy all the computers, what's the point?
Gone are the days of Halon, It's the structure that's been worth more. I've only seen a Halon system activate once. And that was when they used to glass window a room in a university to make the room look cool.

While this datacenter is down partially. The whole point of multiple regions was to have redundancy in other regions. Which of course will cost that customer money. So while clouds can be cheaper in some cases. You still have to pay for it.

Amazon isn't going to be crying about this too long they already have all the spares and ram because they're already been buying it to keep up with AI buying to meet peer levels of equipment hording.
 
Upvote
20 (20 / 0)
The fact that AWS expects the full restoration of cloud services to potentially take half a year speaks to the damage inflicted by the Iranian drone strikes.
Counterpoint: I think the time period is more linked to the lack of safe transport of replacement parts and service personnel than the actual level of damage caused.
 
Upvote
23 (23 / 0)

Andrewcw

Ars Legatus Legionis
19,023
Subscriptor
IIRC, the gas-based suppression systems have issues with the extremely loud noise of their activation causing head crashes in HDDs
That's the least issue. It's traditionally datacenters were in rich countries with lawyers and when your fire suppression system outright can kill the worker. Issues can arise. Granted fire and water with electricity also isn't safe. But it's way slower than taking all the oxygen out of the air.
 
Upvote
34 (34 / 0)
Isn't this rather a bad design? If your sprinklers stop the fire but destroy all the computers, what's the point?
Is halon no longer a thing?
I read the rest of the comments, and I see it is no longer a thing in data centers. I'm old and did mainframes for years when I was in the Airforce 40 years ago. We had a halon dump happen due to a leaking roof. I wasn't working when it happened, but there was no problem getting out in time.
 
Last edited:
Upvote
4 (4 / 0)

sfbiker

Ars Praetorian
598
Subscriptor
Isn't this rather a bad design? If your sprinklers stop the fire but destroy all the computers, what's the point?
Sometimes water sprinklers are required by local fire codes -- the last corporate server room I managed required water sprinklers. They had dry pipes and it was a "preaction" system where it took both a sprinkler head activation and smoke and/or secondary heat sensor to trigger the sprinkler heads.

It was not a deluge system where all heads activate at once, only the heads that were hot enough to trigger would release water, so if you have a fire in one corner of your datacenter, only those heads should trigger reducing the damage (and in theory, once they start releasing water it should cool the fire enough to prevent triggering more heads) - you may lose a few racks of equipment, but the rest of the servers in the room should survive. Larger datacenters also typically have separate fire rated rooms so a sprinkler going off in one room shouldn't affect the servers in another.

Though in a drone attack, not sure how widespread the damage is or how easily it could be contained in one area.

We only had one close call, a vendor hit a sprinkler head with a ladder, it released the air in the system and set off the alarm, but the preaction system didn't release the water. Stlll was a few thousand dollar fix to get someone out to replace the head, recharge the nitrogen and inspect the system, but a lot cheaper than if it had released the water.
 
Upvote
26 (26 / 0)

Chinsukolo

Ars Scholae Palatinae
1,011
Subscriptor++
The fact that AWS expects the full restoration of cloud services to potentially take half a year speaks to the damage inflicted by the Iranian drone strikes.



I don't know that i believe this - I see Ars cited Business Insider for the quote. AWS knows how to build data centers, they have back stocks available - I suspect the delay has WAY less to do with the severity of the damage and way more to do with RAM and other shortages and price increases due to AI build outs.
This was an unforseen/forcasted event so they couldn't lock in prices in advance, leaving them to get all replacements stocks internally and depleting, or on the market at market prices and with market scarcity.

If they can turn out a new data center (pre-AI boom) in the time they forecasting to repair this, then even gutting the damaged region and rebuilding just that should be faster then the forecast.
 
Upvote
-3 (1 / -4)

afidel

Ars Legatus Legionis
18,201
Subscriptor
You're not siding against your country. You're siding against a fascist authoritarian who happens to run your country right now. Big difference. Your conscience should be clear.

They try this verbiage all the time... "You're against the troops." "You're not a true patriot." It's all BS to get you to just go along with the fascism.
I'm for the troops, I want them to not face harm for the greed of oil company execs pocket. I'm for troops, I want them to be able to rotate home to their loved ones, not spend the longest tour of duty since ww2 because our president wants to distract from his pedophile history. I'm for troops, I want them to be fed proper meals, not sub prison rations because we went into a conflict with zero preparation because the tail was wagging the dog.
 
Upvote
61 (64 / -3)

Bill T.

Ars Centurion
327
Subscriptor
Upvote
19 (19 / 0)
No, no. The war is over. Didn't you hear? Pentagon Pete said so, just before the 60 day War Powers Act deadline. The fact that troops have not left the region, the Strait of Hormuz is still blocked, Iranian ports are still blockaded, and drone attacks continue apace is not, not, not evidence of a continuing war. Nope. Nothing to see here. Move along people.
 
Upvote
49 (49 / 0)

clewis

Ars Tribunus Militum
1,794
Subscriptor++
Isn't this rather a bad design? If your sprinklers stop the fire but destroy all the computers, what's the point?

Years ago, a fire supression test destroyed my servers storage. It was a gas suppression system, not water. The gas was so cold, it caused the HDDs to freeze. About half of my servers lost one mirror, and a few lost both boot mirrors. Luckily the more critical fiber channel drives were better shielded, and I didn't lose any database arrays.

For extra fun, all of the servers were required to sync with NTP as part of the boot sequence. Including the NTP servers. It took much longer than it should have to figure out why nothing was finishing the boot, even with the management consoles.

[Edit to clarify] I didn't lose the servers, just storage. The servers with working HDDs all booted.
 
Last edited:
Upvote
12 (12 / 0)
Might I suggest anyone looking to relax this weekend take this opportunity to watch a magnificent video about someone making a rowboat out of Jeff Bezos' head?


View: https://www.youtube.com/watch?v=VGhcSupkNs8


Then we can feel like he has some kind of value to offer to world.

I love this dude's Mel Gibson Malibu arrest model. Guy is amazing at so many artforms - sculpting, 3D modeling, painting... And that dreamy monotone voice with touches of hilarious wit and sarcasm sprinkled in
 
Upvote
7 (7 / 0)

MechR

Ars Praefectus
3,236
Subscriptor
Two things can be true and bad.

1) Iran’s autocratic government is a repressive theocracy that sucks shit.

2) Invading sovereign nations you are actively negotiating with, for no genuine political objective or end goal, is how you kick off global instability and who knows, maybe a fucking world war?!

Bonus laughs: all the allied repressive theocracies that are goading the US into destroying Iran for being a regressive theocracy!
Yeah. After Iran's protest massacres, I could've theoretically gotten behind a principled and well-planned attempt at regime change, but it was never going to be principled or well-planned with Trump and Hegseth in charge. (Meanwhile, a Democrat wouldn't have gone in, and Republicans would've screamed bloody murder forever if they tried.)

Even after the shooting started, I would've accepted a miracle outcome where Iran's regime fell, gave way to a secular liberal democratic revolution, and made Trump undeservedly look like a genius. But that was never a realistic wish either, and blowing up the girls' school nullified all potential goodwill in one shot.

So at this point, I'm also on the side of "Trump and his party need to suffer lasting political damage from this crap, or they'll do it again." Venezuela made them cocky.
 
Upvote
41 (44 / -3)

Zoc

Ars Scholae Palatinae
1,123
Subscriptor
Better than having the entire structure AND all the computers burned to the ground
I see from other comments that these fire systems aren't generally set up to soak and ruin all the equipment, only the part that's actually on fire, so it seems like there is a middle ground between "burn the whole building" and "destroy all the computers with water."
 
Last edited:
Upvote
13 (13 / 0)