Dr Nno

Ars Praefectus
5,236
Subscriptor++
My conspiracy theory is that the initial batch of SpaceX Engineers, who joined SpaceX while Musk appeared to be a benevolent idealistic future-oriented space entrepreneur, are now pissed off by the real Musk, who is a racist misogynistic DOGE-leading white supremacist douchebag. Those Engineers now don't give as much as they did before.

But more probably, Space is Hard ™.
 

xoa

Ars Legatus Legionis
12,364
Subscriptor++
Very sad news. Unlike the previous few tests where it was hard to see any real rationale not to go for it even if it was expected to fail, here seems like there was some serious potential stage 0 damage and in something (basic manufacturing engineering, QA or consistency) which should be better locked in at this point (vs in-flight stuff needing testing or known weaknesses of Raptor 2 which is already obsolete vs Raptor 3 or the like). That points to issues on the manufacturing or project management sides or both. Of course we know they've been trying to shave weight and figure out their margins, but maybe they're trying to do that simultaneously to too many other dev efforts where the margins are still needed, wrong order of operations which is not how they've operated before. Or they're not keeping their institutional knowledge up, or process consistency, or something. But this doesn't seem like purposeful risk taking, however high, just a pure mistake.

Real shame, points to a real delay in a lot of hoped for space stuff and improved Starlink and so on. They'll be able to keep plugging away at it of course but I hope this is a moment that at least helps prompt a pause and refocus on the fundamentals. If there's been any go-fever this could help damp that for awhile. They've had a very long run of success, but as in any industry like that such runs can't ever be taken for granted. At least it's all happening with nothing commercial cargo (let alone people) at risk.
 

xoa

Ars Legatus Legionis
12,364
Subscriptor++
Musk posted that they think a nitrogen COPV in the cargo bay failed.

Early pictures back from the test site show quite a lot of damage to the GSE but perhaps not worst case.
That's something then. Maybe there's a few improvements they can make as well during a rebuild that weren't worth bothering with while it was working. And an enforced pause might be a good thing anyway.
(and man the front page comments are extra... today)
Yeah I've given up on the front page. At one point space threads remained a final old redoubt from the generic-internet toxic waste dump Ars front page threads have mostly become otherwise, but over the last few months they too have fallen. Not that even this thread is entirely free of shitposting either granted.
 

xoa

Ars Legatus Legionis
12,364
Subscriptor++
SpaceX posted a generic update basically acknowledging what happened and repeating already shared info like the suspicion it originated as an issue with a COPV. Which was the cause of the AMOS-6 Falcon 9 explosion too iirc? If that does turn out to be the problem that's seems kinda least-bad in terms of "always trick known tech" vs "path finding new", though conversely seems like it'd be extra infuriating as they will have lost a lot of test potential for something that has long been known to be a source of boomyboom.

Emphasis on "long"! Reminds me of a space discussion from many years ago now where it was mentioned as an issue with Saturn even. As well as interesting history in its own right, reading back the echos to this one are there:
A Saturn V third stage, S-IVB-503, exploded shortly before it was scheduled to be ignited in a January 20 test at SACTO. The explosion completely destroyed the stage at test stand Beta III. Post-accident investigation revealed that one of the eight ambient temperature helium storage spheres located on the engine thrust structure exploded because of weld weakness resulting from use of the wrong weld material.
Titanium instead of composite, but containing extremely high pressures in a system with extreme temperature shifts seems to have always been hard. Though I imagine generating in-situ would also have its own set of challenges (ie, not producing pure inert gas, whatever chemical solids/liquids used having their own storage/plumbing challenges, how it all interacts with cryogenic temps etc).
fig327.jpg
 

xoa

Ars Legatus Legionis
12,364
Subscriptor++
Test stand looking pretty crispy:
1.png


So it may be this represents zero new issue discovered with Starship. On the other hand it'd also mean absolutely zero new learning with Starship, they blew up a ship and test stand for nothing beyond "don't get lazy about your COPVs". And I mean that in a "sheer stupid negligence" sense too, since I think their COPVs for Starship are not made by SpaceX but by Luxfer? Who know their stuff. So while it could be a supplier thing, it could be simple bad handling on SpaceX's side too. I have mixed feelings about linking an ex-employee (Morgan Khan) I know nothing specific about beyond that they did work there until April or so, and who clearly is angry about safety culture there. That may be totally valid, or it may be overblown to one degree or another, or it may be valid yet not actually related to root cause in this particular instance. Any time there is a major aerospace disaster the reminder of "be cautious about jumping to conclusions before a full investigation" is always repeated and for good reason.

Nevertheless given this incident, it does seem relevant to acknowledge he had a lot of safety complaints earlier this year, well before this incident, and about a month ago on May 13 he wrote:
All of this was violated at some point or another by locals to be clear that have little to no Aerospace background or work history.

A lot of "tent era" workers that say this is how it's always been done as they laugh and then slam COPV bottles into the newly retrofitted brackets in payload....
I was assigned work on Issue Ticket operations to fix and identify the extent of damage to the COPV bottles with the only other certified COPV inspector on site.

I brought this up and then was not allowed to touch or be inside payload for 2 vehicles lmfao like wtf are they smoking?
We had to stop the show and wait for new undamaged COPV bottles to arrive because of the "Tent Era" negligence and tomfoolery taking place that is unacceptable behavior.
If this all came down to just damaging composites by pure stupid that'd be shameful. Taking risks and running hardware rich in the service of exploring new unknown-unknowns and known-unknowns is very good. But being negligent about exist knowns is very bad. During the F9 era at least SpaceX was "very aggressive", but never with customer payloads, and their "aggressive" was purposeful, planned aggressive. They were doing specific things they knew might result in explosions, but that they aimed to get specific new data out of in service of a longer term goal. "Slamming COPV bottles into brackets" and then having things go boom would not be that.
 

Ecmaster76

Ars Legatus Legionis
16,977
Subscriptor
Nevertheless given this incident, it does seem relevant to acknowledge he had a lot of safety complaints earlier this year, well before this incident, and about a month ago on May 13 he wrote:


If you scroll down a bit to replies he made yesterday, he describes flanges not even being torqued and the response he got when he brought it up.

It certainly all could be true but if things really were that sloppy, you'd expect nothing to ever go right. Thats not what we've seen.
 

xoa

Ars Legatus Legionis
12,364
Subscriptor++
If you scroll down a bit to replies he made yesterday, he describes flanges not even being torqued and the response he got when he brought it up.

It certainly all could be true but if things really were that sloppy, you'd expect nothing to ever go right. Thats not what we've seen.
Most successful organizations though on the path down have a not insignificant amount of safety "budget", which is historically one of the many factors that make safety/security/reliability tricky as a matter of human management. There rarely is a clear, rapid 1:1 relationship between corner cutting/sloppiness/lack of formalization/etc and ultimate failure. Rather, each bit eats into the overall budget invisibly until bankruptcy happens "all of a sudden" right? We know not everything goes downhill equally fast, nor is every aspect equally important. The designs themselves matter too I think. Reports at least have been that, unsurprisingly, Starship v1 was engineered with significantly higher margins while they were accomplishing the initial envelope exploration. Part of the point of the v2 campaign is supposed to be (just like with Falcon v1.0 to FT Block 5) starting the process of figuring out where the margins were certainly too high and could be reduced for more useful payload mass.

But that's the sort of thing that, in an unhealthy org, would then start to expose already existing problems, or exacerbate existing ones. Incorrect torque but with a bunch of extra flanges might still be ok, but then they start trying to cut that and it's a problem. Mishandling isn't necessarily deterministic, previous COPV might have simply lucked out in not being just right to fail then, or with there could be normalization of deviance and things snowballed over time. Or to the point of formalization/checklists, it could be that there were problems but that there was previously a critical mass of employees who cared. Management might not have taken notice of his/others reports, but perhaps others did and were silently following along and correcting for bad procedures. People on the ground who cared knew who was sloppy on torque, then would redo the work. But that's thankless and by definition kind of invisible/easy to take for granted work. Maybe they burned out without support. Maybe the wrong thing finally got missed at the wrong time. Maybe they were flat out fired, because they were doing important work that wouldn't show up in the formal system so they got dinged for "low productivity".

I mean, this is totally pure speculation and not a lot of data points for sure! But taking NASA as a parallel example, they had major cultural issues developing for quite awhile and repeated close calls before the Challenger disaster. Same with Columbia, design specs for the Space Shuttle was that the external tank would not release any debris to damage the TPS. Yet literally STS-1 (which I believe was in fact Columbia!(?)) and it was damaged by a foam strike. And then kept happening. And they tried some stuff, then eventually just stuck it in the "accepted flight risk" bucket and ignored it from then until disaster. It took well over 100 flights and two decades before the dice finally came up snake eyes but come up they did.

Again not trying to totally overstate this, but it feels more plausible now. Unlike I many I've been very sympathetic with the v2 issues after launch this year. But this time feels sloppy. That's not a place where things should have gone wrong in that manner.
 

xoa

Ars Legatus Legionis
12,364
Subscriptor++
Front page article up on successful test firing this time and upcoming return to flight. For the test failure, it was indeed a COPV:
SpaceX said the explosion on the test stand in June was likely caused by damage to a high-pressure nitrogen storage tank inside Starship's payload bay section. This tank, called a composite overwrapped pressure vessel, or COPV, violently ruptured and led to the ship's fiery demise. SpaceX said COPVs on upcoming flights will operate at lower pressures, and managers ordered additional inspections on COPVs to look for damage, more proof testing, more stringent acceptance criteria, and a hardware change to address the problem.
So yeah while they throw in a bit of "lower pressures" and "hardware change" in there that mostly reads as "we got sloppy and fucked up something we really shouldn't have". I suppose even that is a lesson with some value in terms of hubris etc, and better on a test stand with obsolete hardware than somewhere more serious, but still a bitter and stupid pill.

Also seems of note:
SpaceX has just two Starship Version 2 vehicles in its inventory before moving on to the taller Version 3 configuration, which will also debut improved Raptor engines.
Raptor 3 is supposed to be a pretty big deal, being the real fundamental fix to a bunch of issues/leaks they've had and a further massive refinement over v2 (already a refinement over v3), so I'd kinda wondered if they'd try retrofitting some in asap, at least on the booster where they don't need to wait on any new tile data or other orbital stuff. Apparently not though? Sounds like they'd prefer to use their existing stock to do some aggressive & destructive structural testing of it instead which presumably doesn't benefit from R3, so might as well just burn whatever they've got left.

Well, here's hoping they can get back into shape (in multiple respects...).
 

Skoop

Ars Legatus Legionis
33,218
Moderator
Sounds like they'd prefer to use their existing stock to do some aggressive & destructive structural testing of it instead which presumably doesn't benefit from R3, so might as well just burn whatever they've got left.
R3s don't fit on that booster design anyway. They're not catching them; they don't want them back. (They might salvage the ass ends though, like they've already done.)

What they really, really want is data on the heat tiles. They can't proceed with v3 ships until they know how they have to shield them and how the flaps work under stress.
 
  • Like
Reactions: BrangdonJ

xoa

Ars Legatus Legionis
12,364
Subscriptor++
R3s don't fit on that booster design anyway.
Ah, haven't been following it closely enough recently to realize that.
They're not catching them; they don't want them back.
Yep.
What they really, really want is data on the heat tiles. They can't proceed with v3 ships until they know how they have to shield them and how the flaps work under stress.
I guess I just wasn't really clear on how closely synced the ship and booster aspects are. Could have seen it going either way honestly, and eventually I assume the booster is going to end up the more "stable" of the two parts so to speak, they'll no doubt want to keep optimizing to the extent feasible and make it bigger if needed but it'll never have fulfill the diversity of roles or length of operations envisioned for the ship. But would make sense that right now they're developed fairly tightly in step.
 

xoa

Ars Legatus Legionis
12,364
Subscriptor++
Anyone have a sense of what kind of weather envelope they're looking for specifically at this stage in launch experimentation, and any favorite sources for predictions? As far as ground conditions go, I have Boca Chica saved in Windy and while of course sometimes it's obvious that conditions will be zero issues (low single digit wind, clear as a bell), and conversely if it's mass thunderstorms that's obvious enough as no-go too. But I don't really have a feel for when it's getting marginal. My recollection is that they've launched Falcon 9 in up to 20 knot winds. And upper level sheer can matter more, but while the models I see will show conditions up to FL450 (and the airgram looks promising tonight, I think?) I'm even less clear on when things are getting scrub-ish.

Anyway, just semi-idle curiosity. Since I'm only streaming over the internet right now it doesn't make a major difference either way, it happens or not. But I'd like to go see a launch in person at some point and since we have all these airgrams and data right there might be handy to have some rough feel for weather impact.
 

Skoop

Ars Legatus Legionis
33,218
Moderator
why does it matter what time they go
Windows exist because there are marine and air exclusion zones; flight and sail paths are disrupted and these have to be declared and minimized.

As for the time, they want eyes on both the launch vehicle ascent and landing, AND they have a ship coming down on the other side of the planet. They want day-lit video of that landing.
 

xoa

Ars Legatus Legionis
12,364
Subscriptor++
Windows exist because there are marine and air exclusion zones; flight and sail paths are disrupted and these have to be declared and minimized.
Yep, not the busiest air spot in the world but definitely plenty of air traffic in the area and of course down the full potential debris field range too. And then in terms of shipping the Totinos Pizza Rolls Presents Gulf of America Powered by The Home Depot is absolutely full of shipping.
As for the time, they want eyes on both the launch vehicle ascent and landing, AND they have a ship coming down on the other side of the planet.
As a side note "other side of the planet" is quite literal, middle-ish of Indian Ocean is I think (around -26.379284541981026, 83.27145011350808) is exactly opposite Boca Chica, so a touch over 20k km great circle. And southern Indian Ocean really is nowheresville in terms of both air and sea traffic which of course is precisely why it's chosen.
 

PsionEdge

Ars Legatus Legionis
21,398
Subscriptor
I appreciate the heads up, I'll have to see where it is in the video.
If this was referring to the Starlink factory peek, that was at about 37 minutes into the previously linked video.

Scrubbing through it I heard Elon say the Starship factory is alien-level technology and man, the best thing for SpaceX's credibility is to leave him out of everything...