With Starliner stuck in space, has NASA’s safety culture changed since Columbia?

"The lessons of Columbia have not been forgotten."

See full article...

 

[ranthog]

Just to add on to my comment earlier: NASA completely screwed up the failure analysis of the foam on Columbia. NASA concluded it was manufacturing defects and told the workers it was their fault.

Nope. It was cracking caused by thermal cycling.

The issue I take with this analysis is that it assumes that manufacturing defects weren't also a cause, just because they found another possible cause. Compounding issues are sneaky like this, especially when you can't possibly get your parts back to inspect.

Testing is hard when you can't fully replicate everything and have to come up with some form of artificial testing. Nor can you compare a used part with the failure to your replicated failures.

Also, I was fairly certain the Columbia report indicated the problems were with the procedure for applying the foam. Not the workers themselves.

 

[Deleted member 764661]

Can Starliner be sent back unmanned? If it can, it seems like a no-brainer to send them back on Dragon.

Technically yes, but some of the previous Ars articles pointed out that that's a mess of its own. (The software currently running on Starliner doesn't support autonomous undocking, so they'd have to install new software before doing it. And then there's the possibility that something goes wrong mid-undocking and now they've got a stranded/uncontrollable Starliner capsule dangerously close to the station.)

 

[ranthog]

In retrospect, the proper way to fly a side-mounted Orbiter would have been to have the wings on the far side of the fuselage from the ET. The easiest way to accomplish that would have been to have the same geometry and just have the heat shield pointed out. It would have looked odd, but it would have been more immune to foam strike damage.

Of course, the proper, proper way to mount the Orbiter would have been on top.

In triple retrospect, we would have been better served flying the Saturn V 135 more times (manned or not) for what was spent on STS.

To be fair, we also would have been far better served if the shuttle had been built to only serve NASA's goals and that the program had been properly funded without being tied to DOD operations.

The cross range capabilities alone significantly increased the chances of a foam strike.

 

[jig]

is it fair to ask the stranded astronauts what they prefer? do they feel too much pressure from the interests wanting starliner to return home with humans aboard?

i wouldn't want to put pressure on them, but at the same time, i want to trust them more than those with less actual skin in the game, so i want to hear their input after they've heard about the valves bulging. they should know, based at least in part on the last recovered starliner test, whether (1) they're still within some kind of reasonable safety standard, and (2) what the conditions were like on the way up this time.

note - i remember that initially, Dragon was pulsing thrusters too often as it approached the ISS (to some reasonable ridicule by the Soyuz group), but that they adjusted fairly quickly to reduce the spew (there were concerns about the extra thrust events coating nearby surfaces with possibly corrosive junk). so, the heating thing is fixable and manageable, particularly when pulling away (when there is less need for precise, minute movements - they just need to aim aft and gooooo).

 

[Tom the Melaniephile]

The issue I take with this analysis is that it assumes that manufacturing defects weren't also a cause, just because they found another possible cause. Compounding issues are sneaky like this, especially when you can't possibly get your parts back to inspect.

Fair enough.

 

[Tom the Melaniephile]

To be fair, we also would have been far better served if the shuttle had been built to only serve NASA's goals and that the program had been properly funded without being tied to DOD operations.

The cross range capabilities alone significantly increased the chances of a foam strike.

IIRC, NASA was the agency pushing for going into DOD operations and taking over all spaceflight activities.

 

[SinclairZX81]

In the last press telcon, they stated that the software still has the auto mode logic. What's missing is the mission data load (configuration file) that sets parameters for what the software does in various situations, currently set to "notify the operator". The alternate settings tell the software to execute the appropriate auto actions. Apparently the vehicle can only fly one of those parameter sets at a time, so they need to install the auto parameters over the manual ones. And, since the software has changed since they last used the auto parameters, it needs to be tested in that configuration.

Okay, that seems a logical sequence, and I'll bow to your experience here.

But as I noted in the other article's thread, if that's the case, how can this possibly qualify as a certification flight? Presumably the operational configuration would have both manual and autonomous modes, and yet those can't both be invoked on this software load? And if the software changed since they last used the auto-function parameters, how again is this a certification flight and not a test flight?

I have zero question with your explanation and your expertise, but rather that there's something really fishy about all these changes and untested flight software modes and loads being used on a certification flight. That's not how it's done.

 

[Chuckstar]

IIRC, NASA was the agency pushing for going into DOD operations and taking over all spaceflight activities.

IIRC, they were trying to take over DoD funding, that way, because Congress wasn’t giving them enough money to build the shuttle. It backfired, in that the shuttle became that much more costly and complex in order to try to meet DoD requirements that were never even used.

 

[snoopy.369]

I understand the principled reasons for having 2 options (Dragon and Starliner), but at this point, what logistical benefits is it providing? What possible technical challenges could arise with Crew Dragon that couldn't be fixed/resolved faster than fixing Starliner? It's clearly demonstrated itself as a reliable crew vehicle.
I no longer understand the practical reason for continuing to entertain the Starliner "option" when the primary vehicle is not going anywhere for the lifespan of the intended Starliner contract.

If there were a problem with Dragon that caused them to not be able to leave safely, they'd want another option (not another Dragon). That's why 2 totally different options is always better than 1. It also would allow them to have a proper competition for launch contracts - while SpaceX is a bit more 'free market company' than 'monopolist' right now, that doesn't mean it always will act that way.

 

[Defenestrar]

In triple retrospect, we would have been better served flying the Saturn V 135 more times (manned or not) for what was spent on STS.

Good thing we've got Artemis now

 

[ranthog]

If there were a problem with Dragon that caused them to not be able to leave safely, they'd want another option (not another Dragon). That's why 2 totally different options is always better than 1. It also would allow them to have a proper competition for launch contracts - while SpaceX is a bit more 'free market company' than 'monopolist' right now, that doesn't mean it always will act that way.

Right now, we also don't want to have to go begging for seats from Russia.

 

[Qtrfoil]

What troubles me about all of this is somewhat simple.

We in this community are the anomaly: we pay attention to this and we care about it.

Speaking from the country to the north that has a pathetically small and not particularly capable space agency, my observation is that NASA seems to be all but invisible and irrelevant to the overwhelming majority of the US public.

The only real 'mainstream' media coverage of this situation has been a couple of small articles describing the astronauts as 'stranded' on the ISS.

Nothing substantial reporting on how Boeing has screwed this up, very little on the NASA normalization of deviation that permitted it to fly in the first place. Nothing discussed within the context of the presidential campaigns about the situation.

It is very disheartening to see such a state of affairs on space policy and space efforts, and why I'm concerned is "well no one is paying attention: no matter what we do, it won't matter"

Not knocking the other dozen Canadian astronauts, but Roberta Bondar and Chris Hadfield get you full credit to have any opinions you want.

 

[Steve austin]

The real problem is Bill Nelson's ties to Boeing. How much of that is going to influence his decision (and is there some sort of back office talk of him taking on a far more lucrative position with Boeing at some point). He's a politician at heart, so my faith is extremely diminished in that regard, especially with how much Boeing greased those palms during his Senate years.

Much as I “really” love innuendo, referencing “back office talk” is rumor or worse (and shortly turning 82 and the very small likelihood that he would have significant value to Boeing, especially given the current and likely next Congress and Boeing’s level of disasterous screw-ups in recent years, makes Boeing hiring him in any capacity seem massively unlikely to me), and claiming he got paid off by Boeing while in the Senate is pretty much slander without evidence. So do you have any actual citations for the claims you made?

 

[Tom the Melaniephile]

IIRC, they were trying to take over DoD funding, that way, because Congress wasn’t giving them enough money to build the shuttle. It backfired, in that the shuttle became that much more costly and complex in order to try to meet DoD requirements that were never even used.

Yes, I believe grabbing DoD funding was the motivation.

 

[Tom the Melaniephile]

If there were a problem with Dragon that caused them to not be able to leave safely, they'd want another option (not another Dragon). That's why 2 totally different options is always better than 1. It also would allow them to have a proper competition for launch contracts - while SpaceX is a bit more 'free market company' than 'monopolist' right now, that doesn't mean it always will act that way.

There's still Soyuz.

NASA has never had more than one option for domestic crew transfer to/from the ISS. For quite awhile they had zero.

I don't see how giving Boeing a free pass on their obviously deficient capsule is a meaningful benefit to NASA.

 

[WilDeliver]

"From my discussions with people inside NASA, there is plenty of opportunity for dissent. People are encouraged to raise their concerns, and they are taken seriously. They are being duly considered."

This is part of the Culture of Safety type of thinking implemented in the Airline industry in the 70's and hospitals in the 2000's that have resulted in fewer accidents, errors and tragedies. The "Captain of the Ship" thinking replaced by "The captain plus the crew." (Well, maybe not at Boeing since its merger with McDonnel Douglas, with its focus on the next quarter's profit and loss statement, stock price and executive compensation package.)

 

[r0twhylr]

A/B title which I am responding to: "I trust NASA’s safety culture this time around, and so should you"

You know what the clincher was for why I don't trust NASA this time around? This snippet from Steven's article:

"If Starliner comes back without its crew, this would almost certainly lead to debate within NASA about whether to require Boeing to complete yet another Starliner test flight before clearing the spacecraft for operational missions"

Why would this "lead to debate"? Boeing has failed their milestone. There should be no debate. Boeing should be required to fix the issues and try again. Or they simply don't get paid.

Now, I do have a base level of distrust, because NASA is so far incapable of a 20 year stretch of crewed spaceflight without killing astronauts because of "go fever" and normalization of deviance. We've been told time and time again that they have "learned their lesson". They were supposed to have learned their lesson after Apollo 1. They were supposed to have learned their lesson after Challenger. They were supposed to have learned their lesson after Columbia.

I have questions about the contract stipulations around this flight. If they bring the astronauts home alive on Starliner, does that mean that the vehicle is automatically approved? How much say does NASA even have in it after that?

 

[1Zach1]

I have questions about the contract stipulations around this flight. If they bring the astronauts home alive on Starliner, does that mean that the vehicle is automatically approved? How much say does NASA even have in it after that?

Contracting Officer makes the final call on if it's successful or not, it's not an automatic thing.

 

[10Nov1775]

The real problem is Bill Nelson's ties to Boeing. How much of that is going to influence his decision (and is there some sort of back office talk of him taking on a far more lucrative position with Boeing at some point). He's a politician at heart, so my faith is extremely diminished in that regard, especially with how much Boeing greased those palms during his Senate years.

Being a politician at heart could be a very good thing here.

They generally try and avoid getting disasters laid at their door, and are quite good at explaining to people how their hands were tied when making unpopular decisions (i.e. if he has to go back to Boeing and tell them why he brought the crew home on Dragon).

I suspect part of this prolonged twisting in the wind is to give every opportunity for Boeing to hang itself (or demonstrate the issue is actually totally safe), while allowing public sentiment to build to levels that can't be ignored.

"I'm sorry, guys. I gave you all the time I could. There's nothing I can do."

 

[ranthog]

There's still Soyuz.

NASA has never had more than one option for domestic crew transfer to/from the ISS. For quite awhile they had zero.

I don't see how giving Boeing a free pass on their obviously deficient capsule is a meaningful benefit to NASA.

Then you hold Boeing's ass to the fire to get them to correct the issues with the capsule at their own expense. That is the advantage of these contracts. Its flown three times now, I think we have a fairly good idea what problems need fixed.

Yeah, because we want to be reliant on Soyuz when we're basically backing the other side in a war with Russia to the tune of billions of dollars of munitions and equipment.

Right now, more than ever, the international community needs its own backup ride to the ISS.

 

[butcherg]

Okay, that seems a logical sequence, and I'll bow to your experience here.

But as I noted in the other article's thread, if that's the case, how can this possibly qualify as a certification flight? Presumably the operational configuration would have both manual and autonomous modes, and yet those can't both be invoked on this software load? And if the software changed since they last used the auto-function parameters, how again is this a certification flight and not a test flight?

I have zero question with your explanation and your expertise, but rather that there's something really fishy about all these changes and untested flight software modes and loads being used on a certification flight. That's not how it's done.

These are good questions; flight-qualified software development is a lot more tedious than most other types of development, for good reason...

Your presumption that both modes should be enabled is prudent, but not necessarily how the requirements might flow. Since the mission data load apparently can contain only one set of parameters, it's a configuration change with all the attendant CM paperwork. That's in itself is not particularly onerous, what appears to be onerous is the need to re-qual the software, and that would likely be driven by the requirements of the specific mission. I'm pretty sure the requirements writers looked at the mission and figured there'd not be any need for the auto parameters, so they did not include testing of them against the cert candidate version, as that would require a configuration change of the test lab.

Just guessing, mind you, but this would fit with how we'd have handled such on the systems on which I worked...

 

[Kiwi_Andy]

With all due respect to Eric, if the tragic lessons of the past had been even slightly embedded then a human launch of Starliner wouldn’t have happened, because of the raised uncertainty.

 

[gefitz]

I hope you’re right, but all the bluster and bullshit we have seen which got us here in the first place looks a lot like go fever with a side order of normalized deviance.

The right decision is the hard one, and I’m cautiously optimistic but in no way confident they won’t just throw the dice and cross their fingers.

I'm personally waiting to hear the news that Boeing has "gotten into the ears" of the House Committee on Science Space and Tech, or the Senate Subcommittee on Space and Science on the subject, and that the pressure on NASA to throw the dice is loud and clear.

It's gross to think of this...but boy I sure wouldn't put the chances of this very thing happening right now at zero.

 

[Tom the Melaniephile]

Right now, more than ever, the international community needs its own backup ride to the ISS.

I agree. My concern is that Boeing is being declared to be that backup when they have failed again and again and again - for a FAR higher price tag than the successful Crew Dragon.

 

[Tom the Melaniephile]

These are good questions; flight-qualified software development is a lot more tedious than most other types of development, for good reason...

Your presumption that both modes should be enabled is prudent, but not necessarily how the requirements might flow. Since the mission data load apparently can contain only one set of parameters, it's a configuration change with all the attendant CM paperwork.

That sounds like a really, really shitty set of requirements/parameters. Any semi-modern vehicle carrying crew should be capable of both manual and autonomous mode. Hell, STS/Shuttle should have had that capability, and there was no technical reason to eliminate that capability.

 

[HBrown]

Sure, but if they make that choice after management says go (or even while management is still pondering) - they will be permanently grounded. NASA culture demands an exceptionally gung-ho, always-positive output from astronauts. Express any doubt about NASA or the program, and you're done. Permanently.

Butch is 61 and Sunni is 58. The next 3 missions (crew 9, crew 10, and starliner 1) to the ISS already have crews assigned. So: the absolute earliest they would fly again with NASA is 2026. And this is far from guaranteed.

And NASA is no longer the only option. Later this year, Peggy Whitson will fly her 2nd mission with Axiom space since retiring from NASA when she commands the Axiom 4 mission. And if any of the proposed commercial space stations currently in planning get built, there will be increased demand for astronauts among private space companies in the coming decade.

 

[Oldmanalex]

I think I could believe NASA has learned something since 1986. Boeing, not so much.

I do not think that the problem is Boeing not learning new stuff. Learning new stuff from the MBAs while forgetting the engineering drilled into them by Bill Boeing is the problem.

 

[butcherg]

That sounds like a really, really shitty set of requirements/parameters. Any semi-modern vehicle carrying crew should be capable of both manual and autonomous mode. Hell, STS/Shuttle should have had that capability, and there was no technical reason to eliminate that capability.

Okay, this might seem pedantic, but the vehicle does have that capability. They didn't eliminate it, they just didn't configure and test the auto mode for this mission. Now, it would seem based on all that's transpired that the ability to switch configurations mid-flight would be a thing, remains to be seen if that is implemented as a capability given that the eventual nominal operation of the vehicle is crewed.

Now, I'd surmise that they'll be testing both in every subsequent version, so they don't have to do four weeks of unplanned testing if another such situation arises...

 

[ranthog]

I agree. My concern is that Boeing is being declared to be that backup when they have failed again and again and again - for a FAR higher price tag than the successful Crew Dragon.

Yes, anything that wasn't flying on F9 was going to be significantly higher costs these days. I think we did eliminate our third choice from the competition too early, but that is congress for you.

We don't want to go back to having one source for launches, and right now that means making it so that someone besides SpaceX is going to win the second place contract. You'll also never have redundancy right now because SpaceX can significantly undercut its competition on launch costs.

As far as the current problems, NASA needs to be holding Boeing responsible to live up to the needed standards.

 

[blookoolaid]

I feel like the root of this issue and many other issues in the Starliner program is the bespoke nature of the whole development process. When you only ever make a few capsules and only fly them a few times it's really hard to amortize the costs to be able to afford a reasonable test campaign.

The sane thing to do would be to insist on unmanned test flights until everything functions nominally. Then, and only then, fly the exact same craft with the exact same software but with people. The only reason I can see for avoiding this pattern is cost. Everything is low volume and likely highly manual (witness the month Boeing expects to need to retest their software in a different configuration). This leads to a cycle of rising costs in the same way that mass production leads to a cycle of falling costs.

One of SpaceX's biggest advantages is that they have a culture of mass production. Mass production requires higher levels of QA automation and process standardization. The infrastructure and culture that supports mass production then gets applied to lower volume programs like Dragon as well leading to much more reliable outcomes for minimal extra cost.

Really the lesson to learn here is that the SpaceX way fo doing development works and the Boeing way doesn't. Continuing to fund Starliner is a waste of money. If NASA can't cancel the contract they should at least insist that Boeing fly at least one flawless unmanned flight before allowing them to try again with people. Not that this will prove the craft safe, but it is a minimal standard they should have to meet. Anything less and NASA definitely hasn't learned the right lessons from their past failures.

 

[DistinctivelyCanuck]

Not knocking the other dozen Canadian astronauts, but Roberta Bondar and Chris Hadfield get you full credit to have any opinions you want.

Given the capacity of Canada's technology sector, the relative strength of our economy, the past history of Canadian space accomplishments: the current state of the CSA is what I was referring to when I spoke of our weakness.

This is a Canadian Space Agency that is, quite seriously, shutting down a major space sciences test center, a facility with a decades long history of achievement, because it is, effectively, "in the wrong city"

 

[Wickwick]

It's perhaps a good job they didn't spin up a Dragon Flight. That might have gone up on the Falcon 9 that launched recently and had a launch anomaly. SpaceX / NASA were fortunate that that 2nd stage that failed had only StarLink satellites on top, and not a crewed Dragon capsule.

If it had been a crew launch, goodness knows what would have happened to them. Probably - best case - a scarcely controlled reentry into a random location; pretty life threatening. Worst case - an uncontrolled reentry in the wrong orientation, and burning up.

What would have happened is ... the capsule would have been inserted into its orbit correctly and SpaceX would have discovered the problem and been able to fix it without losing a mission's worth of Starlink satellites.

The second stage problem only happened during ignition of the second second-stage burn. During a Dragon launch, there is only one such burn.

 

[rick9004]

I would feel better if the two astronauts had the final say. After all their lives are at stake.

 

[ranthog]

One of SpaceX's biggest advantages is that they have a culture of mass production. Mass production requires higher levels of QA automation and process standardization. The infrastructure and culture that supports mass production then gets applied to lower volume programs like Dragon as well leading to much more reliable outcomes for minimal extra cost.

The counterpoint to this would be the 737 Max. Mass production really doesn't have anything to do with it. It is almost certainly just a cultural issue now at Boeing.

 

[barrattm]

NASA itself does seem more willing to let people say "stop", or "no". However, it appears that it is corporately still vulnerable to commercial and / or political pressures. Somehow NASA were sweet-talked into OKaying the launch of StarLiner, a decision that has looked to be unnecessary and poorly judged from the moment things started to go wrong.

But it was anyway clearly unnecessary before they even launched the crew. They could have accomplished all they needed to without a crew on board. Ok, so they may have a ton of useful telemetry, but that's not a result of the crew writing down numbers in a notepad. Had they managed to get to the ISS trouble free and crewless, that'd have been an even greater feather in their cap ("so good it doesn't need a crew to drive it"). As things are, it's a bad outcome ("so bad it needs a crew to drive it, and even then they only just made it").

Frankly, I suspect they could have accomplished a whole lot of what they needed to do without launching at all. No one else seems to have issues ground testing and getting to a higher TRL before bothering with a launch.

So as far as I can see, putting a crew on board and sending it up certainly has the look and feel of a commercial/political stunt, and it's got NASA's name on it.

So an interesting question is, how vulnerable to influence from stressed external suppliers are NASA's team? Clearly, they're not invulnerable to such external influence. How then to ensure that a supplier's woes aren't brought to the decision making table?

Probably the best thing to learn from this is that a stressed supplier is always going to struggle to be a safe supplier. If they're under stress - stop. NASA - back in the Challenger days - were themselves a stressed organisation, and that killed a crew.

 

[Tom the Melaniephile]

Okay, this might seem pedantic, but the vehicle does have that capability. They didn't eliminate it, they just didn't configure and test the auto mode for this mission.

Thank you for confirming that the Starliner currently in question does NOT have that capability because it was eliminated by Boeing. "Didn't configure" is functionally equivalent to "deleted" when it's going to take weeks (more?) of software updates to allow autonomous mode.

 

[Ragashingo]

I find it fascinating that both Eric Berger and Stephen Clark published lengthy, well argued articles on why NASA should not send the astronauts home on Starliner. I'm enjoying both articles, but it feels like some sort of bigger picture thing is going on.

A journalistic warning to NASA upon hearing that the wrong decision was made? Or journalistic praise to help NASA higher ups justify their correct decision to others in the political spheres? Or just crazy coincidence when it seems like coverage and page views could have been milked more successfully by running these articles a day or two apart?

I'm not criticizing at all. In this case, I think experienced Space journalists should put out opinion pieces to help NASA along. It's just curious to maybe see that actually happening.

 

[rich6285]

Where s the NASA Safety Advisory Group ? They are suppose to be independent of the NASA bureaucracy and we have not heard one bit from them on this problem.

 

[Wickwick]

I would feel better if the two astronauts had the final say. After all their lives are at stake.

As I noted in an earlier comment thread, the two astronauts in question are both test pilots and are probably more comfortable with professional risk than NASA's administration is. As such, they probably won't bat an eye if NASA says they're flying home in a Starliner, but if NASA feels that prudence requires a return on a Dragon, I don't know that the astronauts should necessarily have a vote. They may have some input on their willingness to stay until February, potentially, but I have yet to hear of an astronaut turning down a longer mission in space.