You're selecting a scenario to assume your conclusion. But in traffic safety, for example, we could easily demonstrate the efficacy of AI if/when it exists.
The point is that the "black box" can certainly be a critical failure in certain cases, but it isn't necessarily a critical failure in all.