Finally, engineers have a clue that could help them save Voyager 1

A new signal from humanity's most distant spacecraft could be the key to restoring it.

See full article...

 

[Nexus]

Wasn't this an episode of Star Trek?

Jokes aside, how freaking cool is it that this thing is still going? Hats off to the team still working to keep it running.

Hell I think they made a whole motion picture about it!

That said:

it takes about 45 hours for engineers on the ground to know how the spacecraft reacted to their commands—the one-way light travel time is about 22.5 hours.

I was going to ask this exact question, thanks.

 

[TheoLib]

My understanding is that there was an assembler, but it ran on a computer that no longer exists ...

The old tools ran on Univac mainframes. There might have been 3 sets of assemblers/linkers. One for the CCS command computer (whose CPU was borrowed from Viking orbiter). One for the AACS attitude control computer, which used a modified CCS CPU (especially an added address index register); I don't know if the changes affected the instruction set. And definitely another one for the FDS computer's custom-designed CPU.

I don't know for sure, but, in line with what om1 said, I don't think it would have been that hard to port the tools to a minicomputer (e.g., VAX) and eventually to a UNIX workstation or PC.

However ... I am not sure how useful those tools would be now, except for the semi assembler you describe. This 1985 paper seems to indicate that they've always just patched the onboard flight software:

The AACS and CCS programs were modified without being reassembled as is the case with all AACS and CCS changes since [the 1977] launch.

McLaughlin and Wolff, "Voyager Flight Engineering: Preparing for Uranus", p.11, https://arc.aiaa.org/doi/abs/10.2514/6.1985-287

(I don't know if that applies to the FDS software too or not.)

This reminds me of an account about the ground testing of the earlier Viking lander computer (GCSC). The assembler running on an IBM mainframe was not very good, so they manually patched the software until they got it running. (The lander's CPU itself was the third choice in CPUs, chosen because of its low weight despite being more difficult to program than the top two.)

The flight software development process was hindered by the lack of the assembler that was adequate for the task. The first assembler provided fixed address code. About halfway through the development phase, the Flight Software group made the decision to develop a relocatable assembler, which was needed to cope with the change activity and the multiple revisions of the flight code. Later on a decision was made to use the IBM 370 assembler, which was also being used for the Flight Operations Command software. After one or two assemblies were made with this assembler, the project manager made the decision that all future updates would be accomplished by manual "patches" using the basic GCSC octal code. This was done even through a great many changes had to be "patched" because of problems discovered in integrated testing (see section 3.3). The result was that a "listing" of the flight code had to be created post facto using the IBM 370 assembler, which was a laborious and costly process. Because of the patching, i.e. using "jumps" to unused memory and jumping back, the resulting listing of a contiguous function was not contiguous in core. This proved to be a burden for Flight Operations during the mission in reviewing Memory readouts and making software changes via uplink.

Martin Marietta Corporation Viking Software Data: Final Technical Report, p. 110, https://apps.dtic.mil/sti/pdfs/ADA040770.pdf (15MB PDF)

Rereading that reminds me that you can use a host computer's macro assembler to cross-assemble programs for another CPU. I remember that the Kermit distribution c. 1982 used a DEC mainframe macro assembler to generate Z-80 binaries for the CP/M version of Kermit.

 

[hagmanti]

Oh dear, young man:
I am not a man. I started working on VGR in 1975. I worked on it off and on until I retired from JPL in 2012, as a principal telemetry engineer. We were not all young men, and are not all now old men. Language is important. Best wishes to you in your enlightment.

Thank you so much for your work here! We are all in awe of what you accomplished!

If you have stories to tell about this, I (for one) would be super interested.

 

[uhuznaa]

I love the Ars comment section.

Yeah, you sometimes have a sweet spot where somehow enough of the right people gather (while also having other and really big spots where there's nothing but blunt opinions).

There also was a time in the late 90ties when in good old Usenet in sci.space.* you had people from NASA and from companies around it, and from the then just coming up New Space companies, and even actual astronauts coming together and discussing things.

Just that back then this was absolutely frustrating, because basically everyone from the trenches agreed that the right way forward would be simple and cheap two-stage launchers and returning and landing and reusing the first stage would be low-hanging fruit and Hydrolox engines plus solids are idiotic compared to hydrocarbon engines... but what was being launched and paid for was exactly the idiotic choice and NASA was going for hydrolox SSTO, the most idiotic of all idiotic approaches.

At least this is different today and I'm so happy about this.

I hesitate to apply this to the software side of things though. I fear we will have to see some spacecraft fail due to software being too complex and overambitious and relying on too much dependencies. Only when a spacecraft about to land on a Mars base will crash and kill the crew because the software decides to do an update while coming in at 1 km/s and aiming at the landing pad we will look back and search for the point where we took the wrong step...

 

[acefsw]

Oh dear, young man:
I am not a man. I started working on VGR in 1975. I worked on it off and on until I retired from JPL in 2012, as a principal telemetry engineer. We were not all young men, and are not all now old men. Language is important. Best wishes to you in your enlightment.

Thank you very much for your contributions on Voyager and at JPL. Having a 37 year career there, I would imagine you have some great stories to tell and I would love to hear them.

Edit to add that having someone like Betsyw who had a 37 year career at JPL comment, (even if it's to chastise someone for sounding sexist, - and it did, sorry panton41), is what makes ars great.

 

[Slik]

It is indeed very impressive. Almost a light day, as someone said.

Now, I understand that voyager q moves through space at a little over 60 000 km/h but how fast could we possibly accelerate a space probe to, using current tech? Say with maximum acceleration from both chemical, nuclear and stuff like solar sails in addition to a larger number of high energy slingshot manoeuvres?

Current tech doesn't include nuclear rockets, at least not in a design that has been built and tested in space.

The fastest vehicle we have launched outwards was the Dawn spacecraft, which has an ion thruster, and a gravity assist from Mars. It launched on a Delra II. The Falcon heavy currently has more thrust, but I'm not sure by how much. (The Parker solar probe is MUCH faster, but it's near the bottom of a massive gravity well)

The gravity assists are the real problem though. The Voyager spacecraft had the advantage of a once in every 150 year (roughly) alignment that allowed all those gravity assists. Getting more than that just isn't possible.

We could probably manage a Venus-Mars-Jupiter series, but I think that would be about it.

So, if you took a fully used Falcon Heavy (no reuse), put a whomping big extra stage, like the Star 48, did a 3 planet assist, used an ion thruster, and had a small probe (one the order of 500 kg or less) We could likely exceed the Voyagers' velocity by a good margin. How fast, exactly? I'll leave that to smarter folks than me, like Scott Manley, Wickwick and others.

 

[Deranged]

Current tech doesn't include nuclear rockets, at least not in a design that has been built and tested in space.

The fastest vehicle we have launched outwards was the Dawn spacecraft, which has an ion thruster, and a gravity assist from Mars. It launched on a Delra II. The Falcon heavy currently has more thrust, but I'm not sure by how much. (The Parker solar probe is MUCH faster, but it's near the bottom of a massive gravity well)

The gravity assists are the real problem though. The Voyager spacecraft had the advantage of a once in every 150 year (roughly) alignment that allowed all those gravity assists. Getting more than that just isn't possible.

We could probably manage a Venus-Mars-Jupiter series, but I think that would be about it.

So, if you took a fully used Falcon Heavy (no reuse), put a whomping big extra stage, like the Star 48, did a 3 planet assist, used an ion thruster, and had a small probe (one the order of 500 kg or less) We could likely exceed the Voyagers' velocity by a good margin. How fast, exactly? I'll leave that to smarter folks than me, like Scott Manley, Wickwick and others.

Good answer, thanks!
As for nuclear rockets, you’re right we don’t have any working but at least we know it’s feasible. so maybe I should’ve said tech that is realistic given our current technological level.

 

[arc-tu-rus]

We can infer this was at least part of the expected response from this:

IOW: it did the same thing when it was working.

No, it did not. If Voyager-1 responded as usual, then understanding the format of the response message wouldn't have required the sleuthing abilities of one engineer.

What happened is that NASA crafted a specific command that requested Voyager 1 to run a bunch of different instructions in different modules as an attempt to circumvent what NASA believes to be a data corruption issue in the FDS. As a result, Voyager 1 responded with what seemed at first to be yet another "unreadable" data stream, similar to what NASA has been receiving for several months. However, someone managed to notice that part of that "unreadable" data stream actually included a full dump of the FDS memory. In normal circumstances, Voyager is not supposed to send back a dump of its memory - the usual message is a packet of science and telemetry data, which is packaged by the FDS and sent back to Earth by the telemetry modulation unit. Anyway, this could enable NASA understanding if there is some error in the FDS instruction set or data set.

 

[Qtrfoil]

If this leads to the mission continuing I'd sure like to know the name(s) of the engineer(s) who noticed that the signal was no longer gibberish. It seems the equivalent of John Aaron noticing something he'd seen before and coming up with setting SCE to AUX.

 

[terrydactyl]

There's three different computers (actually 6, a main and a backup of each one) of two different architectures running on voyager. Most of the code is compiled Fortran. Actually FORTRAN, this is old school computing.

There was recently a cool documentary called "It's quieter in the twilight" about the small group of greying engineers who are responsible for the day-to-day operations for Voyager, a job most of them have been doing for decades.

Thanks for pointing out "It's quieter in the twilight." On Amazon Prime at least and I plan to watch it.

 

[terrydactyl]

Oh dear, young man:
I am not a man. I started working on VGR in 1975. I worked on it off and on until I retired from JPL in 2012, as a principal telemetry engineer. We were not all young men, and are not all now old men. Language is important. Best wishes to you in your enlightment.

I'm acquainted with a woman who, in the 60s, worked at JPL on the Mariner programs. She recounted how some telemetry data was relayed to scientist... via a phone call, calling out the numbers.

 

[iced_and_alone]

I guess it could refer to interrogation techniques where interrogators are trying to crack the suspect using questionable methods involving abuse of the suspect? But, that does sound like a stretch if so, almost to the point of intentionally misunderstanding the original expression of “cracking a case”.

To open the first can of a case of beer.

 

[hagmanti]

I wasn't excluding women, I was discussing the men specifically. (Women generally don't grow wizard beards.)

The thing is, when you are only talking about the men-- a perfectly rational way to describe that is "excluding women". I'll believe or grant that you didn't have nefarious goals or motives, but . . . that's just what it is. The thing about excluding people who are in the minority is that no one means to do it, it's just that they never notice when they do it. Inclusion takes effort.

Also, the thing where you are explaining to a woman who has a lifetime of experience dealing with institutional sexism how you're not being sexist -- that feels a lot like mansplaining (in addition to being wrong). Maybe I'm just senstized by your arguing with someone I'd really like to hear more from, but that's the way it feels to me.

Look I'll probably not change any minds here. But this seems like such an obvious example to me, and I couldn't figure out how not to say anything. I literally slept on it, and couldn't keep my trap shut.

 

[Kahurangikea]

I can't believe the Voyager's computer hardware from 1977 is still working almost 50 years later. What on earth kind of tech is in this thing? What programming language does it use? It's all really cool. Do they use older engineers familiar with the old hardware and language or do they train new ones to handle it? Because it must be completely obsolete in general usage by now.

Almost certainly assembly language for precise control and efficiency, considering the date and small capacity of the system. Anyone working with the software would familiarize themselves with the hardware. There were only 32k words total memory in all all computers on the spacecraft, according to Wikipedia article, presumably referenced.

 

[Emperor_of_Mankind]

Does NASA have a duplicate of the Voyager-1 hardware in a lab here on earth that they can load the memory readout from V-1 into and power on, to see if it behaves the same way, and to test potential fixes on?

Everybody has a testing environment. Some folks are lucky enough to have a separate environment to run production in.

I'd be really surprised if they had accurate test beds to play with, but I'm just a guy on the internet.

 

[fellow human]

nvm

 

[Derecho Imminent]

They have finally translated the code:
The Creator must join with V'ger.

 

[Qtrfoil]

Thanks for pointing out "It's quieter in the twilight." On Amazon Prime at least and I plan to watch it.

Watch "Good Night Oppy" while you're at it, just brilliant.

View: https://www.amazon.com/Good-Night-Oppy-Ryan-White/dp/B0B8JMCCJR

 

[rpali]

They would not have to re-create the hardware, they "just" need to create an emulator.

But then if a glitch pops up, it could be Voyager, or it could be the emulator. Not ideal.

 

[Betsyw]

Thanks - I was just repeating what I had found all over the place about the language, not realizing that the factual well was poisoned. What you describe makes a hell of a lot more sense.

I did not say that hardware duplication was a new idea. What I said was that NASA didn't do it in the 1970s, and didn't routinely build them for space probes until several years after the Voyagers launched. Having some processors and partial systems in the lab is not the same thing as building an entire spacecraft and keeping it on hand for testing.

Interesting side note to this. When the Galileo spacecraft tape recorder (yes, tape!) had a problem, the team had to go to the proof test model which was in the JPL museum to get to the only working copy. They took it out and didn’t “solve” the problem, but realized what the problem was and figured out workarounds. It made for some very tortured telemetry. Now, of course, it is all solid-state, but that can have bit errors anyway. It was then my job to work on that telemetry with the flight s/w team, and then write the requirements for the ground processing of that tortured telemetry.

 

[hagmanti]

Interesting side note to this. When the Galileo spacecraft tape recorder (yes, tape!) had a problem, the team had to go to the proof test model which was in the JPL museum to get to the only working copy. They took it out and didn’t “solve” the problem, but realized what the problem was and figured out workarounds. It made for some very tortured telemetry. Now, of course, it is all solid-state, but that can have bit errors anyway. It was then my job to work on that telemetry with the flight s/w team, and then write the requirements for the ground processing of that tortured telemetry.

What physical size was that tape?

Did you have to adjust the incoming signal processing to undo the damage the tape was causing to the data? It was deterministic enough to do that?

 

[Betsyw]

What physical size was that tape?

Did you have to adjust the incoming signal processing to undo the damage the tape was causing to the data? It was deterministic enough to do that?

The whole apparatus was somewhat larger than a shoe box. I think the tape was maybe 1”, but flight h/w was not my thing. You might be able to find out more about it somewhere. The problem was not damaged data, but tape slippage around the capstan. So tape usage and speed changes were highly confined. Downlinked data was in pieces and not always sequential. It had to be put back together. Think of a donut. Sometimes you got a vertical slice through a side, sometimes a horizontal shaving off the top, with gaps in it. The filling for the gaps came down at different times. I forget a lot of the details, of course. But the 1s and 0s were correct. Due to the failure of the high gain antenna, the data rate was extremely low, so every bit was precious. I once got a call from a scientist to find a missing chunk of about 500 bytes. It was taken when going through particles from an Io eruption. The data was not good enough for our normal processes, but some special reprocessing got it.

 

[Me Myself And I1]

Instead of troubkeshooting, the should start decoding

"That yours? Flew off your planet and hit my fender!"

 

[peppeddu]

ChatGPT is extraordinary good, but I suspect it’ll be a while before I can copy some hex into the prompt and it’ll realize that it looks like a memory readout from Voyager 1

CharGPT as we know it today is only a language manipulation engine.
It can't solve anything that humans haven't solved before, so don't expect to give it a problem unknown to any of us and get a useful answer.

 

[marsilies]

CharGPT as we know it today is only a language manipulation engine.
It can't solve anything that humans haven't solved before, so don't expect to give it a problem unknown to any of us and get a useful answer.

Hell, don't give it a problem known to any of us and get a useful answer. You see studies where it gets math problems right like 87% of the time, which, sure, is impressive for something that technically never learned math, but since its output is all statistical probabilities, it's never going to reach 100%.

 

[fenncruz]

Came across this question, but not entirely satisfied with the answer:
Is the pulsar map on the Pioneer/Voyager probes only meaningful for earthlings?

Essentially, it concludes the Pulsar map won't be usable or accurate as a reference after a million years due to all the interplaying motions from the pulsars up to the galaxy level.

My thought is this: Whoever if ever finds this will more likely than not also have some idea of how things move around in galaxies and how they change over time. They should also know the universe is expanding. Could they just take their own existing model (easier if in our own galaxy), and run it backwards until they see the pattern match? There seems to be enough reference points there to make it stand out in a crowd.

Open to other thoughts.

The expansion of the Universe doesn't matter for this as the pulsars are in our galaxy. The difficulty is that the further you try to look backwards the more your errors in positions and velocities addup. Also pulsars change their timings over time. Also a few million years is the kinda of timescales to start worrying about new pulsars being formed to mess things up.

It's not impossible, and any alien race sufficiently advanced to fly in interstellar space to pick up what's left of voyager would have the capability. It's probably easier though for them to just search for planets around stars and look for things they think life should exist on. Or even just visit all the star systems given they must have some faster than light technology if they collected voyager from deep space.

 

[ghostofchristmas]

So what you're saying, what you're actually saying, is that a 50 year old computer, OUTSIDE the solar ystem, did a BSOD and sent back a memory dump, on it's own? That's just awesome.

Is it running Vista?

Meanwhile, at Apple: "Oh we changed something in ios....hey all of you millions of app developers, rebuild your apps quick otherwise it won't work in 3 months"

 

[ranthog]

It has to be a good sign that it is healthy enough to actually dump the entire memory. That requires a lot of things to go right to dump everything. Not great that it wasn't the data they were asking for, but hopefully it contains what they need to figure out how to fix the problem.

 

[guavasec]

There was NO operating system on the flight computers. Each computer ran what's called a real-time, cyclic executive. A timer interrupt marked out regularly spaced time slots. The FDS computer that's the subject of this article had 2.5-ms time slots. Within a time slot, the executive would execute each routine/function scheduled for that slot. The sum total time taken by the scheduled functions had to fit within the time slot.

So, as I understand it, that's massively more primitive than the Apollo LEM guidance code, which sounds like an astonishingly modern preemptive priority based task switching RTOS.

Interesting that they had that architecture and didn't leverage it, though without modern language tools maybe it would've been too labor intensive to implement and validate bespoke each time.

 

[ranthog]

So, as I understand it, that's massively more primitive than the Apollo LEM guidance code, which sounds like an astonishingly modern preemptive priority based task switching RTOS.

Interesting that they had that architecture and didn't leverage it, though without modern language tools maybe it would've been too labor intensive to implement and validate bespoke each time.

I would not say that this is more primitive. Both are valid methods of handling and scheduling real-time tasks. For something like Voyager, having something relatively simple for this scheduling makes sense if you can withstand the overhead this style of scheduling requires.

 

[fellow human]

The whole apparatus was somewhat larger than a shoe box. I think the tape was maybe 1”, but flight h/w was not my thing. You might be able to find out more about it somewhere. The problem was not damaged data, but tape slippage around the capstan. So tape usage and speed changes were highly confined. Downlinked data was in pieces and not always sequential. It had to be put back together. Think of a donut. Sometimes you got a vertical slice through a side, sometimes a horizontal shaving off the top, with gaps in it. The filling for the gaps came down at different times. I forget a lot of the details, of course. But the 1s and 0s were correct. Due to the failure of the high gain antenna, the data rate was extremely low, so every bit was precious. I once got a call from a scientist to find a missing chunk of about 500 bytes. It was taken when going through particles from an Io eruption. The data was not good enough for our normal processes, but some special reprocessing got it.

Some wizards don't wear beards

 

[Zapmymac]

"its been a long road, getting from here to there..."

I have faith of the heart & in these old school Scientists / Engineers

NASA: the gift that keeps on giving

 

[mrwerley]

I just read about this! It (unfortunately) wasn't a thing in those days to build a duplicate and keep it at home and operational for exactly this situation. That didn't start as a standard practice for another decade.

I wonder how much it would cost to manufacture a new one now? Probably a lot because I assume all of the hardware would have to be manufactured custom, to spec. . . although those specs should be pretty easy to achieve.

It shouldn't be as expensive as building, say, another James Webb Space Telescope.

They wouldn't have to duplicate everything, just the computer systems, but essentially all of that electronics will be long out of manufacturing. It'd be hopeless. Instead they're using virtual machine simulations running on modern hardware. Far from ideal, but at least it's something.

 

[stilgars]

Oh dear, young man:
I am not a man. I started working on VGR in 1975. I worked on it off and on until I retired from JPL in 2012, as a principal telemetry engineer. We were not all young men, and are not all now old men. Language is important. Best wishes to you in your enlightment.

Your graceful but firm a comment speaks volume about your character, and the challenges you had faced as a lady working as an engineer in the seventies.
Happy retirement.

 

[fellow human]

They wouldn't have to duplicate everything, just the computer systems, but essentially all of that electronics will be long out of manufacturing. It'd be hopeless. Instead they're using virtual machine simulations running on modern hardware. Far from ideal, but at least it's something.

It'd probably be overkill but theoretically they could probably even emulate the hardware. I know in the world of synthesizers they do this, emulating the actual analog circuits of old machines. In some cases the reason is that you can't buy components with as low tolerances as used to be common so even if you rebuilt the machine identically it wouldn't sound the same because today's components are too good.

 

[DM619]

I'll take it a stepnfurr

I was going to say something similar. I imagine some of young men who started their careers on Voyager are old enough to have a truly epic wizard beard by now.

I'll take it a step further and speculate that some of the engineers working the problem weren't even born when Voyager was launched.

Think about it: you're an engineer on a team working a problem on a spacecraft not even in this solar system with no Earth-bound twin to test on, launched when you were in diapers. How cool is that??

 

[TheoLib]

For something like Voyager, having something relatively simple for this scheduling makes sense if you can withstand the overhead this style of scheduling requires.

I thought I'd read something somewhere about Voyager vs. Apollo and I finally found it, though it was actually about the Viking orbiter's sole computer, the CCS (the same computer used on Voyager):

Even though many interrupts were available, most routines as coded had all but the internal error and counting interrupts disabled. Many routines were free to run out without being interrupted, in contrast to the highly interrupted Apollo and shuttle software. Programmers avoided the memory and processing time overhead required to preserve the current accumulator and register contents during an interrupt.

James E. Tomayko, Computers in Spaceflight: The NASA Experience, "Viking computer systems", 1988.

(Incidentally, from the same book, "Voyager - The flying computer center" describes the Voyager computers.)

 

[domikai]

The expansion of the Universe doesn't matter for this as the pulsars are in our galaxy. The difficulty is that the further you try to look backwards the more your errors in positions and velocities addup. Also pulsars change their timings over time. Also a few million years is the kinda of timescales to start worrying about new pulsars being formed to mess things up.

It's not impossible, and any alien race sufficiently advanced to fly in interstellar space to pick up what's left of voyager would have the capability. It's probably easier though for them to just search for planets around stars and look for things they think life should exist on. Or even just visit all the star systems given they must have some faster than light technology if they collected voyager from deep space.

I remember a short story about a very lost vessel noticed by an advanced entity who notes the isotopic ratios and pops the ship back to Sol.

 

[Xerxex]

Since Voyager 10 was never launched we know we’re not in the canonical timeline.

We must be on Futurama's then (watch out for V-GINY).