Onboard Software Update

A spacecraft flies with the software it launched with, and every subsequent line of code has to reach it over a link that runs at a few hundred to a few thousand bits per second, with a round trip of minutes to hours and no way to plug in a cable if the write goes wrong. Voyager and Galileo were already being reprogrammed in flight by the 1970s and 1980s, using distributed onboard computers whose word length, memory and instruction sets were built for exactly that kind of update [9]. The practice has not gotten safer with time: Viking 1 was lost when a patch permanently overwrote the code needed to talk to Earth, and the MSL patch checklist still asks a cognizant engineer who has never studied that loss to stop and read about it before continuing. Curiosity alone has taken nine full updates or cold patches and more than 57,000 hot patch installations since 2012 [2][3].

Three mechanisms, one shared design problem

Method	What it changes	Persistence	Uplink volume
Hot patch	RAM only, applied by an install script at boot	Reinstalled on every boot; skipping the script reverts the vehicle	small
Cold patch	Modifies a copy of the existing image, result burned back to non-volatile memory	Survives reboot	2 MB for MER R9.1, February 2005
Full update or load	Entire new image uplinked and burned to non-volatile memory	Survives reboot	about 8 MB on MER, 9.97 MiB in 49 files on MSL

Sources: [1] for MER, [2] and [3] for MSL. A hot patch is fast to author, review, test and fly, but must be reinstalled at every boot, which is also its safety property: booting without the script gives a clean, unpatched system [2]. A cold patch is more permanent and more dangerous, because memory write commands bypass the built-in checks on data and address validity [3]. Cassini’s AACS team treated that danger as reason enough to avoid full loads almost entirely: a full load resets the prime attitude control flight computer, so all three software loads flown in the first 2.5 years of Cassini’s prime mission were constant-parameter patch loads, with variable parameters changed by memory write commands instead, each verified by reading the memory back three times before and three times after the write [8]. MER and MSL made the opposite trade and accepted full loads because the flight software itself changed, not just its parameters.

Neither mechanism is easy to test on the ground. Hot and cold patches are, by design, not integrated into the flight software monolith, so they cannot be run against the same unit tests as the baseline build, and several are release-specific and do not work in the development sandboxes. Multiple simultaneous patches have to be exercised together in operationally realistic combinations, a combinatorial burden that grows with the number of active patches [2]. A single patch can still be tested exhaustively where a full release cannot: Curiosity’s arm kinematics fix was validated against 3,749,460 randomized unit test cases before flight, a scale of testing that a monolithic release recompile could not repeat on the same schedule [7].

MER: how a load is actually uplinked

The MER rovers were launched knowing the software would be replaced. The flight software needed for entry, descent, landing and initial surface operations was not ready at launch, so a full replacement was uplinked during cruise and completed in December 2003, about a month before landing [1]. A second full load followed in April 2004 at the end of the prime mission, and a patch in February 2005 [1].

The image lives in two copies. The RAD6000 card holds the CPU, 128 MB of DRAM and one 3 MB EEPROM bank; the non-volatile memory card holds two 4 MB EEPROM banks and 256 MB of flash [1]. The A image occupies part of bank A EEPROM plus flash, the B image part of bank B plus flash, and the two are not required to be identical; for most of the surface mission they held different versions [1]. Each image is a single binary containing a boot loader plus the software; the boot loader specifies the image order to fetch from and moves to its next selection if the first fails to boot and initialize. On a cold boot the initial image is chosen by a ground-commandable latching relay; on a warm boot the flight software picks the order.

Full load products are the new image split into sequential data files plus control files carrying the assembly order, a checksum and identification data. Patch products are the difference between the new image and an image containing an earlier version, plus control files with the instructions and locations for applying them [1]. This is the same mechanism used to fix the Pathfinder priority inversion in 1997: differences between the onboard copy and the intended copy were uplinked and applied by custom onboard software with validation, rather than typed into the VxWorks shell, although the shell was available [5].

The operation is split into Uplink Days and a Build Day, all planned with no other activities running in parallel [1]. Uplink used X-band exclusively rather than UHF relay: the extra per-file overhead, the roughly 15 minute Odyssey pass and the approximately 1000 bit/s uplink rate to the orbiter more than offset the higher relay rate. The high gain antenna was used for its higher uplink rate, which constrained rover orientation to avoid mast and deck occlusions, antenna hard stops and the one minute communications loss of a 180 degree azimuth flop, and constrained mast shadowing of the antenna that cuts warm-up temperature margin and can stall the drive motors mid-pass [1]. Uplink Days used receive-only Direct from Earth passes of 3.5 to 4.5 hours, affordable because not transmitting saves 50 W, at the cost of no confirmation of receipt until the UHF pass at the end of the sol [1].

The Build Day sequence is conditional, so the ground is not in the loop between steps. One-way light time to Mars is 3 to 23 minutes, averaging 10, so each telemetry, decision and command cycle costs at least 30 minutes [1]. The sequence validates that the stored images are not corrupted, builds the new image in DRAM (directly from the uplinked files for a full load, or by copying an existing image and modifying it for a patch), saves it to the designated non-volatile location, and terminates at the first step that fails. The relays are then set so the next cold boot runs the new version, a shutdown is commanded to force that cold boot, the rover wakes 15 minutes later, and a Direct to Earth pass confirms it [1]. Every MER load and patch executed correctly on the vehicle, after test campaigns in which nominal runs were hard to complete cleanly and most contingency procedure exercise happened as a side effect of test anomalies.

MSL: the update history and what a full release costs

Release	First use	Date	Type
R9.4.7	sol 1	Aug 2012	full update, cruise
R10.5.7	sol 5	Aug 2012	full update, cruise
R10.5.8	sol 217	Mar 2013	cold patch
R10.6.4	sol 264	May 2013	full update, surface
R11.0.4	sol 446	Nov 2013	full update, surface
R11.0.5	sol 772	Oct 2014	cold patch
R12.0.3	sol 875	Jan 2015	full update, surface
R12.0.4	sol 2808	Jun 2020	cold patch, prime computer only
R-Hope	sol 2960	Dec 2020	full update, backup computer only

Source: [3], Table 1. As of sol 3610, October 2022, Curiosity had taken 13 distinct hot patches installed more than 57,000 times, and seven cold patches [2]. Release binaries are about 10 MiB: R12 was 9.97 MiB in 49 files, R-Hope 9.97 MiB in 49 files, R13 10.07 MiB in 51 files, the file count set by orbiter file size constraints [3].

R12 was installed on sol 875 when the rover had driven 9.8 km and climbed 62 m above the landing site [3]. By sol 3610 it had driven 28.998 km and climbed 692 m, all of it on the same full release plus patches [2]. One of the fixes that shipped as a hot patch on the running release, rather than waiting for the next full load, corrected a defect in the arm’s generalized inverse kinematics solver: unit testing found that 97.2 percent of commanded poses resolved to within 5 mm of the intended position, but 113,494 of 3.7 million test cases missed by more than 5 mm, 16 by more than 20 cm, and the worst by 1.21 m, a bad solution about once every 144 calls [7]. The defect had been present for roughly eight years of flight without being observed, because only 2,769 arm commands had ever exercised the affected code path; the fix, patched onto sol 2642 and in nominal use two sols later, included simply enabling root-polishing code that had shipped disabled since launch [7].

R13, proposed in April 2016 and kicked off in early 2017, took until September 2022 to be approved for flight [3]: 56 approved bug fixes and 53 new features, roughly 70 distinct approved changes surviving triage, 27 developers mostly borrowed from other projects, and a 2.25 year test campaign. The January 2017 cost estimate offered two options: seven change requests for 462,000 dollars, or 18 change requests for 659,000 dollars [3]. Cost per change request was lower for the larger option because several changes fell in the same module and the cost of regression testing a module does not depend on how many changes it contains.

The R-Hope campaign: updating a computer with no working file system

On sol 200 a hardware failure in the upper bank of the NAND flash on Curiosity’s A-side rover compute element made that string unusable as prime, and the rover was swapped to RCE-B, where it stayed for 5.5 years and almost 2000 sols [3]. On sol 2172 a separate flash problem on RCE-B forced a swap back to RCE-A, which ran prime until further failures reached the lower NAND bank on sol 2339 and the rover swapped to RCE-B again [3]. R-Hope was installed on sol 2960 and left the prime computer on cold-patched R12 and the backup on R-Hope, a split the project carried until R13 [2].

R-Hope is a branch of R12.0.3 that removes RCE-A’s dependence on NAND entirely, relocating the file system into the NOR flash. Each compute element has 4 GB of NAND and 128 MB of NOR; the NOR normally holds up to four redundant copies of the flight software in two ground- selectable groups [3]. R-Hope leaves Group B holding two redundant copies of the software and repurposes Group A as the file system, which gives it 64 MB where NAND gave about 2 GB, a reduction of about 99 percent in file system volume [3]. The resulting layout is 30 MB of data products, 22 MB of engineering partition and two 6 MB non-volatile parameter memory copies. R-Hope is defined as a lifeboat: able to take over as prime, diagnose the other compute element, and install flight software on both, with telecommunications, power monitoring, thermal control, safing and basic system functions, but not the science instruments, the engineering cameras, mobility or the arm.

The install could not use the normal staging path, which copies compressed uplink files to the target computer’s engineering partition and assembles them there, because powering on RCE-A’s NAND was itself judged risky. The prime computer verified and assembled the files instead [3]. Two standard safeguards were also unavailable: the toe-dip, in which a new image is booted once without being burned to non-volatile memory, is done only on the prime computer and was not performed for R-Hope; and because R-Hope consumes one of the two NOR groups, the usual guarantee that the previous known-good version stays onboard could not be kept, so the installation had to overwrite the executing R12.0.3 in Group B. There was no capability to checksum the full copied image, so strategic segments of RCE-A NOR were read back instead. Formatting is not a command but a consequence of setting volatile relays and forcing a power-on reset, and NOR is slow enough that a full set of partition reformats that takes under 5 minutes on NAND takes considerably longer [3].

The 49 files were uplinked over six weeks, 24 September to 31 October, split across three paths: 19 files, 39 percent, through MAVEN; 11 files, 22 percent, through the Mars Reconnaissance Orbiter; and 19 files, 39 percent, by X-band direct from Earth [3]. Forward link sessions carried 3 to 4 files each. Files uplinked through MRO are deleted automatically once forwarded, files staged on MAVEN are not [3].

Ingenuity: updating an aircraft through a rover

Ingenuity’s largest post-landing software update, Release 8.0, was installed in October 2022 and changed both onboard computers: the navigation computer, which performs image processing, and the flight computer, which performs real-time flight control [4]. The release bundled new flight capabilities, operations improvements, additional flight telemetry and bug fixes, and it enabled most of the flights the helicopter made afterward. Two capabilities drove it. A landing hazard detection and divert algorithm examined the visual navigation tracking features in each navigation camera frame and selected the part of the frame with the fewest features and least texture as a landing target, generated at 3 Hz by the navigation computer, so the vehicle could divert away from rock hazards instead of being restricted to airfields known to be clear [4]. And digital elevation maps were incorporated into the vision navigation filter, replacing the flat-plane assumption that had been introducing tens of meters of navigation error over sloped terrain. Smaller changes added preflight telemetry checks that let operators confirm battery voltage and temperature and abort before an unsafe flight, a new telemetry packet level exploiting a higher-throughput radio mode to the helicopter base station on Perseverance, and memory for 15 rather than 10 return-to-Earth color images per flight [4].

The new capabilities were not trusted until they had been exercised on Mars. Flights 34 through 39 were flown as a surface commissioning campaign [4]. Flight 34 was a popup with hazard detection and digital elevation map processing enabled, which demonstrated that the flight software still met all execution deadlines under the added processor load. Flight 36 was an out-and-back over hazardous terrain with the hazard algorithm enabled but the final divert disabled, so the selected targets could be compared against the terrain without risking the landing [4]. Flight 37 landed at that site and diverted to a target the algorithm selected in flight, validating hazard mitigation for operational use, and flight 39 was an out-and-back over a 15 m hill, the largest relief attempted in a single flight, to quantify the navigation accuracy the digital elevation map filter provided [4].

Commissioning did not catch everything. During flight 42 Ingenuity did not divert as commanded: the flight computer rejected the landing target as beyond the 25 m maximum divert distance because the target it held was stale [4]. Landing sites are delivered sporadically inside the 500 Hz state packets the navigation computer sends the flight computer, and the flight computer, following an assumption that predated Release 8.0, discarded the older of two packets received in a single processing cycle in favor of the newer one. The landing sites were consistently arriving in the discarded packet [4]. The fault was reproduced on the Ingenuity testbeds and Release 8.1, delivered in March 2023, patched the flight computer to check every packet in a cycle for landing sites; hazard avoidance has worked consistently since.

What the record says about fallback

Every mission here keeps at least two bootable images and can fall back either autonomously or by ground command, and every mission has found that the fallback guarantee weakens exactly when it is most needed. MER’s boot loader walks the image list on initialization failure [1], but the Spirit flash anomaly showed a fault that reproduced identically in both images because it lived in the file system state rather than in the code [6]. MSL’s toe-dip exists to fly a new image once before committing it, and was skipped for R-Hope because the target was the backup computer [3]. Cassini avoided the question by nearly never flying a full load: after the pre-launch software was set, a full load forces a computer reset that the AACS team judged worth avoiding for 2.5 years of prime mission, relying instead on memory-write patches to change parameters [8]. A 2026 survey of post-launch software adaptation across eight planetary missions, including Galileo’s data-compression reprogramming and Deep Space 1’s camera-as-star-tracker reprogramming, lists onboard software reconfigurability as one of six recurring enablers that let ground teams rescue or extend a mission, alongside redundant hardware degrees of freedom and design margin, but does not attempt to quantify how often the fallback path itself was exercised across those cases [10]. The images being replaced always have more logged flight time than the ones replacing them.

What is not established

None of the sources held here states a defect escape rate, a measured stability comparison between patched and full-release software, or a margin figure for how close a file system was to its limit before a growth-driven failure like Pathfinder’s or MER’s file system overruns. Update timing is reported only in aggregate: no source breaks out how many hours or sols an uplink campaign actually consumed versus planned, so the claim that shortening the campaign is the biggest available improvement rests on operator judgment rather than measurement [1]. Ingenuity’s account of Release 8.0 gives no processor utilization, margin or memory figure for the added hazard-detection and terrain-mapping load, only that all execution deadlines were met [4]. R-Hope’s account does not state how long the 64 MB file system is expected to support backup operations or characterize the NAND degradation mechanism that forced the move [3]. The 2026 cross-mission survey is a post-hoc narrative coding of interviews and public accounts, not a measurement of how often each enabler, including software reconfigurability, made the difference between recovery and loss [10].