Onboard Software Update

Replacing the software on a vehicle that is already flying is a routine capability on Mars surface missions and a dangerous one. Curiosity has taken nine full updates or cold patches and more than 57,000 hot patch installations since 2012 [2][3]. Viking 1 was lost when a software patch permanently overwrote code needed to talk to Earth, and the MSL patch development checklist still asks a cognizant engineer who has never studied that loss to stop and research it before continuing.

Three mechanisms

Method	What it changes	Persistence	Uplink volume
Hot patch	RAM only, applied by an install script at boot	Reinstalled on every boot; skipping the script reverts the vehicle	small
Cold patch	Modifies a copy of the existing image, result burned back to non-volatile memory	Survives reboot	2 MB for MER R9.1, February 2005
Full update or load	Entire new image uplinked and burned to non-volatile memory	Survives reboot	about 8 MB on MER, 9.97 MiB in 49 files on MSL

Sources: [1] for MER, [2] and [3] for MSL. A hot patch is fast to author, review, test and fly, but must be reinstalled at every boot, which is also its safety property: booting without the script gives a clean, unpatched system [2]. A cold patch is more permanent and more dangerous, because memory write commands bypass the built-in checks on data and address validity [3].

Neither method is easy to test. Hot and cold patches are by design not integrated into the flight software monolith, so they cannot be unit tested against the source used to build the unit tests, and several are release-specific and do not work in the development sandboxes. Multiple simultaneous patches have to be tested in operationally realistic combinations, which grows exponentially with the number of active patches [2].

MER: how a load is actually uplinked

The MER rovers were launched knowing the software would be replaced. The flight software needed for entry, descent, landing and initial surface operations was not ready at launch, so a full replacement was uplinked during cruise and completed in December 2003, about a month before landing [1]. A second full load followed in April 2004 at the end of the prime mission, and a patch in February 2005 [1].

The image lives in two copies. The RAD6000 card holds the CPU, 128 MB of DRAM and one 3 MB EEPROM bank; the non-volatile memory card holds two 4 MB EEPROM banks and 256 MB of flash [1]. The A image occupies part of bank A EEPROM plus flash, the B image part of bank B plus flash, and the two are not required to be identical. For most of the surface mission they held different versions [1]. Each image is a single binary containing a boot loader plus the software; the boot loader specifies the image order to fetch from, and moves to its next selection if the first fails to boot and initialize. On a cold boot the initial image is chosen by a ground-commandable latching relay; on a warm boot the flight software picks the order.

Full load products are the new image split into sequential data files plus control files carrying the assembly order, a checksum and identification data. Patch products are the difference between the new image and an image containing an earlier version, plus control files with the instructions and locations for applying them [1]. This is the same mechanism used to fix the Pathfinder priority inversion in 1997: differences between the onboard copy and the intended copy were uplinked and applied by custom onboard software with validation, rather than typed into the VxWorks shell, although the shell was available [5].

The operation is split into Uplink Days and a Build Day, all planned with no other activities running in parallel [1]. Uplink used X-band exclusively rather than UHF relay: the extra per-file overhead, the roughly 15 minute Odyssey pass and the approximately 1000 bit/s uplink rate to the orbiter more than offset the higher relay rate. The high gain antenna was used for its higher uplink rate, which constrained rover orientation to avoid mast and deck occlusions, antenna hard stops and the one minute communications loss of a 180 degree azimuth flop, and constrained mast shadowing of the antenna that cuts warm-up temperature margin and can stall the drive motors mid-pass [1]. Uplink Days used receive-only Direct from Earth passes of 3.5 to 4.5 hours, affordable because not transmitting saves 50 W, at the cost of no confirmation of receipt until the UHF pass at the end of the sol [1].

The Build Day sequence is conditional, so that the ground is not in the loop between steps. One-way light time to Mars is 3 to 23 minutes, averaging 10, so each telemetry, decision and command cycle costs at least 30 minutes [1]. The sequence validates that the stored images are not corrupted, builds the new image in DRAM (directly from the uplinked files for a full load, or by copying an existing image and modifying it for a patch), saves it to the designated non-volatile location, and terminates at the first step that fails. The relays are then set so the next cold boot runs the new version, a shutdown is commanded to force that cold boot, the rover wakes 15 minutes later, and a Direct to Earth pass confirms it [1]. Every MER load and patch executed correctly on the vehicle, after test campaigns in which nominal runs were hard to complete cleanly and most contingency procedure exercise happened as a side effect of test anomalies.

MSL: the update history and what a full release costs

Release	First use	Date	Type
R9.4.7	sol 1	Aug 2012	full update, cruise
R10.5.7	sol 5	Aug 2012	full update, cruise
R10.5.8	sol 217	Mar 2013	cold patch
R10.6.4	sol 264	May 2013	full update, surface
R11.0.4	sol 446	Nov 2013	full update, surface
R11.0.5	sol 772	Oct 2014	cold patch
R12.0.3	sol 875	Jan 2015	full update, surface
R12.0.4	sol 2808	Jun 2020	cold patch, prime computer only
R-Hope	sol 2960	Dec 2020	full update, backup computer only

Source: [3], Table 1. As of sol 3610, October 2022, Curiosity had taken 13 distinct hot patches installed more than 57,000 times, and seven cold patches [2]. Release binaries are about 10 MiB: R12 was 9.97 MiB in 49 files, R-Hope 9.97 MiB in 49 files, R13 10.07 MiB in 51 files, the file count set by orbiter file size constraints [3].

R12 was installed on sol 875 when the rover had driven 9.8 km and climbed 62 m above the landing site [3]. By sol 3610 it had driven 28.998 km and climbed 692 m, all of it on the same full release plus patches [2]. R13, proposed in April 2016 and kicked off in early 2017, took until September 2022 to be approved for flight: 56 approved bug fixes and 53 new features, roughly 70 distinct approved changes surviving triage, 27 developers mostly borrowed from other projects, and a 2.25 year test campaign. The January 2017 cost estimate offered two options: seven change requests for 462,000 dollars, or 18 change requests for 659,000 dollars [3]. Cost per change request was lower for the larger option because several changes fell in the same module and the cost of regression testing a module does not depend on how many changes it contains.

The R-Hope campaign: updating a computer with no working file system

On sol 200 a hardware failure in the upper bank of the NAND flash on Curiosity’s A-side rover compute element made that string unusable as prime, and the rover was swapped to RCE-B, where it stayed for 5.5 years and almost 2000 sols [3]. On sol 2172 a separate flash problem on RCE-B forced a swap back to RCE-A, which ran prime until further failures reached the lower NAND bank on sol 2339 and the rover swapped to RCE-B again [3]. R-Hope was installed on sol 2960 and left the prime computer on cold-patched R12 and the backup on R-Hope, a split the project carried until R13 [2].

R-Hope is a branch of R12.0.3 that removes RCE-A’s dependence on NAND entirely, relocating the file system into the NOR flash. Each compute element has 4 GB of NAND and 128 MB of NOR; the NOR normally holds up to four redundant copies of the flight software in two ground- selectable groups [3]. R-Hope leaves Group B holding two redundant copies of the software and repurposes Group A as the file system, which gives it 64 MB where NAND gave about 2 GB, a reduction of about 99 percent in file system volume [3]. The resulting layout is 30 MB of data products, 22 MB of engineering partition and two 6 MB non-volatile parameter memory copies. R-Hope is defined as a lifeboat: able to take over as prime, diagnose the other compute element, and install flight software on both, with telecommunications, power monitoring, thermal control, safing and basic system functions, but not the science instruments, the engineering cameras, mobility or the arm.

The install could not use the normal staging path, which copies compressed uplink files to the target computer’s engineering partition and assembles them there, because powering on RCE-A’s NAND was itself judged risky. The prime computer verified and assembled the files instead [3]. Two standard safeguards were also unavailable: the toe-dip, in which a new image is booted once without being burned to non-volatile memory, is done only on the prime computer and was not performed for R-Hope; and because R-Hope consumes one of the two NOR groups, the usual guarantee that the previous known-good version stays onboard could not be kept, so the installation had to overwrite the executing R12.0.3 in Group B. There was no capability to checksum the full copied image, so strategic segments of RCE-A NOR were read back instead. Formatting is not a command but a consequence of setting volatile relays and forcing a power-on reset, and NOR is slow enough that a full set of partition reformats that takes under 5 minutes on NAND takes considerably longer [3].

The 49 files were uplinked over six weeks, 24 September to 31 October, split across three paths: 19 files, 39 percent, through MAVEN; 11 files, 22 percent, through the Mars Reconnaissance Orbiter; and 19 files, 39 percent, by X-band direct from Earth [3]. Forward link sessions carried 3 to 4 files each. Files uplinked through MRO are deleted automatically once forwarded, files staged on MAVEN are not [3].

Ingenuity: updating an aircraft through a rover

Ingenuity’s largest post-landing software update, Release 8.0, was installed in October 2022 and changed both onboard computers: the navigation computer, which performs image processing, and the flight computer, which performs real-time flight control [4]. The release bundled new flight capabilities, operations improvements, additional flight telemetry and bug fixes, and it enabled most of the flights the helicopter made afterward. Two capabilities drove it. A landing hazard detection and divert algorithm examined the visual navigation tracking features in each navigation camera frame and selected the part of the frame with the fewest features and least texture as a landing target, generated at 3 Hz by the navigation computer, so the vehicle could divert away from rock hazards instead of being restricted to airfields known to be clear [4]. And digital elevation maps were incorporated into the vision navigation filter, replacing the flat-plane assumption that had been introducing tens of meters of navigation error over sloped terrain. Smaller changes added preflight telemetry checks that let operators confirm battery voltage and temperature and abort before an unsafe flight, a new telemetry packet level exploiting a higher-throughput radio mode to the helicopter base station on Perseverance, and memory for 15 rather than 10 return-to-Earth color images per flight [4].

The new capabilities were not trusted until they had been exercised on Mars. Flights 34 through 39 were flown as a surface commissioning campaign [4]. Flight 34 was a popup with hazard detection and digital elevation map processing enabled, which demonstrated that the flight software still met all execution deadlines under the added processor load. Flight 36 was an out-and-back over hazardous terrain with the hazard algorithm enabled but the final divert disabled, so the selected targets could be compared against the terrain without risking the landing [4]. Flight 37 landed at that site and diverted to a target the algorithm selected in flight, validating hazard mitigation for operational use, and flight 39 was an out-and-back over a 15 m hill, the largest relief attempted in a single flight, to quantify the navigation accuracy the digital elevation map filter provided [4].

Commissioning did not catch everything. During flight 42 Ingenuity did not divert as commanded: the flight computer rejected the landing target as beyond the 25 m maximum divert distance because the target it held was stale [4]. Landing sites are delivered sporadically inside the 500 Hz state packets the navigation computer sends the flight computer, and the flight computer, following an assumption that predated Release 8.0, discarded the older of two packets received in a single processing cycle in favor of the newer one. The landing sites were consistently arriving in the discarded packet [4]. The fault was reproduced on the Ingenuity testbeds and Release 8.1, delivered in March 2023, patched the flight computer to check every packet in a cycle for landing sites; hazard avoidance has worked consistently since.

What the record says about fallback

Every mission here keeps at least two bootable images and can fall back either autonomously or by ground command, and every mission has found that the fallback guarantee weakens exactly when it is most needed. MER’s boot loader walks the image list on initialization failure [1], but the Spirit flash anomaly showed a fault that reproduced identically in both images because it lived in the file system state rather than in the code [6]. MSL’s toe-dip exists to fly a new image once before committing it, and was skipped for R-Hope because the target was the backup computer [3]. The images being replaced always have more logged flight time than the ones replacing them.