Onboard Storage

Onboard storage on a planetary rover is flash and EEPROM behind a file system, sized so that a sol’s data products fit until the next relay pass clears them. Both Mars rovers that have run long missions have lost a compute string to it: Spirit was out of normal operation from sol 18 to sol 33 in 2004 because of a defect in the file system library [1], and Curiosity’s A-side computer was taken off prime duty by NAND flash hardware failures beginning on sol 200 [5] and has since lost both flash banks [4].

What is carried

	Spirit and Opportunity	Curiosity	Perseverance
RAM	128 MB	128 + 512 MB	128 MB x2 plus 512 MB x2
Flash	256 MB	4096 MB	4096 + 3072 MB
EEPROM	11 MB, of which the 3 MB on the RAD6000 card has no error detection and correction	128 MB NOR per compute element	not reported

Sources: [6] Table 1 for the RAM and flash row, [2] for the MER EEPROM detail, [4] for the Curiosity NOR figure. On MER the RAD6000 card holds the CPU, 128 MB of DRAM and one 3 MB EEPROM bank, and a separate non-volatile memory card holds two 4 MB EEPROM banks and the 256 MB of flash [3].

The file systems

MER partitions storage by medium rather than by function, and there are no disk drives [1]:

File system	Medium	Size	Purpose
RAM file system	RAM	4 MB	Temporary storage of uplinked files
Temporary file system	RAM	2 MB	Temporary data product store
FLASH file system	flash	224 MB	Science and engineering data products
Primary sequence	EEPROM	700 KB	Sequence storage
Secondary sequence	EEPROM	700 KB	Redundant sequence storage
Primary downlink	EEPROM	50 KB	Downlink table storage
Secondary downlink	EEPROM	50 KB	Redundant downlink table storage

Source: [1], Table 1. The 224 MB flash file system is the visible part of the 256 MB device; the remainder holds the flight software images. Each flight software image is a single binary containing a boot loader and the software, and the A and B images each live partly in an EEPROM bank on the non-volatile memory card and partly in flash [3].

Curiosity’s layout is different. Each compute element carries 4 GB of NAND in two banks plus 128 MB of NOR [4]. The NOR normally holds up to four redundant flight software copies in two ground-selectable groups; the NAND holds the file system, with two parameter partitions and a data products partition [4]. On RCE-B the data products partition is 3936 MB with a 16 MB backup non-volatile parameter memory; on RCE-A, after the upper NAND bank was mapped out, the data products partition is 1888 MB.

The MER flash anomaly

The bundled DOS file system library builds an interlinked representation of the file system in RAM when the file system is mounted, and updates it as files change. Deleting a file frees the space in flash but never releases the corresponding internal structure: the memory required is set by the maximum number of files that ever existed in each subdirectory, not by the number present [1]. MER stores every data product as a file in a subdirectory, with a metadata file alongside carrying collection time and priority, and the data product management software limits the system to 8192 unique products.

Two configuration parameters turned that into a vehicle loss. The library’s private area was initialized at 256 KB with expansion from free system memory permitted and no ceiling, so it grew in 256 KB increments until the more than 4 MB of free system memory was gone; and the memory library was configured to suspend a task silently when an allocation could not be satisfied rather than failing the file operation [1]. Because the failure happened inside a critical region, the DOS library semaphore was never released, blocking the task that reads files for recorded telemetry, the task that pushes frames to the radio and the task that idles the file system during shutdown.

Recovery took 15 sols [1]. A hardware crippled mode command, one of a small set processed entirely by hardware, sets a register bit that the flight software reads during initialization and that tells it to build a RAM file system with the same logical device name instead of mounting flash. That file system is ten times smaller and volatile, but because every application uses the logical name the change is transparent to the rest of the software [1]. On sol 27 the team deleted the obsolete data products and subdirectories left over from the launch software load; on sol 32 they booted into crippled mode, copied the flight software image into RAM as insurance, erased flash in small chunks while verifying both flight software images after each chunk, then commanded a format on the next boot. Normal science resumed on sol 33.

The Curiosity NAND failures

Six months after landing, on sol 200, telemetry reported uncorrectable errors in the NAND flash on the prime compute element. Several flight software tasks had hung, so RCE-A could not shut down for its normal battery recharge session, and the watchdog timers that should have forced a reboot were being reset instead. Continued operation would have browned out the rover in three to six days [5]. Within 16 hours the team bypassed the flight software and commanded a hardware swap to RCE-B, which came up prime and entered safe mode.

The cause was a single chip in the flash array generating errors during erase cycles, attributed to a circuit board connectivity problem or infant mortality of a commercial part. Pre-flight testing had erased the NAND about 12 times against an additional 38 erases after launch, and the part is rated for 100,000 cycles [5]. Recovery segregated the bad memory and hardware reset RCE-A to run with a half-size flash file system, which the mission absorbed because the data storage volume had been sized with substantial margin. A maximum up-time watchdog was added to the flight software. The swap left the rover single string 35 days before a solar conjunction during which it would not be commandable for 25 sols, with the backup carrying the same software and the same flash design [5].

That was not the end of it. RCE-A remained the backup for 5.5 years and almost 2000 sols until an unrelated flash problem on RCE-B on sol 2172 forced a swap back; RCE-A then ran prime until further failures reached its lower NAND bank on sol 2339 and the rover swapped to RCE-B again, where it has stayed [4]. The R-Hope software release exists to make RCE-A usable without any NAND at all, relocating the file system into 64 MB of NOR, about 1 percent of the NAND volume it replaces. That relocation costs performance as well as capacity: a full set of partition reformats that takes under 5 minutes on NAND takes considerably longer on NOR [4].

Redundant compute elements do not synchronize their storage on their own, and that gap is what the sol 200 recovery depended on. Perseverance carries two rover compute elements, each with a prime and a backup nonvolatile engineering partition, so a critical file exists in up to four places, none of them kept current automatically: every parameter change, sequence and file has to be pushed to the backup by ground command, including turning the backup RCE on to apply the same change twice [9]. The MSL precedent this design answers is exactly the sol 200 case: RCE-A’s filesystem became unusable, its data products could not be copied out of volatile memory before the swap, and the rover ran indefinitely on the backup RCE while recovery proceeded, a scenario the mission survived only because RCE-B held a properly formatted, current copy of everything RCE-A needed [9]. The response was a reusable synchronization activity scoped to copy only already-flown, already-reviewed configuration from the prime side, so each run needs a short review rather than a full activity re-certification, at the cost of a recurring operations task that has no automated trigger.

File system performance and the sol 200 fix

Mars 2020 rewrote the MSL-derived file system and parameter management code rather than carrying it forward unchanged. Where MER and MSL data here come from anomaly reports, the Mars 2020 numbers are routine performance figures from the flight record through sol 216 [10]. Surface file system mount rate reached 13.5 to 15.4 MB/s, a 56 to 63 percent reduction in mount time against the MSL baseline measured in lab testing, and background deletes ran at 19.7 plus or minus 3 deletes per second against data product delete sequences that occupied 638 plus or minus 574 seconds per sol, both large enough fractions of the sol that trimming them was worth a dedicated redesign rather than a parameter tweak. Bulk-loading a full parameter set, more than 60,000 parameters from a single file, took about 6 minutes instead of the hours the equivalent MSL operation required, and non-image downlink data was GZIP compressed in flight, removing more than two thirds of the volume over the first 216 sols, 88.8 Gibit reduced to 27.5 Gibit [10].

The rewrite also closed the specific health-monitoring hole behind the sol 200 failure, where several flight software tasks hung and the watchdog timers that should have forced a reboot were being reset instead [5]. MSL’s health service accepted a task’s response to a liveness ping as proof the task was working even while that task’s work queues were disabled, so a hung queue produced no fault indication [10]. Mars 2020’s queue monitoring adds a disable time limit, 15 minutes in the example given, after which a stalled queue itself raises a fault rather than waiting on the task-level ping to fail. File system integrity checks recorded zero failures on the surface through the period reported.

Capacity against daily volume

MER’s flash file system holds 224 MB, about 1792 Mbit [1], against an average relay return of about 56 Mb/sol per rover through Odyssey [8], so the vehicle can retain roughly a month of returnable data. The binding constraint is the 120 Mbit per rover allocation in Odyssey’s own memory and the pass schedule rather than the rover’s storage [8]. What actually filled the file system was the count of files rather than their size: the anomaly was triggered by motion history data products and a per-communication-window data summary report, not by image volume [1].

Compression is what keeps the ratio manageable. As of 7 February 2004, MER had downlinked 5132 ICER-compressed regular images totaling 2867.9 Mpixel in 387.3 MB, an average 1.13 bits/pixel against a 12 bit/pixel source [7]. A further 6075 thumbnails, 64x64 pixel averages of the full frames, took 3.8 MB at 1.27 bits/pixel, about 0.005 bits per pixel of the original image.

What is not established

Every account here is a single-mission case study written by the team that owns the design, not an independent or comparative study: MER’s own flight software team describes the sol 18 anomaly, and neither the memory margin consumed before the failure nor its growth rate is quantified [1]. MSL’s sol 200 record is a secondhand summary of an internal engineering account, with no independent verification of its erase-cycle or thermal figures [5]. The R-Hope relocation to NOR carries no stated reliability argument, no expected service life for the 64 MB backup file system, and no characterization of the NAND degradation as a mechanism or rate [4]. Mars 2020’s mount-rate and delete-throughput gains are stated against an MSL baseline that is itself never given a number, so the improvement fractions cannot be checked independently, and several of the capabilities described had only ground and lab testing behind them at the sol 216 cutoff of the record [10]. The redundant-partition synchronization design has no reported activity durations, data volumes or error counts, only that early estimates were conservative [9]. The ICER compression rates cover about 34 sols of early, flat-terrain imagery under an operator-set byte quota, so they describe that mission phase’s downlink budget rather than a general property of Martian image content or a fidelity claim at any bit rate [7].