Onboard Storage

Onboard storage on a planetary rover is flash and EEPROM behind a file system, sized so that a sol’s data products fit until the next relay pass clears them. Both Mars rovers that have run long missions have lost a compute string to it: Spirit was out of normal operation from sol 18 to sol 33 in 2004 because of a defect in the file system library [1], and Curiosity’s A-side computer was taken off prime duty by NAND flash hardware failures beginning on sol 200 [5] and has since lost both flash banks [4].

What is carried

	Spirit and Opportunity	Curiosity	Perseverance
RAM	128 MB	128 + 512 MB	128 MB x2 plus 512 MB x2
Flash	256 MB	4096 MB	4096 + 3072 MB
EEPROM	11 MB, of which the 3 MB on the RAD6000 card has no error detection and correction	128 MB NOR per compute element	not reported

Sources: [6] Table 1 for the RAM and flash row, [2] for the MER EEPROM detail, [4] for the Curiosity NOR figure. On MER the RAD6000 card holds the CPU, 128 MB of DRAM and one 3 MB EEPROM bank, and a separate non-volatile memory card holds two 4 MB EEPROM banks and the 256 MB of flash [3].

The file systems

MER partitions storage by medium rather than by function, and there are no disk drives [1]:

File system	Medium	Size	Purpose
RAM file system	RAM	4 MB	Temporary storage of uplinked files
Temporary file system	RAM	2 MB	Temporary data product store
FLASH file system	flash	224 MB	Science and engineering data products
Primary sequence	EEPROM	700 KB	Sequence storage
Secondary sequence	EEPROM	700 KB	Redundant sequence storage
Primary downlink	EEPROM	50 KB	Downlink table storage
Secondary downlink	EEPROM	50 KB	Redundant downlink table storage

Source: [1], Table 1. The 224 MB flash file system is the visible part of the 256 MB device; the remainder holds the flight software images. Each flight software image is a single binary containing a boot loader and the software, and the A and B images each live partly in an EEPROM bank on the non-volatile memory card and partly in flash [3].

Curiosity’s layout is different. Each compute element carries 4 GB of NAND in two banks plus 128 MB of NOR [4]. The NOR normally holds up to four redundant flight software copies in two ground-selectable groups; the NAND holds the file system, with two parameter partitions and a data products partition [4]. On RCE-B the data products partition is 3936 MB with a 16 MB backup non-volatile parameter memory; on RCE-A, after the upper NAND bank was mapped out, the data products partition is 1888 MB.

The MER flash anomaly

The bundled DOS file system library builds an interlinked representation of the file system in RAM when the file system is mounted, and updates it as files change. Deleting a file frees the space in flash but never releases the corresponding internal structure: the memory required is set by the maximum number of files that ever existed in each subdirectory, not by the number present [1]. MER stores every data product as a file in a subdirectory, with a metadata file alongside carrying collection time and priority, and the data product management software limits the system to 8192 unique products.

Two configuration parameters turned that into a vehicle loss. The library’s private area was initialized at 256 KB with expansion from free system memory permitted and no ceiling, so it grew in 256 KB increments until the more than 4 MB of free system memory was gone; and the memory library was configured to suspend a task silently when an allocation could not be satisfied rather than failing the file operation [1]. Because the failure happened inside a critical region, the DOS library semaphore was never released, blocking the task that reads files for recorded telemetry, the task that pushes frames to the radio and the task that idles the file system during shutdown.

Recovery took 15 sols [1]. A hardware crippled mode command, one of a small set processed entirely by hardware, sets a register bit that the flight software reads during initialization and that tells it to build a RAM file system with the same logical device name instead of mounting flash. That file system is ten times smaller and volatile, but because every application uses the logical name the change is transparent to the rest of the software [1]. On sol 27 the team deleted the obsolete data products and subdirectories left over from the launch software load; on sol 32 they booted into crippled mode, copied the flight software image into RAM as insurance, erased flash in small chunks while verifying both flight software images after each chunk, then commanded a format on the next boot. Normal science resumed on sol 33.

The Curiosity NAND failures

Six months after landing, on sol 200, telemetry reported uncorrectable errors in the NAND flash on the prime compute element. Several flight software tasks had hung, so RCE-A could not shut down for its normal battery recharge session, and the watchdog timers that should have forced a reboot were being reset instead. Continued operation would have browned out the rover in three to six days [5]. Within 16 hours the team bypassed the flight software and commanded a hardware swap to RCE-B, which came up prime and entered safe mode.

The cause was a single chip in the flash array generating errors during erase cycles, attributed to a circuit board connectivity problem or infant mortality of a commercial part. Pre-flight testing had erased the NAND about 12 times against an additional 38 erases after launch, and the part is rated for 100,000 cycles [5]. Recovery segregated the bad memory and hardware reset RCE-A to run with a half-size flash file system, which the mission absorbed because the data storage volume had been sized with substantial margin. A maximum up-time watchdog was added to the flight software. The swap left the rover single string 35 days before a solar conjunction during which it would not be commandable for 25 sols, with the backup carrying the same software and the same flash design [5].

That was not the end of it. RCE-A remained the backup for 5.5 years and almost 2000 sols until an unrelated flash problem on RCE-B on sol 2172 forced a swap back; RCE-A then ran prime until further failures reached its lower NAND bank on sol 2339 and the rover swapped to RCE-B again, where it has stayed [4]. The R-Hope software release exists to make RCE-A usable without any NAND at all, relocating the file system into 64 MB of NOR, about 1 percent of the NAND volume it replaces. That relocation costs performance as well as capacity: a full set of partition reformats that takes under 5 minutes on NAND takes considerably longer on NOR [4].

Capacity against daily volume

MER’s flash file system holds 224 MB, about 1792 Mbit [1], against an average relay return of about 56 Mb/sol per rover through Odyssey [8], so the vehicle can retain roughly a month of returnable data. The binding constraint is the 120 Mbit per rover allocation in Odyssey’s own memory and the pass schedule rather than the rover’s storage [8]. What actually filled the file system was the count of files rather than their size: the anomaly was triggered by motion history data products and a per-communication-window data summary report, not by image volume [1].

Compression is what keeps the ratio manageable. As of 7 February 2004, MER had downlinked 5132 ICER-compressed regular images totaling 2867.9 Mpixel in 387.3 MB, an average 1.13 bits/pixel against a 12 bit/pixel source [7]. A further 6075 thumbnails, 64x64 pixel averages of the full frames, took 3.8 MB at 1.27 bits/pixel, about 0.005 bits per pixel of the original image.