Flight Computers and Real-Time Operating Systems

Every planetary surface vehicle flown by NASA since Mars Pathfinder has run its flight software as a set of preemptive prioritized tasks under Wind River VxWorks on a radiation-hardened PowerPC or POWER derivative. The operating system supplies task scheduling, timers, message queues, semaphores, and a DOS file system library. What it costs is a scheduling model whose worst cases are hard to bound by test, and a set of commercial libraries whose internal behavior the flight team does not own. Both costs have caused loss of vehicle control in flight [8][5].

The processors

	Sojourner	Spirit and Opportunity	Curiosity	Perseverance
Landing year	1997	2004	2012	2021
CPU	80C85	BAE RAD6000	BAE RAD750	BAE RAD750 x2 plus Xilinx Virtex5QV FPGA
Clock	2 MHz	20 MHz	133 MHz	133 MHz x2, FPGA at 22M disparities/s
RAM	0.56 MB	128 MB	128 + 512 MB	128 MB x2 plus 512 MB x2
Non-volatile storage	0.17 MB	256 MB flash	4096 MB flash	4096 + 3072 MB flash

Source: [7], Table 1. MER also carried 11 MB of EEPROM, of which the 3 MB supplied with the RAD6000 board has no error detection and correction [3]. Curiosity and Perseverance carry two flight computers, only one prime at a time, with dual sets of avionics, sensors, heaters and telecommunications paths [6]. Perseverance’s second RAD750 board is the Vision Compute Element, which hosts the Virtex-5 FPGA that runs stereo correlation and visual odometry image processing, six times as many pixels per step as Curiosity in less time [7].

What the operating system supplies, and what the software adds

MER flight software is 93 modules and about 300,000 source lines of code, primarily ANSI C with targeted assembly and some C++, excluding the operating system [1]. Inheritance and polymorphism were deliberately not exploited: fully exercised in C++ they were judged to cost code size and to introduce non-deterministic behavior. It was written by one team over 35 months from July 2000, inheriting structure, architecture and source from Mars Pathfinder and the Athena rover software and, less directly, from Deep Space 1, Mars Surveyor 98 and Mars Surveyor 01 [1].

VxWorks contributes tasks, time and timers, math libraries, I/O, files, logging, message queues and semaphores. The real-time clock tick that drives the system clock comes from MER-specific hardware rather than from the computer board clock [1].

Above that, the 93 modules map onto 97 tasks, not one to one: sixteen sequence machine tasks share common source, and thirteen fault protection response tasks live in a single module [1]. Curiosity keeps the same shape, one VxWorks task per module communicating by inter-process communication rather than by shared memory under semaphores [6].

Scheduling

There are no rate groups on MER. A task with a rate requirement subscribes to the timer module, which delivers a message at the required interval; the task’s event loop receives that message and performs the work its current state calls for [1]. Time event messages are usually delivered to a higher-priority queue than commands, so periodic work runs before command processing within a task. Several tasks wait on the 8 Hz time event, and task priority alone decides the order in which they run [1]. The cruise attitude control task carries an 8 Hz rate requirement and also processes commands and produces telemetry in the same context.

Tasks that have no rate requirement are entirely event driven and never poll; interrupts produce a message that wakes the target task [1]. Priority follows function: sequence engine tasks outrank image processing tasks because command execution is more urgent than imaging, and both are outranked by the task that controls autonomous communication. Every task has at least one message queue, messages are point to point with no broadcast, and the underlying transport is the VxWorks pipe.

The imaging chain shows what priority buys in throughput. MER Imaging Services runs three tasks, Imaging Services, Camera Control and Post Processing, with two request queues per priority level, one for cameras on the pointable mast and one for body-mounted cameras [4]. Acquisition takes many seconds, so a request is handed to Post Processing for compression and product generation as soon as its pixels are read, freeing Camera Control to start the next exposure. A body-mounted camera request that arrives while the mast is still slewing is serviced first, which means the order in which images are actually taken cannot be predicted from the order they were commanded.

Memory discipline

MER makes at most 75 percent of the 20 MHz RAD6000 available to autonomy software, and telemetry processing can reduce that further [2]. Dynamic allocation is strongly discouraged by the coding standards. Non-system RAM is available in dedicated pools of 4 MB, 9 MB and up to ten further 2 MB blocks, and using those 2 MB blocks reduces the memory left for image processing, so their use is discouraged as well [5]. By convention, applications request memory only during initialization and never return it.

The file systems are partitioned by medium, and there are no disk drives [5]:

File system	Medium	Size	Purpose
RAM file system	RAM	4 MB	Temporary storage of uplinked files
Temporary file system	RAM	2 MB	Temporary data product store
FLASH file system	flash	224 MB	Science and engineering data products
Primary sequence	EEPROM	700 KB	Sequence storage
Secondary sequence	EEPROM	700 KB	Redundant sequence storage
Primary downlink	EEPROM	50 KB	Downlink table storage
Secondary downlink	EEPROM	50 KB	Redundant downlink table storage

Source: [5], Table 1.

Watchdogs, resets and boot

Severe errors are detected either inside an application or by a software health function that continuously checks for unresponsive or suspended tasks [3]. Suspension of a task is treated as a condition that should never occur, and the health monitor responds by forcing a reset of the avionics and re-initialization of the flight software [5]. The system response after initialization turns off science instruments and warm-up heaters, reinforces survival heaters on, and deactivates all sequences, while continuing to execute the scheduled communication windows in autonomous shutdown mode [3].

If a severe error is detected during initialization itself, the reset is delayed to give the ground time to intervene. The interval is 15 minutes for the first severe error, 15 minutes for the second, one hour for the third, then the pattern repeats [5]. Commands can be processed and communication windows can occur during the delayed reset period, and the boot logic loads alternating copies of the flight software on successive attempts [5][3].

Recovery below the software level is hardware. The mission clock lives in an FPGA powered directly from the batteries, so time survives a processor power-off; co-located on it is the alarm clock that tells the Battery Charge Board to power the avionics [3]. If the flight software cannot shut down within 60 seconds of a low battery detection, the Battery Charge Board takes the batteries off the bus regardless. A small set of commands is processed entirely in hardware, setting bits in a register the flight software reads during initialization; one of them is crippled mode, which tells the software not to use the flash file system and to create a RAM file system with the same logical device name instead [5].

Failure modes established in flight

Priority inversion, Mars Pathfinder, July 1997. The 1553 bus scheduler task, the highest priority task in the system apart from the VxWorks tExec task, checked each 0.125 s cycle that the data distribution task had completed [8]. The meteorology task, near the bottom of the priority order, called select, which called pipeIoctl, which called selNodeAdd, and was preempted mid-semGive while holding the mutex protecting the select wait list. Medium priority tasks then ran long enough that the distribution task, blocked in pipeWrite on that same mutex, missed its deadline, and the scheduler declared the error and reset the computer [8]. The mutex had been created without the priority inheritance option because Wind River left it out of the default selectLib service for performance. The problem had been seen before landing but could not be reproduced, and was ranked below entry, descent and landing work; it was reproduced in the testbed in under 18 hours once the mission was on the surface, using a trace facility built into the flight software from the outset and left in deliberately, which is the only way the sequence could have been captured [8][9]. The fix set the global selectLib option variable to enable priority inheritance, which changed the behavior of every select semaphore created afterward, not just the one at fault; Wind River assessed the performance impact as minimal and the behavior of select as unchanged given that only one task ever waits on a given file descriptor in that system [8]. The patch was uplinked as a difference against the onboard image and applied by custom onboard software with validation, not typed into the VxWorks shell, although the shell was available.

Memory exhaustion through a file system library, Spirit, sol 18, 21 January 2004. The bundled DOS library builds an interlinked representation of the file system in RAM at mount time and updates it as files change [5]. Deleting a file frees its flash space but never releases the corresponding internal structure, so the memory required is set by the maximum number of files that ever existed in each subdirectory rather than by the number present [5]. Two configuration parameters were wrong: expansion of the library’s 256 KB private area from free system memory was permitted, with no ceiling, and the memory library was configured to suspend a task silently when an allocation could not be satisfied rather than failing the file operation. The first reset came from creating a large number of motion history data products; the second, two hours later, from a high gain antenna encoder calibration that produced more motion history files; the third from the data summary report generated automatically for each communication window. Because the failed allocation happened inside a critical region, the suspended task never released the DOS library semaphore, which blocked the task that reads files for recorded telemetry, the task that pushes telemetry frames into the hardware, and the task that idles the file system during shutdown. The rover therefore transmitted the same transfer frame eleven times, returned no recorded data, and did not sleep. Spirit was reset repeatedly from sol 18 until sol 21, when a hardware crippled mode command let it initialize with an empty RAM file system; the flash was erased in chunks and reformatted on sol 32 and normal science resumed on sol 33.

The consequence for the design is stated plainly by the flight software lead of both missions: know how commercial off-the-shelf code works before flying it, and enforce the design guidelines on it that are enforced on in-house code, in this case the prohibition on allocating system memory after initialization [5][8].

What the redundancy figures inherited from aircraft avionics mean

The 1e-9 that flight computing quotes as a reliability requirement is not a per-hour rate and not a spacecraft number [11]. It is the probability of system failure over a ten-hour civil transport flight, met in the SIFT design by five-fold replication of critical tasks while five processors remain and degrading to three-fold with four or three, and it is computed from an assumed 2e-4 per hour solid processor fault rate, an assumed 2e-3 per hour transient error rate, a 0.36 s reconfiguration time and no intermittent faults modeled [11]. The report states those rates as assumptions and gives determining them empirically as a job for the test program, so the figure is the output of a model whose inputs were unmeasured, against a mission profile of ten hours and ground repair between flights.

Masking a fault in hardware instead costs a fixed factor. The FTMP architecture the same program built triplicates processors, system memory modules, real-time clocks and system bus lines, and packages them as ten line replaceable units of which nine form three triads and the tenth is a spare [12]. Voting happens bit-serially at every receiver with no software involvement, which is what makes masking transparent to the application, and it holds only while the three members of a triad stay in tight synchronism, so the architecture does not survive being stretched over physically separated units. Its fault model is random hardware failure in an aircraft environment; single event upsets, latchup and total dose are outside it.

Alternatives to a monolithic per-mission stack

The NASA Goddard core Flight System factors the same job into an operating system abstraction layer over VxWorks, RTEMS, FreeRTOS or Linux, a platform support package, and a core Flight Executive providing software bus, time, event, table and executive services [10]. It reached flight on the Lunar Reconnaissance Orbiter in 2009 and the cFE application programming interface has not changed since. Its cost and reuse figures are on the core Flight System page.