Flight Computers and Real-Time Operating Systems

Every planetary surface vehicle flown by NASA since Mars Pathfinder has run its flight software as a set of preemptive prioritized tasks under Wind River VxWorks on a radiation-hardened PowerPC or POWER derivative. The operating system supplies task scheduling, timers, message queues, semaphores, and a DOS file system library. What it costs is a scheduling model whose worst cases are hard to bound by test, and a set of commercial libraries whose internal behavior the flight team does not own. Both costs have caused loss of vehicle control in flight [8][5].

The processors

	Sojourner	Spirit and Opportunity	Curiosity	Perseverance
Landing year	1997	2004	2012	2021
CPU	80C85	BAE RAD6000	BAE RAD750	BAE RAD750 x2 plus Xilinx Virtex5QV FPGA
Clock	2 MHz	20 MHz	133 MHz	133 MHz x2, FPGA at 22M disparities/s
RAM	0.56 MB	128 MB	128 + 512 MB	128 MB x2 plus 512 MB x2
Non-volatile storage	0.17 MB	256 MB flash	4096 MB flash	4096 + 3072 MB flash

Source: [7], Table 1. MER also carried 11 MB of EEPROM, of which the 3 MB supplied with the RAD6000 board has no error detection and correction [3]. Curiosity and Perseverance carry two flight computers, only one prime at a time, with dual sets of avionics, sensors, heaters and telecommunications paths [6]. Perseverance’s second RAD750 board is the Vision Compute Element, which hosts the Virtex-5 FPGA that runs stereo correlation and visual odometry image processing, six times as many pixels per step as Curiosity in less time [7].

The two RTOS generations differ in size as well as clock. A MER flight software plus VxWorks 5.3.1 image is 10 MB on the 20 MHz RAD6000, against 21 MB under VxWorks 6.7 on the 133 MHz RAD750, with the same 128 MB of RAM and 32 MB of addressable code RAM and, on MSL, an additional 512 MB of SDRAM of which half is a RAM file system [27]. Hardware is also a source of flight anomalies: Curiosity has failed to mount Rover Compute Element flash three times, on sol 200 and sol 2320 for RCE-A and on sol 2172 for RCE-B, with solder joint cracking from thermal cycling the leading theory [28]. The same flash design is carried on Mars 2020, and the paper projects flight fatigue against a Mars 2020 qualification test of one flight-like sample, so the margin is a projection from a single sample [28]. The Mars 2020 Helicopter Base Station is the opposite choice: a Snapdragon 801 running Linux, a commercial part with no radiation hardening and no ECC RAM, which also hosts the onboard global localization [16][31].

What the operating system supplies, and what the software adds

MER flight software is 93 modules and about 300,000 source lines of code, primarily ANSI C with targeted assembly and some C++, excluding the operating system [1]. Inheritance and polymorphism were deliberately not exploited: fully exercised in C++ they were judged to cost code size and to introduce non-deterministic behavior. It was written by one team over 35 months from July 2000, inheriting structure, architecture and source from Mars Pathfinder and the Athena rover software and, less directly, from Deep Space 1, Mars Surveyor 98 and Mars Surveyor 01 [1].

VxWorks contributes tasks, time and timers, math libraries, I/O, files, logging, message queues and semaphores. The real-time clock tick that drives the system clock comes from MER-specific hardware rather than from the computer board clock [1].

Above that, the 93 modules map onto 97 tasks, not one to one: sixteen sequence machine tasks share common source, and thirteen fault protection response tasks live in a single module [1]. Curiosity keeps the same shape, one VxWorks task per module communicating by inter-process communication rather than by shared memory under semaphores [6].

Scheduling

There are no rate groups on MER. A task with a rate requirement subscribes to the timer module, which delivers a message at the required interval; the task’s event loop receives that message and performs the work its current state calls for. Time event messages are usually delivered to a higher-priority queue than commands, so periodic work runs before command processing within a task. Several tasks wait on the 8 Hz time event, and task priority alone decides the order in which they run [1]. The cruise attitude control task carries an 8 Hz rate requirement and also processes commands and produces telemetry in the same context [1].

Tasks that have no rate requirement are entirely event driven and never poll; interrupts produce a message that wakes the target task. Priority follows function: sequence engine tasks outrank image processing tasks because command execution is more urgent than imaging, and both are outranked by the task that controls autonomous communication. Every task has at least one message queue, messages are point to point with no broadcast, and the underlying transport is the VxWorks pipe [1].

The imaging chain shows what priority buys in throughput. MER Imaging Services runs three tasks, Imaging Services, Camera Control and Post Processing, with two request queues per priority level, one for cameras on the pointable mast and one for body-mounted cameras [4]. Acquisition takes many seconds, so a request is handed to Post Processing for compression and product generation as soon as its pixels are read, freeing Camera Control to start the next exposure. A body-mounted camera request that arrives while the mast is still slewing is serviced first, which means the order in which images are actually taken cannot be predicted from the order they were commanded.

Memory discipline

MER makes at most 75 percent of the 20 MHz RAD6000 available to autonomy software, and telemetry processing can reduce that further [2]. Dynamic allocation is strongly discouraged by the coding standards. Non-system RAM is available in dedicated pools of 4 MB, 9 MB and up to ten further 2 MB blocks, and using those 2 MB blocks reduces the memory left for image processing, so their use is discouraged as well [5]. By convention, applications request memory only during initialization and never return it.

The file systems are partitioned by medium, and there are no disk drives [5]:

File system	Medium	Size	Purpose
RAM file system	RAM	4 MB	Temporary storage of uplinked files
Temporary file system	RAM	2 MB	Temporary data product store
FLASH file system	flash	224 MB	Science and engineering data products
Primary sequence	EEPROM	700 KB	Sequence storage
Secondary sequence	EEPROM	700 KB	Redundant sequence storage
Primary downlink	EEPROM	50 KB	Downlink table storage
Secondary downlink	EEPROM	50 KB	Redundant downlink table storage

Source: [5], Table 1.

Watchdogs, resets and boot

Severe errors are detected either inside an application or by a software health function that continuously checks for unresponsive or suspended tasks [3]. Suspension of a task is treated as a condition that should never occur, and the health monitor responds by forcing a reset of the avionics and re-initialization of the flight software [5]. The system response after initialization turns off science instruments and warm-up heaters, reinforces survival heaters on, and deactivates all sequences, while continuing to execute the scheduled communication windows in autonomous shutdown mode [3].

If a severe error is detected during initialization itself, the reset is delayed to give the ground time to intervene. The interval is 15 minutes for the first severe error, 15 minutes for the second, one hour for the third, then the pattern repeats [5]. Commands can be processed and communication windows can occur during the delayed reset period, and the boot logic loads alternating copies of the flight software on successive attempts [5][3].

Recovery below the software level is hardware. The mission clock lives in an FPGA powered directly from the batteries, so time survives a processor power-off; co-located on it is the alarm clock that tells the Battery Charge Board to power the avionics [3]. If the flight software cannot shut down within 60 seconds of a low battery detection, the Battery Charge Board takes the batteries off the bus regardless. A small set of commands is processed entirely in hardware, setting bits in a register the flight software reads during initialization; one of them is crippled mode, which tells the software not to use the flash file system and to create a RAM file system with the same logical device name instead [5].

Failure modes established in flight

Priority inversion, Mars Pathfinder, July 1997. The 1553 bus scheduler task, the highest priority task in the system apart from the VxWorks tExec task, checked each 0.125 s cycle that the data distribution task had completed [8]. The meteorology task, near the bottom of the priority order, called select, which called pipeIoctl, which called selNodeAdd, and was preempted mid-semGive while holding the mutex protecting the select wait list. Medium priority tasks then ran long enough that the distribution task, blocked in pipeWrite on that same mutex, missed its deadline, and the scheduler declared the error and reset the computer [8]. The mutex had been created without the priority inheritance option because Wind River left it out of the default selectLib service for performance. The problem had been seen before landing but could not be reproduced, and was ranked below entry, descent and landing work; it was reproduced in the testbed in under 18 hours once the mission was on the surface, using a trace facility built into the flight software from the outset and left in deliberately, which is the only way the sequence could have been captured [8][9]. The fix set the global selectLib option variable to enable priority inheritance, which changed the behavior of every select semaphore created afterward, not just the one at fault; Wind River assessed the performance impact as minimal and the behavior of select as unchanged given that only one task ever waits on a given file descriptor in that system [8]. The patch was uplinked as a difference against the onboard image and applied by custom onboard software with validation, not typed into the VxWorks shell, although the shell was available.

Memory exhaustion through a file system library, Spirit, sol 18, 21 January 2004. The bundled DOS library builds an interlinked representation of the file system in RAM at mount time and updates it as files change [5]. Deleting a file frees its flash space but never releases the corresponding internal structure, so the memory required is set by the maximum number of files that ever existed in each subdirectory rather than by the number present [5]. Two configuration parameters were wrong: expansion of the library’s 256 KB private area from free system memory was permitted, with no ceiling, and the memory library was configured to suspend a task silently when an allocation could not be satisfied rather than failing the file operation. The first reset came from creating a large number of motion history data products; the second, two hours later, from a high gain antenna encoder calibration that produced more motion history files; the third from the data summary report generated automatically for each communication window. Because the failed allocation happened inside a critical region, the suspended task never released the DOS library semaphore, which blocked the task that reads files for recorded telemetry, the task that pushes telemetry frames into the hardware, and the task that idles the file system during shutdown. The rover therefore transmitted the same transfer frame eleven times, returned no recorded data, and did not sleep. Spirit was reset repeatedly from sol 18 until sol 21, when a hardware crippled mode command let it initialize with an empty RAM file system; the flash was erased in chunks and reformatted on sol 32 and normal science resumed on sol 33 [5].

The consequence for the design is stated plainly by the flight software lead of both missions: know how commercial off-the-shelf code works before flying it, and enforce the design guidelines on it that are enforced on in-house code, in this case the prohibition on allocating system memory after initialization [5][8].

What the redundancy figures inherited from aircraft avionics mean

The 1e-9 that flight computing quotes as a reliability requirement is not a per-hour rate and not a spacecraft number [11]. It is the probability of system failure over a ten-hour civil transport flight, met in the SIFT design by five-fold replication of critical tasks while five processors remain and degrading to three-fold with four or three, and it is computed from an assumed 2e-4 per hour solid processor fault rate, an assumed 2e-3 per hour transient error rate, a 0.36 s reconfiguration time and no intermittent faults modeled [11]. The report states those rates as assumptions and gives determining them empirically as a job for the test program, so the figure is the output of a model whose inputs were unmeasured, against a mission profile of ten hours and ground repair between flights.

Masking a fault in hardware instead costs a fixed factor. The FTMP architecture the same program built triplicates processors, system memory modules, real-time clocks and system bus lines, and packages them as ten line replaceable units of which nine form three triads and the tenth is a spare [12]. Voting happens bit-serially at every receiver with no software involvement, which is what makes masking transparent to the application, and it holds only while the three members of a triad stay in tight synchronism, so the architecture does not survive being stretched over physically separated units. Its fault model is random hardware failure in an aircraft environment; single event upsets, latchup and total dose are outside it [12].

Testing the boot and wake paths

On Curiosity, the flight software reads the last known state of the power, compute, transponder and UHF hardware from non-volatile memory on waking and compares it against expectation, with seven wake types [37]. A test campaign on the Work Station Test Set ran about 1000 nominal cruise configurations plus about 50 dead-prime power module cases and about 50 fault detection cases, of which about 97 percent cleared with no discrepancy; the other 3 percent were hypothesized, not shown, to be simulator limits, the sweep is explicitly not exhaustive, and a large vector takes days on one workstation [37].

Updating the software in flight

The stack has to be replaceable in flight. A full MER flight software load is about 8 MB of uplink product and the February 2005 R9.1 patch was about 2 MB [26]. The procedure uses quiet uplink days with no mobility, arm or significant science, then a build day on which the image is assembled in DRAM, checksummed, saved to non-volatile memory and cold booted after a commanded shutdown and a 15 minute wake [26]. The source gives no uplink durations or defect data, so the science lost to the stand-down is not quantified.

Other real-time operating systems and where they have run

VxWorks is the planetary surface standard, not the only flown choice. RTEMS runs on the SPARC-based Electra baseband processor, a 24 MHz Sparc V7 with 16 Mbit SRAM and 256 Mbit EDAC SDRAM, on which a real-time entry, descent and landing navigation filter was timed on breadboard hardware rather than a flight unit [29]. An independent RTEMS 4.10.2 measurement of context-switch overhead and interrupt latency used an ERC32 starter kit at 12 MHz with an external FPGA timestamping GPIO transitions at 50 MHz; it covers one engineering-sample part, no cache, a non-floating-point build and synthetic task sets, so it does not transfer to other CPUs or clocks [13]. The Ingenuity helicopter used no VxWorks: a Linux navigation computer, a bare metal microcontroller flight computer running a 500 Hz motor and GNC cycle, and the base station on the rover [24]. Background on how these choices were made for earlier computers is in the NASA history of onboard computing from Gemini to Galileo, which describes hardware only and stops in 1988 [22].

Framework families

Two NASA framework families sit above the RTOS. cFS is described in an official 218 slide course: an operating system abstraction layer, platform support package, cFE services and applications [14]. The framework itself holds the lowest software classification under NPR-7150.2C and supplies artifacts to support a Class B mission, and the course carries no footprint, CPU or latency figures, so cFS cannot be sized for a processor from it [14]. F Prime is a component and port architecture with autocoded C++ and an OS abstraction layer; its first flight was ISS RapidScat and its first CubeSat was ASTERIA [23]. Its Ingenuity use is reported by the flight software lead as roughly 60 percent generated code and 20 percent unique, with the metric undefined and no memory or CPU figures given [24]. F Prime has been developed in the open under Apache 2.0 since 2017 [32]. At the small end, cFS was ported to the Dellingr 6U CubeSat, delivered to the ISS in 2017, on a 40 MHz ARM7 with 2 MB of RAM; the cFE and cFS code executed from flash, CFDP file transfer and Housekeeping were omitted, and the flash file system held at most 64 files [36]. Development of such software is increasingly done against simulation: an event-driven environment runs compiled flight binaries with modeled scheduling, I/O and memory, simulated 20.05 h in 9.53 s on average, and found a dynamic-memory exhaustion defect in distributed-spacecraft GNC code before flight [33]. Neither framework is shown to cost less than the other: the F Prime papers compare by listing properties, not by measurement [23][24].

Software size is the driver behind both. The NASA multi-center complexity study recorded flight software growth by year of launch and issued sixteen findings, with the growth read from a chart whose line counts depend on the counting rule [25]. Onboard autonomy adds to the load: the Mars 2020 Onboard Planner is a flight scheduler with 154 verification items across 32 verification activities reported by its own team [18].

Alternatives to a monolithic per-mission stack

The NASA Goddard core Flight System factors the same job into an operating system abstraction layer over VxWorks, RTEMS, FreeRTOS or Linux, a platform support package, and a core Flight Executive providing software bus, time, event, table and executive services [10]. It reached flight on the Lunar Reconnaissance Orbiter in 2009 and the cFE application programming interface has not changed since. Its cost and reuse figures are on the core Flight System page.

Commercial and multicore processors

The flown rad-hard processors are single cores, and the proposals to replace them are mostly unflown. A Goddard and JPL position paper argued for commercial multicore parts over FPGAs and set requirements of at least 40 GOPS and 20 GFLOPS, power scaling at core granularity, homogeneous cores, and partitioning strong enough that a crashed core can be reloaded without affecting the rest; nothing was built or measured [15]. The New Millennium ST8 Dependable Multiprocessor combined a PowerPC 750FX and Virtex-II cluster with a rad-hard controller and fault-tolerant middleware, and was reported at TRL4 with a TRL5 testbed, with no radiation test or flight data [17]. A JPL framework for adaptive fault tolerance on many-core parts, using introspection, contains no fault-injection results or overhead figures [20]. A 2025 benchmark gives the flight baseline on the RAD750 plus Virtex-5 vision compute element: 107 ms for a 1024 by 1024 forward FFT, 153 ms for FFT correlation, and 800 ms and 7520 ms for 400 and 2200 variable GFOLD solves [19]. A Snapdragon VOXL2 took 14 and 27 ms on the FFTs and a 250 times speedup on the large solve, but the paper does not state how the next generation timings were obtained [19]. Ground results on a commercial many-core part exist from JPL. Rock detection from the MER autonomous science code took 10 to 15 s per Navcam image on the flight RAD6000, against 2.9 s on one core of a 750 MHz Tilera TILE64 and 0.3 s on 32 cores, a speedup that flattens near 10 fold, with the authors suspecting memory and cache limits [35]. Software fault mitigation for terrain relative navigation on the same part (error correction on inter-core packets, triple modular redundancy and algorithm-based fault tolerance) was exercised with simulated fault injections only, and the correlation stage needs about 40 cores [34]. What has flown is the software mitigation applied to a commercial part: Perseverance runs localization on the Snapdragon 801 by executing the algorithm twice with a co-processor reboot between runs, comparing the results and checksumming inputs and outputs [16].

Fault tolerance, older designs

The JPL-STAR computer contrasted massive redundancy with selective replacement, and its 1967 paper defers all reliability and coverage results to later publications [30]. The standard fault, error and failure vocabulary comes from a dependability taxonomy that contains no measured numbers and nothing specific to spacecraft [21].

What is not established

Fault protection and RTOS behavior are described, not measured. Curiosity’s engine polls over 1000 monitors at 1 Hz, but no CPU cost, trip counts or response statistics are published [6]. The 1e-9 requirement rests on assumed fault rates [11]. The RTEMS timing result is one part at 12 MHz [13]. The F Prime and cFS comparison, the ST8 and many-core fault tolerance proposals, and the multicore benchmark do not report flight data or a stated measurement setup [14][23][24][15][17][20][19]. No source held here gives VxWorks scheduling margins or CPU utilization for any flown rover.