Flight Software Verification

Flight software verification is a ladder of test venues ordered by fidelity and inversely by throughput. The bottom rung is a workstation running the flight software against simulated hardware; the top is the flight vehicle itself. The engineering question at every rung is which defect classes it can find and which it structurally cannot.

The ladder, and what each rung costs

MSL’s internal testing described four stages: unit tests, the Work Station Test Set, System Integrated Test, and Assembly Test and Launch Operations on real spacecraft hardware, with cost rising and availability falling at each step [4]. The Work Station Test Set runs the flight software under a VxWorks simulator on a Linux workstation and is described as cheap relative to every other method.

Venue	Fidelity	Speed	Mission
Work Station Test Set (WSTS)	flight software against modeled avionics, no real hardware	2 to 10 times real time depending on scenario complexity [3]	MSL
WSTS, Mars 2020 build	same, enhanced	6 times real time [5]	Mars 2020
SSim	flight software in the loop with rover mechanisms and terrain	up to 1000 times faster than the flight compute element [1]	MSL, Mars 2020
Mission System Testbed (MSTB)	hardware in the loop with two Rover Compute Elements, no actuators [5]	real time	Mars 2020
Vehicle System Testbed (VSTB)	near-duplicate rover [5]	real time	Mars 2020
Integrated Test Laboratory (ITL)	system-level, mixed real and simulated units [6]	real time	Cassini

The workstation simulator earns its place on usage, not on fidelity. MSL’s WSTS was first deployed in October 2006 and had reached its thirtieth release by 2008, by which point its monthly usage peaked near 3500 hours and it averaged an order of magnitude more use than the project’s hardware-in-the-loop testbeds [3]. It models every hardware behavior visible to the flight software as finite state machines, integrating differential equations where the dynamics require it, and supports fault injection at either the functional level or by setting individual bits, along with pause, resume and single step.

Mars 2020 built the Surface System Development Environment for the same reason inverted: Rover Compute Elements and motor controller assemblies were scarce and shared across subsystems, and the physical test venues, running three shifts a day seven days a week, still could not absorb the demand [2]. The sampling and caching subsystem alone has 19 actuators and over 100 sensors, including strain gauges, resolvers, temperature sensors and contact switches. The environment presents one interface to the flight software over three interchangeable backends: an abstraction layer giving deterministic force and motion simulation, a terrain simulator with slip modeling, visual odometry, pose tracking and rendered camera imagery, and a motor server that drives real testbed hardware over TCP/IP. Substituting the backend rather than the test case is what lets the same test run in software and on hardware.

What simulation found that nothing else would have

Two flight-software state interactions on MSL are the documented catches. In one, a MastCam focus mechanism lock was not properly monitored before a motion command, and a visual odometry command unconditionally reinforced that state even though it was not a drive command; SSim caught the improperly set focus lock that would have failed the entire drive, and SSim was extended to model MastCam as a result [1]. In the other, SSim was used to model ChemCam laser targeting in terrain-relative frames to check the beam against the rover body for self-intersection.

These are combinatorial state defects, and they are found in simulation because simulation is where the combinations can be run. MSL flight software carries more than 4200 commands with dozens of arguments each, 54,000 parameters and tens of thousands of state variables [1]. The arm at Rocknest alone executed 2404 moves, against 2303 for Opportunity over its entire Victoria Crater journey, and self-collision checking requires at least 3 mm of margin.

Hardware in the loop found a different class. Mars 2020 ran the development environment against brassboard hardware at three venues from 2017 and 2018, the docking testbed with robotic arm and dock hardware, a coring software venue with the corer, and a multi-station testbed with the caching assembly, which gave the flight software early runtime against real hardware at a point in the project when changing the software and the algorithms was still comparatively cheap [2]. Engineering model testing of the arm ran August 2018 to January 2019 covering requirements, preloading, docking and fault protection, engineering model caching assembly testing followed from December 2018, and the flight model caching assembly was run in a Mars-like thermal vacuum chamber in late 2019 and early 2020.

What the defect record says the testing misses

The one quantitative study of the question analyzed 199 critical post-launch anomalies across seven spacecraft, Galileo, Mars Global Surveyor, Cassini and Huygens, Deep Space 1, Mars Climate Orbiter, Mars Polar Lander and Stardust, using orthogonal defect classification: when the defect occurred, the trigger condition that surfaced it, the high-level entity fixed, and the actual fix [7]. Two results were unexpected to the authors. Outdated or missing procedures were involved in one fifth of the critical incidents, so the fix was to documentation rather than to code. And requirements changes were rarely because a previous requirement was wrong; they were new requirements for handling rare events or for compensating for hardware failures and degradation that had not been anticipated.

The same study found more problem reports classified as “nothing fixed” than expected. Those false positives, where test personnel believed the flight software was wrong when it was correct, turned out to forecast where operational personnel would later be confused: 311 developmental problem reports from MER avionics integration and test and assembly test and launch operations between April 2002 and January 2003 were analyzed on that basis, and the reports identified which features needed targeted training and documentation before launch [7]. The finding is about the operators rather than about the code.

What a fault injection campaign can and cannot report

The reference measurement on fault detection and recovery in a fault-tolerant flight computer is the FTMP engineering model campaign: 21,055 pin-level stuck faults injected on eight card types in one of ten line replaceable units, of which 17,418 were detected, every detected fault correctly identified and recovered by purging the module and taking a spare or degrading, and an average system reconfiguration time of 82 ms, with per-card averages from 46 ms for the system bus controller to 113 ms for PROM [9]. The floor under that recovery time is not the voters but the configuration control task running in the 3.125 Hz rate group, which is the same reason a spacecraft whose fault response is a scheduled task recovers in tens of milliseconds rather than microseconds.

The report refuses to turn the remaining 17.3 percent into a coverage figure, and the refusal is the transferable part: unused gates inside a package and unused signal pins are undetectable and also harmless, no classification separating them from real misses was done, so no coverage number is stated at all [9]. Two card types are missing from the data set because the only socketed CMOS sample was destroyed in handling, everything was injected in one unit of ten, and a stuck pin is a hardware failure model rather than an upset, a latch that nothing monitors, or a common-mode design fault.

Verifying software whose behavior is not enumerable

The Mars 2020 onboard planner is the hardest case in the record because its output depends on the state the rover happens to be in. The verification campaign ran in spring 2022, after more than a year of Perseverance surface operations under the master and submaster scheme, and drew its scenarios from hundreds of sols of real planning [5]. Requirements were managed in DOORS Next Generation with a matrix linking each requirement to the verification activities providing its evidence, test setup and configuration were standardized in Jupyter notebooks, and the bulk of execution ran on WSTS at 6 times real time, with a lightweight planner simulator for cheaper cases, the Mission System Testbed for performance and thermal fidelity, and the Vehicle System Testbed for end-to-end runs. The planner entered primary operations on 5 October 2023 [5].

What the campaign explicitly could not cover is stated: the unbounded problem space of conditions the planner will meet in flight, which had to be constrained to a finite set of testable units; unknown deviations from predicted activity execution; and the behavioral uncertainty the flexible execution scheme introduces [5]. The anomalies closed during testing cluster around heating: managing onboard heating is difficult because temperatures must be monitored before heating starts, the thermal tables carry uncertainty, and heating can span a sleep cycle. The planner runs on a 133 MHz RAD750 rated at 266 MIPS, shared with the rest of the flight software [5].

Regression as the dominant cost after launch

Cassini’s methodology is the long-duration case: attitude control flight software implementation began in 1993, the launch load completed in 1997, and the prime mission ran from July 2004 [6]. Pre-launch testing was a ten-step sequence from requirements trace matrices through test plans, case design, test trace matrices, procedures, a readiness review, execution, anomaly reporting, analysis and reporting. Unit tests verified functions, modes and states and boundary conditions with interfaces stubbed; an integration baseline was configuration-controlled at the end of each phase; and acceptance testing verified the requirement test matrix with scenario and functional tests covering whatever the matrix did not.

The test environment was tiered like MSL’s but with real units mixed in. The flight software development simulator ran faster than real time on the same UNIX platform as the software; the subsystem test bench, and after launch the Cassini attitude control test station, ran real time with two flight computers, two inertial reference units, one real reaction wheel assembly plus simulated ones, and a simulated power subsystem; and the Integrated Test Laboratory ran system level in real time with two flight computers, two engineering flight computers, two stellar reference units, two Sun sensor assemblies, two main engine valve drivers, five engine gimbal actuators of which two were real, and two solid state recorders [6].

Each critical event got its own build and its own campaign: the A7 cruise build, then A8.6.7 for Saturn orbit insertion, A8.7.1 for probe release and relay, and A8.7.2 and its variants for the remainder of the prime mission [6]. Operations testing over the first two and a half years covered functionality and mission performance, stress testing to find the system’s limits, and review of uplink products and procedures. Full flight software uploads had become virtually non-existent by the prime mission, and the operating philosophy was conservative: do not change logic unless necessary. The same instinct governs the Mars 2020 phased delivery of onboard planning authority [5].

Testbeds that are not computers

The Phoenix lander’s Payload Interoperability Testbed at the University of Arizona held a near-duplicate lander and arm and served a purpose no software simulator does: image comparison [8]. Before a sample delivery, the scoop was positioned over the selected instrument inlet port, a Robotic Arm Camera image was taken, and it was overlaid on an image of the same pose taken in the testbed, with positioning corrected from the difference. The testbed image is the reference the flight image is judged against, and no model substitutes for it because the discrepancy being looked for is exactly what the model does not know.

What each rung catches and misses

Method	Catches	Structurally misses
Workstation simulator	flight software logic, state machine interactions, fault responses to injected faults [3]	data content of instruments it models only in format [3]; real-time behavior, since it deliberately runs off real time [3]
Flight software in the loop with mechanism models	command sequence errors, state interaction defects, self-collision [1]	forces it averages from historical data rather than models [1]; instrument power, thermal and telecom behavior [1]
Hardware in the loop, brassboard	interface and timing behavior against real actuators early enough to change the algorithm [2]	microsecond-scale bus responses the simulation coordinates on time slices [2]
Flight-like vehicle	end-to-end behavior [5]	the state combinations that only a real mission produces [5]
Post-launch defect analysis	that one fifth of critical incidents trace to procedures, not code [7]	nothing, but only after the fact [7]