Fault Protection

Fault protection is the onboard logic that detects an anomalous condition and puts the vehicle into a state from which the ground can recover it.

The problem

Fault protection has to act without the ground, because light time is often longer than the time to criticality on a deep space mission, and because safe mode is not a usable response during orbit insertion or landing, where entering it forfeits the event [17]. On a rover the cost of a false trip is measured in sols of lost driving, and the cost of a missed detection is the vehicle [4][2]. The environment adds constraints: the rovers must hold thermal stability and energy balance between once-per-sol command opportunities [2], and Curiosity is power-negative whenever its prime computer is awake, so a safe state has to be a schedule of sleep and communication rather than a quiescent configuration [3].

The vocabulary comes from the dependability literature: a fault is the cause, an error the corrupted state it produces, and a failure the deviation of delivered service, and fault tolerance divides into error detection and recovery [18]. The lineage of the flown approach starts with the JPL-STAR computer of the 1960s, a laboratory breadboard that contrasted massive redundancy with selective, replacement redundancy and defined detection, isolation and recovery as separate functions; it was never flown [19]. Growth in flight software size is the pressure behind the designs that followed. The NASA study of flight software complexity found that fault protection and autonomy received little attention in education and recommended treating them as architectural rather than add-on concerns [20].

What has flown

Monitor and response engines: MER, MSL, Mars 2020

Distributed monitors report to a central engine that activates hard-coded responses by priority. MER put autonomous behaviors into flight software logic rather than command sequences, which removed the separation between nominal operation and fault protection [10]. MSL and Mars 2020 flew the engine pattern [1][9], and Psyche inherits it from Soil Moisture Active Passive with one nominal mode and three tiers of safe mode, where fault protection may demote modes and only the ground may promote them [12]. It is the only family with surface flight history on a rover.

The MSL implementation separates detection from action [1]. A monitor carries a persistence count, so a single erroneous reading does not provoke a response, and passes through four telemetered states: black for no data yet, green for good data, yellow for erroneous data persisting long enough to run a local response, and red for erroneous data persisting long enough that the system engine must act. Once red, a monitor latches red until it is reset, even if good data arrives. The engine polls once per second: it checks which monitors have just turned red, looks up the associated responses in a hard-coded table, runs the highest priority one (aborting a lower priority response at a step boundary), and on completion clears every monitor associated with that response [1].

Latching is what makes polling sound: a monitor that could move between red and green on its own would let a fast transient slip between polls [1]. The cost is that the engine acts on the first highest priority response it examines, not the monitor that tripped first chronologically, which is more likely to be the root cause. Priority is hard-coded and ordered by hardware dependency, with flight software examination order as the implicit tiebreak, and aborts are allowed at any step boundary, accepting that the full space of abort locations cannot be tested [1]. The ground can disable the response tied to a specific monitor, but cannot disable a response outright, and responses can dead-end themselves after a set number of runs [1]. MER responses likewise drove the vehicle to a single safe configuration rather than a per-fault state [2].

Safe state design

MER autonomous shutdown puts the CPU to sleep when no communication window is active and the solar array cannot support the load. Wakeup comes from the alarm clock in the always-powered FPGA, or from solar wakeup declared after array current exceeds 2.0 A for more than 10 minutes and at least 16 hours have passed since the last one [2]. Resets, thermal faults, power faults and loss of uplink all use this path, and the flight software arbitrates between them: a low battery event shuts the rover down immediately, and uplink loss begins executing UHF windows only after several days without ground contact [2].

Curiosity’s safing interleaves sleep that recharges the batteries with communication windows timed to orbiter overflights, stored in non-volatile memory on both computers so safing survives an autonomous computer swap, plus a hail mode in which the Electra-Lite radio stays powered and wakes the rover on an orbiter hail [31]. In safe mode the rover follows system fault protection windows, a preset schedule of antenna, telecommunication configuration and solar time [3]. X-band windows are receive-only through the low gain antenna at 7.8125 bps, chosen because it works with corrupted mechanism position knowledge; UHF windows through the relay orbiters are the only source of actionable telemetry [3]. The design was optimized for landing-day conditions and needs periodic manual adjustment as Earth rise drifts, and at the time of its publication flight experience was two safing events in 342 sols [31].

Mobility faults

Curiosity’s drive faults are threshold-plus-persistence tests on quantities that visual odometry supplies. Excessive slip faults when a slow limit is exceeded on a set number of contiguous updates [5]. On sol 2434 the visual odometry failure ceiling was set to 10 in anticipation of feature-poor terrain, and the eleventh failure stopped the drive 2.82 m short of predicted odometry. Suspension angle limits fault independently, with an 18 degree limit on the right bogie [6], and on sol 1789 78.3 percent wheel slip in cohesionless sand drove the heading past the yaw limit and stopped the drive early [5]. Traction control introduced exactly one new fault type, a per-drive-step timeout, and one of the first 146 nominal-use drives ended on it: sol 1786 faulted after 15.86 m of a planned 27.9 m when a 32.77 s timeout expired with the right rear wheel on a large rock [6].

Operators trade fault protection against progress when the failure is understood. On Perseverance sol 1347, after 34 visual odometry failures over three drives at Lookout Hill, rover planners disabled the standard visual odometry failure fault protection and hand-sequenced a slip check requiring visual odometry to succeed only once every 20 m, justified by the mechanical specification that the vehicle can extract itself after 20 m of motion and by firm terrain [7].

Flight case histories

Spirit flash memory, sol 18. A flash file system table of contents retained entries for every file that had ever existed and grew unbounded until it consumed all available RAM. The out-of-memory condition silently suspended a critical task, which blocked the shutdown algorithm from powering off the CPU, so the system reset on the delayed reset timer and again on every reboot. Crippled mode, which does not mount flash, stopped the loop. Two weeks after the first symptoms the anomaly team reformatted flash, and a new flight software load fixed the deleted-file representation [2].

Cruise solar flare, 28 October 2003. Both rovers lost attitude knowledge and dropped out of nominal attitude control [2]; Mars Odyssey detected corrupted RAM from the same storm. The alarm clock triggered wakeup and operations commanded both spacecraft back to three-axis control [2].

Opportunity stuck-on heater. A failed-on heater switch on an instrument deployment device joint drew 0.5 A overnight, costing 180 Wh per night [2]. The fix was deep sleep, a flight software change that removes the batteries from the power bus at night [2]. Deep sleep later thermally cycled the miniTES below its allowable limit, which led to a software change that introduced the vulnerability behind miniTES resets, and on sol 628 a dust storm delayed the battery charge board wakeup past a scheduled alarm [4].

Curiosity sol 200. The anomaly cost 16 hours from first notification to recovery of the shutdown capability by hardware commands swapping to the backup computer; a hang prevented autonomous safe mode entry [3].

Perseverance sol 390. A flight software fatal put the rover in safe mode on the first sol of a three-sol weekend plan, precluding drives on sols 391 and 392, with sol 393 spent on recovery and sol 394 on imagery to plan the next drive [8].

Cassini, 2012. The original attitude-control logic could swap thruster branches only once, so a leak on the B branch in a mixed A and B configuration would never be isolated [30]. Version A8.9.0 allows a second swap at tier 7, chosen over two more capable options because it needed the least code change and testing; ground testing found two defects that were accepted rather than fixed, and the software was uplinked in December 2012 with no on-orbit behavior reported [30].

Curiosity drill feed, sol 1536, and InSight mole. The drill feed motor stalled at current limit with no encoder motion. Eighteen months of flight diagnostics and bench work localized a high-friction zone about one third of a motor revolution wide, no root cause was established, and the project drove the feed to full extension while semi-operable, with feed-extended drilling collecting its first sample on sol 2057, 521 sols after the stall [25]. For the InSight HP3 mole stall, a testbed campaign ran from the sol 87 frangibolt firing to the end of recovery at sol 754, with simulant results that the authors state do not replicate Martian regolith [32].

Cost of anomalies

On MER, anomaly recovery consumed less than 5 percent of surface time, about 30 sols out of 700 per vehicle, which the authors attribute to the same operational design that lets a driving error be recovered in a single day [4]. A catalog of 48 major anomalies (four or more consecutive sols of lost progress) across Spirit, Opportunity, Phoenix, Curiosity, InSight and Perseverance puts the average loss at 11 percent of prime mission duration, 6.6 percent over the full mission, with an average anomaly of 15 sols and a P95 of 45 sols [24]. Not every fault reaches the engine: the ordinary MER case is a sequence that did not produce the intended result, handled by replanning in the next tactical plan [4].

What is being tried now

Verification campaigns (ground-tested, applied to flight projects). Exhaustive testing is impossible, so campaigns bound the problem. Mars 2020 derived more than 400 scenario-based test cases from thousands of analyzed faults, prioritized a minimal set of 150, and executed 265 across all test venues using a shared fault injection library and a catalog of expected event records per response [11]. Psyche scoped a few thousand failure modes, about 650 monitors and 185 responses against 176 end-to-end test cases, of which 76 had run when the launch slipped [13]. The extra year found attitude and rate monitors whose thresholds, sized to tolerate slews, were also active during static pointing, and an engine hang during a compute element swap [13]. A parallel failure modes analysis found 147 unrecorded failure modes and flagged about 33 percent of roughly 140 GNC monitors for possible update [12]. Psyche’s payload design is constrained by spacecraft responses limited to power off and wait for the ground; the imager has no internal fault detection, and the DSOC terminal’s FMECA produced about ten monitors that are mostly masked during operations [27].

Plan-level validation (prototype). ARTcritic validates Europa Clipper’s nominal command sequence jointly with its Activity Restart Timeline against injected processor-reset faults, and found latent sensitivities in two scenarios that were fixed by shifting restart transitions. It ran on an excerpt of the spacecraft model with exaggerated resource limits, so no result reflects flight margins [29].

Goal-based and model-based control. The Mission Data System architecture commands goals, constraints on a state over an interval, so a failed goal is contained by the module that owns the parent goal and fault protection is not a separate system [14]. Rasmussen argues the same case from control fundamentals [15]. Model-based diagnosis reasons from explicit component models, with Mars Polar Lander’s premature engine shutdown at about 40 m after a latched leg-sensor transient as the standard motivating case [16]. Remote Agent, which flew on Deep Space 1, carried a mode identification and recovery module in this family [21]. A 2017 JPL panel described system-wide model-based reasoning as the intended successor to symptom-to-suspect rules, with two SBIR tool efforts still in development [22]. For assurance, a JPL task ran over 7 million archived ASTERIA CubeSat telemetry points through the MONSID diagnostic engine, finding no false detections on the star tracker model and false positives on the coarse sun sensor traced to tuned parameters and an incomplete model; the flight demonstration was a plan, not a result [28]. None of the sources in this family reports a rover flight [14][15][16][22].

Learned anomaly detection (flight data, offline). Undercomplete autoencoders flagged Curiosity straight-drive anomalies from 8 Hz mobility telemetry on sols 3815 to 3934 over 850 m, with 6 and 11 flags for the two variants, none judged hazardous by the mobility team [26]. It runs on downlinked data and is not part of the onboard engine.

Autonomy risk. A risk paper on autonomy software argues that added autonomy brings state-space discontinuity and reduced situational awareness, and proposes a risk identification table without implementation or flight data [23].

The numbers

Element	Definition	Count on MSL
Monitor	software that detects an anomalous situation, sited near the module that knows the device	over 1000 [1]
Local response	code inside the monitor’s module, correcting the issue with a small part of the system	not enumerated [1]
System response	code invoked by the engine, typically calling multiple modules	39 [1]

Epoch	Safe mode communication per sol	State of charge gain per sol
At landing	9 Martian hours [3]	22 percent [3]
As of 2023	6.6 hours [3]	5.5 percent [3]

Campaign or catalog	Figure
Mars 2020 test cases identified, minimal set, executed	over 400, 150, 265 [11]
Psyche end-to-end test cases, run at launch slip	176, 76 [13]
MER time in anomaly recovery	under 5 percent, about 30 of 700 sols [4]
Major Mars surface anomalies, mean and P95 duration	48, 15 sols and 45 sols [24]
Time lost, prime mission and full mission	11 percent and 6.6 percent [24]
Curiosity sol 200 recovery of shutdown capability	16 hours [3]
Drill feed stall to first feed-extended sample	521 sols [25]

What is not established

The rover engine papers are design descriptions. The MSL paper gives no count of monitors that tripped in flight, no response statistics and no CPU cost for polling over more than 1000 monitors, and its claim that the engine is reusable with table changes is an assertion [1]. The MER fault protection paper gives no verification data, monitor count or detection latency [9]. The 5 percent recovery figure for MER is one aggregate with no definition of a recovery sol, reported by the team it measures [4]. The 48-anomaly catalog is a linear fit through six heterogeneous missions dominated by MSL, with no uncertainty on any rate [24].

The validation papers report effort, not adequacy: no coverage fraction against the fault population exists for Mars 2020 or Psyche [11][12][13], and Psyche’s payload paper reports no latency, test counts or defect counts [27]. The drill feed root cause was never established [25]. No source compares a goal-based or model-based design against a monitor and response design on the same anomaly, and neither has a rover flight record [14][15][16]. Fault detection coverage, false trip rates and time to criticality for any flown rover are not published in the sources cited here.