Fault Protection

Fault protection is the onboard logic that detects an anomalous condition and puts the vehicle into a state from which the ground can recover it. On a rover the cost of a false trip is measured in sols of lost driving, and the cost of a missed detection is the vehicle [4][2].

Architecture

The MSL implementation separates detection from action [1]:

Element	Definition	Count on MSL
Monitor	software that detects an anomalous situation, sited near the module that knows the device	over 1000
Local response	code inside the same module as the monitor, correcting the issue with a small part of the system	not enumerated
System response	code invoked by the fault protection engine, typically calling multiple software modules	39

Source: [1]. Many of the 1000 monitors share code, particularly the thermal ones.

A monitor carries a persistence count, so a single erroneous reading does not provoke a response, and passes through four telemetered states: black for no data yet, green for good data, yellow for erroneous data persisting long enough to run a local response, and red for erroneous data persisting long enough that the system engine must act [1]. Once red, a monitor latches red until it is reset, even if good data arrives.

The engine itself is a once-per-second poll [1]:

1. Check which monitors have just turned red.
2. Look up the associated responses in a hard-coded table indexed by monitor.
3. Choose the highest priority response as the candidate.
4. If nothing is running, start it; if a lower priority response is running, abort it first.
5. On completion, clear every monitor associated with that response, resetting persistence counts and returning them to black.

The MER engine that preceded it provided arbitration between multiple fault responses in the same way, and MER responses likewise drove the vehicle to a single safe configuration rather than to a per-fault state [1][2].

Latching is what makes polling sound: a monitor that could move between red and green on its own would let a fast transient slip between polls [1]. The cost of polling is that the engine acts on the first highest priority response it examines rather than on the monitor that tripped first chronologically, which is more likely to be the root cause.

Priority is hard-coded and ordered by hardware dependency, so a response that fixes a bus outranks one that fixes a device on that bus [1]. Where explicit priorities tie, the implicit tiebreak is the order responses are examined in the flight software, so predicting which response will run requires knowing both the priority levels and that ordering. Aborts were allowed at any step boundary rather than at designated points, accepting that the full space of abort locations cannot be tested in exchange for reaching a high priority response sooner.

Prioritization and arbitration are not optional on a vehicle that must also keep itself warm and charged: on MER the same shutdown machinery serves the reset, thermal, power and uplink loss responses, so the arbitration rule is what prevents two of them fighting [2].

Two limits exist on the engine’s authority. The ground can disable the response associated with a specific monitor, optionally only in specific spacecraft configurations, but cannot disable a response outright [1]. Responses can dead-end themselves after running a set number of times, and only a ground command undoes it.

Entry conditions and the safe state

MER surface fault protection was shaped by the solar power system: the rovers had to maintain thermal stability and energy balance autonomously between the once-per-sol command opportunities [2]. Arrays supplied about 900 Wh per sol at the start of the mission. Autonomous shutdown puts the CPU to sleep when no communication window is active and the solar array cannot support the load; wakeup comes from the alarm clock in the always-powered FPGA or from solar wakeup, declared by the battery control boards after array current exceeds 2.0 A for more than 10 minutes and at least 16 hours have passed since the last solar wakeup [2]. Responses to flight software resets, thermal faults, power faults and loss of uplink all use autonomous shutdown to reach a thermally safe, power safe, communicative state, and the flight software arbitrates between conflicting responses: a low battery event shuts the rover down immediately, and if the operations team has still not regained control after several days the uplink loss response also begins executing UHF windows.

Curiosity’s safe mode is a scheduled communication configuration rather than a quiescent one, and the schedule is regenerated on the ground rather than computed onboard [3]. In safe mode the rover ignores its nominal communication schedule and follows system fault protection windows, a preset schedule specifying the antenna, telecommunication configuration and solar time of each opportunity. X-band windows are receive-only through the low gain antenna, which supports far lower uplink rates than the 2000 bps available on the high gain antenna: the safe mode configuration uses 7.8125 bps, chosen because it works with corrupted mechanism position knowledge [3]. UHF windows through the Mars relay orbiters are the only source of actionable telemetry and carry the beep that tells the ground the rover is in safe mode at all.

What safing costs

Safe mode on Curiosity is an energy problem because the rover is power-negative whenever the prime rover compute element is awake; the MMRTG output is below the collective load, so the state of charge only rises while asleep [3]. The requirement is that state of charge increase over any 24 hour period.

Epoch	Safe mode communication per sol	State of charge gain per sol
At landing	9 Martian hours	22 percent
As of 2023	6.6 hours	5.5 percent

Source: [3]. The window schedule has been revised repeatedly across the mission, driven by changing uplink strategies, Mars seasons and the declining MMRTG output.

Recovery time is the other cost. On MER, anomaly recovery consumed less than 5 percent of surface time, about 30 sols out of 700 per vehicle, which the authors attribute to the same operational design that lets a driving error be recovered in a single day [4]. The escalation ladder runs from replanning the intended motion, through a recovery plan whose only objective is to collect data, to standing up an anomaly team for a multi-sol investigation.

On Curiosity, the sol 200 anomaly cost 16 hours from first notification to recovery of the shutdown capability, achieved by sending hardware commands to swap to the backup rover compute element; the rover had not entered safe mode autonomously because a hang prevented it, and the safe mode beep arrived only after ground intervention [3]. Perseverance lost three sols to a flight software fatal on sol 390 during the Rapid Traverse Campaign: the fatal put the rover in safe mode on the first sol of a three-sol weekend plan, precluding the sol 391 and 392 drives, with sol 393 spent on recovery and sol 394 on acquiring the imagery needed to plan the next drive [8].

Not every fault reaches the engine. On MER the ordinary case is a sequence that did not produce the intended result, which is handled by replanning the motion in the next tactical plan rather than by fault protection at all; the anomaly ladder only escalates when the analysis suggests a persistent component problem [4]. Tactical plans commonly cover one to three days, so an anomaly that stops execution costs at minimum the remainder of that plan plus a recovery plan whose only purpose is data collection.

Mobility faults specifically

Curiosity’s drive faults are threshold-plus-persistence tests on quantities that visual odometry supplies. Excessive slip uses a slow limit with a persistence number, faulting when the limit is exceeded on that many contiguous updates [5]. Visual odometry convergence failures are counted against a configurable ceiling; on sol 2434 the ceiling was set to 10 in anticipation of feature-poor terrain, and the eleventh failure set a mobility goal error that stopped the drive 2.82 m short of predicted odometry. Suspension angle limits fault independently: the right bogie has an 18 degree limit [6]. Yaw limits fault on heading change, as on sol 1789 when 78.3 percent wheel slip in cohesionless sand drove the heading past the limit and stopped the drive early [5].

Faults are also the record of what the drive modes are for: 88 percent of Curiosity’s driving to sol 2488 used some form of visual odometry, and 5 drives in that period were stopped by visual odometry failures against 94 failures in 20,682 attempts [5].

Adding an autonomy capability adds fault modes. Traction control introduced exactly one new fault type, a per-drive-step timeout, and one of the first 146 nominal-use drives ended on it: the sol 1786 drive faulted after 15.86 m of a planned 27.9 m when a 32.77 s timeout expired with the right rear wheel on a large rock, at which moment the right bogie angle was 0.3 degrees from its own limit [6].

Operators can trade fault protection against progress when the failure is understood. On Perseverance sol 1347, after 34 visual odometry failures over three drives in visually bland terrain at Lookout Hill, rover planners disabled the standard visual odometry failure fault protection and hand-sequenced a slip check requiring visual odometry to succeed only once every 20 m, justified by the mechanical specification that the vehicle can extract itself after 20 m of motion without forward progress and by the terrain being firm [7].

Flight case histories

Spirit flash memory, sol 18. A table of contents in the flash file system retained entries for every file that had ever existed, and a configuration error let it grow unbounded. After seven months of cruise, one day of entry descent and landing, a week of deployment and egress and a week of driving and science, the table consumed all available RAM. The out-of-memory condition silently suspended a critical task, which blocked file system access and prevented the shutdown algorithm from powering off the CPU, so the system reset on the delayed reset timer and did so again on every reboot because the table is rebuilt when flash is mounted. Crippled mode, which does not mount the flash file system, stopped the reset loop. Two weeks after the first symptoms the anomaly team reformatted flash, and for the following two months the problem was held off by limiting the number of files allowed in the file system until a new flight software load fixed the deleted-file representation [2].

Cruise solar flare, 28 October 2003. Both rovers lost attitude knowledge during the solar storm and the flight software dropped out of its nominal attitude control mode; Mars Odyssey detected corrupted RAM from the same storm [2]. Wakeup was triggered successfully by the alarm clock and the operations team commanded both spacecraft back to three-axis control.

Opportunity failed-on heater, landing night. An unexpected 0.5 A current appeared around 23:00 local solar time and disappeared around 09:30, with battery state of charge below prediction [2]. The cause was a failed-on heater switch on an instrument deployment device joint; the external thermostat cuts the circuit when temperatures rise but allows the heater on again when they fall, and multiple attempts to open the switch failed. The cost was 180 Wh per night [2]. The fix was deep sleep, a flight software modification that removes the batteries from the power bus at night and powers off every device including the battery control boards and the survival heaters [2]. Deep sleep boots the vehicle at 18:30 local solar time to pull the batteries offline, and must wait for the sun to be low enough that the arrays supply no current, because the battery control board hardware fault protection puts the batteries back on the bus when it sees bus voltage drop.

Testability

The MSL engine carries two utility commands that exist only for verification: one executes a single response directly without waiting for a monitor to turn red, testing the response rather than the engine, and one forces the engine to believe a specified monitor turned red, bypassing the monitor’s actual state [1]. Testing the pieces in isolation was the precondition for having confidence in the end-to-end monitor-engine-response scenarios. The MER experience is the argument for that investment: its fault protection design is distributed across both hardware and software, and it was exercised through cruise, landing and more than a year of surface operations before it could be called proven [2].