Verification
Proving flight software works before it flies: simulation in the loop, testbeds, coverage, and the classes of defect each method does and does not catch.
Flight software cannot be tested in its operating environment before launch, and a defect found after launch is repaired by uplink, if at all. The software also grows faster than the schedule that tests it: JPL avionics verification is described as traditionally performed on flight or flight-like hardware running flight software, while avionics complexity increased and the time allocated for integration and test stayed fixed [17]. The hardware that verification needs is scarce and shared, and the behavior to be checked is a combinatorial space of states, commands and faults rather than a list of cases [1][12].
Families of method
Section titled “Families of method”Workstation simulation. The flight software runs against modeled avionics on a desktop machine. MSL’s Work Station Test Set was used far more than the project’s hardware-in-the-loop testbeds [3], and the Mars 2020 equivalent ran the onboard planner campaign at 6 times real time [4]. It finds logic and state-interaction defects cheaply and cannot find defects that depend on hardware timing or physical response. A combinatorial campaign on the MSL wake cycle drove arbitrary combinations of stored module states through it, and the 3 percent of cases that failed were not diagnosed [25].
Software-in-the-loop with environment models. SSim adds rover mechanisms, terrain and rendered imagery to the flight software loop, and caught a mast camera focus lock defect that would have failed an entire drive [1]. The fidelity limits of such simulators, and what they do to the software under test, are covered in simulation fidelity for software.
Hardware-in-the-loop testbeds. Flight software testbeds hold flight-like compute elements, motor controllers and engineering models of mechanisms. Mars 2020 used them for collision-model verification [12], for the arm, turret and instrument software [2], and for docking and coring venues that gave the flight software early runtime against real hardware [13][14]. The Mars 2020 Vehicle System Testbed was built as a near-duplicate rover with avionics boxes verified pin by pin before mating [27], and motor control for 33 actuators was verified end to end on such venues [26]. Cassini’s Integrated Test Laboratory mixed real and simulated units at system level [7]. Psyche ran fault protection scenarios on two flight software testbeds, with instrument engineering models and simulators for payloads whose hardware was not available [10][11].
System-level test on the flight vehicle. MER ran a surface verification and validation program on flight hardware before shipment to the launch site and again at the launch site, under a compressed three-year schedule [8]. A MER project author argued that cost and criticality kept the flight rover out of realistic terrain testing, which left autonomy under-validated before landing [24]. Sequence rehearsal software for MER was checked against Spirit telemetry, with accuracy reported as bounds without sample sizes [28]. This rung has the highest fidelity and the lowest availability, and it is the only one that exercises the integrated vehicle.
Fault injection. Faults are inserted at the functional level or at individual bits in simulation, or at pins on hardware. The FTMP campaign injected 21,055 stuck faults and declined to publish a coverage figure because undetectable but harmless faults were not separated from real misses [6]. See flight software verification for the numbers.
Autonomy-specific and scenario-based testing. Planner-scheduler and executive software has an input space that cannot be enumerated. The Mars 2020 onboard planner campaign drew its scenarios from about one year of Perseverance operations data, and its stated gap is the unbounded set of conditions the planner will meet in flight [4][9].
Formal and informal methods. Chien’s survey of autonomy software from Deep Space 1 through the Mars 2020 scheduler finds that the bulk of verification effort was testing plus code walkthroughs, coding rules and design reviews, with formal methods usually reduced to static analyzers [9]. The exceptions are SPIN model checking of the Remote Agent executive, which found concurrency bugs, and SPIN with statechart code generation on MSL [9]. Formal methods have flown only in these narrow roles. SPIN found five concurrency bugs in the Remote Agent executive, four judged important, and the Mars 2020 planner verification combined four static analyzers with SPIN to find race conditions in multithreaded code [21][20][22]. Civil aviation guidance sets the wider argument: because software input-output behavior is discontinuous, finite test samples cannot be extrapolated, which is the case for formal methods [23]. Only a worked verification activity is reported for the planner, not a campaign-wide defect count [20].
Model-based verification. A JPL study modeled a remote engineering unit and its command path in SysML and ran test cases against both model and testbed. Its conclusion is that the tooling suited system-level scenarios and not hardware logic below the box level, and the paper reports no test results supporting its claim of matching behavior, so it is a feasibility study and not a validated method [17].
What the flown record shows
Section titled “What the flown record shows”The design of the software verification program follows the failure record more than the reverse. A study of 199 critical post-launch anomalies on seven spacecraft found that outdated or missing procedures were involved in about one fifth, and that requirement changes were new requirements for rare events and degraded hardware rather than corrections [5]. Test procedures that flagged “nothing fixed” problem reports predicted later operator confusion, which is a documentation finding more than a code finding [5].
Fault protection is the area where coverage is most visibly bounded. Psyche’s fault protection validation faced a few thousand failure modes, about 650 monitors and 185 responses, scoped only 176 end-to-end cases, and had executed 76 before the launch slip [10]. A separate account of the extra year reports no detection rates or flight data [19]. Mars 2020 fault protection derived over 400 scenario test cases, prioritized a minimum set of 150, and executed 265 across venues, with no defect counts reported [18]. The extra year surfaced real gaps, including attitude error monitors sized for slews that were also active during static pointing [10]. Payload fault protection was limited by the spacecraft’s response repertoire of power off and wait for ground, and a defect in which flight software infers instrument power state from commanding was recorded as not fixed before launch [11].
Flight data is the only check on what the ground venues predicted, and few papers publish it. The Mars 2020 Lander Vision System was verified on four venues: helicopter field tests, a testbed simulation certified against those tests and used for Monte Carlo runs, hardware-in-the-loop at box and system level, and EDL system simulations [29]. In flight it reduced a 3.2 km initial position error to under 40 m within 10 seconds, and Perseverance touched down within 5 m of the target selected by Safe Target Selection against a 60 m requirement [29]. The authors state that the 5 m figure is a system-level result and not a position error of the vision system, and that it rests on a single landing [29]. A review by the same program reports the flight position error as about 3 m against a 40 m requirement, with reference map error and not the algorithm dominating the pre-flight error budget [31]. The arm and turret commissioning over the first 98 sols compared Earth calibration with Mars behavior: joint resolver angles matched pre-launch values within 5 mrad, arm accuracy calibrated on Earth carried over without recalibration, and the first surface contact was found 2 mm early [30]. One ground calibration that had been descoped, the PIXL micro context camera model, left a 20 to 30 pixel absolute error that was recovered in flight [30].
Software architecture sets what is testable. The shift from hard-coded control logic to layered flight software, using Mars Reconnaissance Orbiter as the case, is argued to have raised both capability and the verification burden [15]. Flight-operations evidence feeds back too: Phoenix arm operations were replayed against ground testbed images and terrain models to correct positioning before sample delivery, a verification step performed during the mission itself [16].
Subsections
Section titled “Subsections”- Flight software verification: the ladder of test venues, what simulation found, the defect record, fault injection and planner verification.
- Simulation fidelity for software: what simulator gaps in force, imagery and time do to the software under test.
What is not established
Section titled “What is not established”No source here gives a coverage fraction for a flown rover or spacecraft flight software load: obligation counts, state-space coverage and fraction of behavior verified are not quantified in the surveys [9]. The FTMP report states no coverage number at all [6], and Psyche’s fault protection reports test counts against its design but no defect counts [10][11]. Residual defect rates for flown flight software are quoted from other literature, not measured [9]. The one quantitative defect study covers seven spacecraft launched before 2000 [5], and no comparable study of surface robotic software was found in the held sources. Model-based replacement of hardware testing is not demonstrated [17]. The flight comparisons above each cover one vehicle and one landing or one commissioning period [29][30].
References
- Verma, V. and Leger, C. (2019). SSim: NASA Mars Rover Robotics Flight Software Simulation
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{verma2019ssim, title = {SSim: NASA Mars Rover Robotics Flight Software Simulation}, author = {Verma, Vandi and Leger, Chris}, booktitle = {IEEE Aerospace Conference}, pages = {1-11}, address = {Big Sky, Montana}, year = {2019}, doi = {10.1109/aero.2019.8741862}, abstract = {Each new Mars rover has pursued increasingly richer science while tolerating a wider variety of environmental conditions and hardware degradation over longer mission operation duration. Sojourner operated for 83 sols (Martian days), Spirit for 2208 sols, and Opportunity is at 5111 sols, and Curiosity operation is ongoing at 2208 sols. To handle this increase in capability, the complexity of onboard flight software has increased. MSL (also known as Curiosity), uses more flight software lines of code than all previous missions to Mars combined, including both successes and failures[1]. MSL has more than 4,200 commands with as many as dozens of arguments, 54,000 parameters, and tens of thousands of additional state variables. A single high-level command may perform hours of configurable robotic arm and sampling behavior. Incorrect usage can result in the loss of an activity or the loss of the mission. Surface Simulation (“SSim”) was developed to address the challenge of making full and effective use of many capabilities of MSL, while managing complexity and risk. SSim is software that performs rapid context sensitive simulation of flight software. NASA Mars missions are comprised of three phases: several months of Cruise, a brief but exciting Entry Descent and Landing (EDL), and a Surface mission that typically lasts as long as the hardware survives. SSim is meant for use during the surface phase when the mission fulfills its primary objectives. The focus of SSim on MSL was the robotic flight software, including rover mobility and navigation, robotic arm manipulation, and sample acquisition, processing, and delivery. It can execute behaviors in simulation a thousand times faster than they execute in real time on the flight compute element. SSim is used by rover drivers to develop and validate command sequences throughout the planning cycle. SSim has been used to plan all of the Curiosity robotic operations since landing and is expected to continue to be used for the remaining life of the rover. Due to the impact of SSim on MSL, the Mars 2020 mission plans to increase the scope of SSim during flight operations, simulating not only rover planner operations, but all surface operations, including the instrument, power, thermal and telecommunication behavior. SSim is part of the Rover Sequencing and Visualization (RSVP) suite of Rover Planning tools [2]. In the paper we provide an overview of SSim architecture, design, implementation, and usage on MSL, as well as an overview of plans for Mars 2020.} } - Brooks, S., Litwin, T., Biesiadecki, J., Abcouwer, N., Del Sesto, T., McHenry, M., Myint, S., Twu, P. and Wai, D. (2022). Testing Mars 2020 Flight Software and Hardware in the Surface System Development Environment
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{brooks2022testing, title = {Testing Mars 2020 Flight Software and Hardware in the Surface System Development Environment}, author = {Brooks, Sawyer and Litwin, Todd and Biesiadecki, Jeffrey and Abcouwer, Neil and Del Sesto, Tyler and McHenry, Michael and Myint, Steven and Twu, Philip and Wai, Dennis}, booktitle = {IEEE Aerospace Conference}, pages = {1-13}, address = {Big Sky, Montana}, year = {2022}, doi = {10.1109/aero53065.2022.9843794}, abstract = {The Mars 2020 (M2020) Perseverance Rover is NASA's most advanced planetary rover mission to date. It includes a novel Sample Caching Subsystem (SCS) which will collect rock cores for possible future return to Earth, as well as an improved mobility system with enhanced autonomous navigation which will enable it to traverse faster and farther than prior rovers. The development of both systems required extensive flight software and flight hardware testing. To support this testing, we developed the Surface System Development Environment (SSDEV) and used it for a wide variety of testing. SSDEV is a bundled subset of M2020 Flight Software which runs on commercially available Linux computers and can be combined with multiple backend options for simulation and hardware control. The SSDEV architecture enabled our teams to perform much more testing of flight software and flight hardware than would have otherwise been possible. As a secondary benefit, the SSDEV-based test campaigns also helped our teams enter the operations phase of the mission with greater readiness of operations products and tools. In this paper, we summarize the motivation for SSDEV, provide an overview of the SSDEV architecture, list several examples of how SSDEV was used, and summarize lessons learned. SSDEV is not a substitute for integrated testing with flight-like avionics, but it enabled substantially more testing than would have otherwise been possible and also provided some unique benefits. We recommend architectures like SSDEV to future projects that need to perform extensive hardware and software testing using a limited set of flight-like avionics.} } - Henriquez, D., Canham, T., Chang, J. T. and McMahon, E. (2008). Workstation-Based Avionics Simulator to Support Mars Science Laboratory Flight Software Development
. AIAA SPACE Conference. Source
BibTeX
@inproceedings{henriquez2008workstation, title = {Workstation-Based Avionics Simulator to Support Mars Science Laboratory Flight Software Development}, author = {Henriquez, David and Canham, Timothy and Chang, Johnny T. and McMahon, Elihu}, booktitle = {AIAA SPACE Conference}, year = {2008}, doi = {10.2514/6.2008-6550}, abstract = {The Mars Science Laboratory developed the WorkStation TestSet (WSTS) to support flight software development. The WSTS is the non-real-time flight avionics simulator that is designed to be completely software-based and run on a workstation class Linux PC. This provides flight software developers with their own virtual avionics testbed and allows device-level and functional software testing when hardware testbeds are either not yet available or have limited availability. The WSTS has successfully off-loaded many flight software development activities from the project testbeds. At the writing of this paper, the WSTS has averaged an order of magnitude more usage than the project's hardware testbeds.} } - Parjan, S. and Gaines, D. (2024). In OBP We Trust: Verification and Validation of the M2020 On Board Planner Flight Software
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{parjan2024obp, title = {In OBP We Trust: Verification and Validation of the M2020 On Board Planner Flight Software}, author = {Parjan, Shreya and Gaines, Dan}, booktitle = {IEEE Aerospace Conference}, address = {Big Sky, Montana}, year = {2024}, doi = {10.48577/jpl.67lmez}, abstract = {With a planned deployment to the Perseverance roverin October 2023, the Mars 2020 On Board Planner (OBP) flightsoftware (FSW) will allow the rover to conserve energy and au-tonomously perform more activities when surplus resources (i.e.energy, data volume, time) are available. This functionality iscritical to support efficient resource management and maximizethe productivity of the vehicle as the rover’s battery degradesover time, particularly as the Mars Sample Return mission(MSR) may rely on Perseverance to facilitate sample deliveryto the return lander. Up until OBP’s deployment, operatorshave used the Master/Sub-Master paradigm (M/SM) for allMars missions to-date, in which rigid plans are uplinked to therover based on highly conservative models of activity resourceconsumption and duration that can unnecessarily constrain on-board plans. Thus, OBP must be robust enough to merit thetrust of mission science and operations personnel and preservethe health and safety of the spacecraft. To this end, we havecompleted a thorough verification and validation (V&V) cam-paign of the first iteration of OBP that will be deployed to therover as part of new Simple Planner (SP) operations paradigm.The campaign heavily relied on software testing to ensure thatcore flight software capabilities were met and that the softwarecomprehensively covered potential edge cases. In this paper,we motivate the need for a thorough V&V campaign going intothe deployment of On Board Planner to the Perseverance rover,contextualize how OBP integrates with the rest of the vehicle’sflight software, and leverage three case studies to evidence howtesting provided critical support towards the successful infusionof onboard scheduling software on the rover.} } - Lutz, R. R. and Mikulski, I. C. (2003). Patterns of Software Defect Data on Spacecraft
. NASA Office of Safety and Mission Assurance, Jet Propulsion Laboratory. Source
BibTeX
@techreport{lutz2003patterns, title = {Patterns of Software Defect Data on Spacecraft}, author = {Lutz, Robyn R. and Mikulski, Ines Carmen}, institution = {NASA Office of Safety and Mission Assurance, Jet Propulsion Laboratory}, year = {2003}, url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/7132} } - Lala, J. H. and Smith, T. B. I. (1983). Development and Evaluation of a Fault-Tolerant Multiprocessor (FTMP) Computer. Volume III: FTMP Test and Evaluation
. Charles Stark Draper Laboratory for NASA Langley Research Center, NASA CR-166073. Source
BibTeX
@techreport{lala1983development, title = {Development and Evaluation of a Fault-Tolerant Multiprocessor (FTMP) Computer. Volume III: FTMP Test and Evaluation}, author = {Lala, Jaynarayan H. and Smith, T. Basil, III}, number = {NASA CR-166073}, institution = {Charles Stark Draper Laboratory for NASA Langley Research Center}, year = {1983}, url = {https://ntrs.nasa.gov/citations/19850022395}, abstract = {The experimental test and evaluation of the Fault-Tolerant Multiprocessor (FTMP) is described. Major objectives of this exercise include expanding validation envelope, building confidence in the system, revealing any weaknesses in the architectural concepts and in their execution in hardware and software, and in general, stressing the hardware and software. To this end, pin-level faults were injected into one LRU of the FTMP and the FTMP response was measured in terms of fault detection, isolation, and recovery times. A total of 21,055 stuck-at-0, stuck-at-1 and invert-signal faults were injected in the CPU, memory, bus interface circuits, Bus Guardian Units, and voters and error latches. Of these, 17,418 were detected. At least 80 percent of undetected faults are estimated to be on unused pins. The multiprocessor identified all detected faults correctly and recovered successfully in each case. Total recovery time for all faults averaged a little over one second. This can be reduced to half a second by including appropriate self-tests.} } - Wang, E. and Brown, J. (2007). Cassini's Test Methodology for Flight Software Verification and Operations
. AIAA SPACE Conference. Source
BibTeX
@inproceedings{wang2007cassini, title = {Cassini's Test Methodology for Flight Software Verification and Operations}, author = {Wang, Eric and Brown, Jay}, booktitle = {AIAA SPACE Conference}, year = {2007}, url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/41356} } - Welch, R. V. (2005). Verification and validation of Mars exploration Rover Survace capabilities. Source
BibTeX
@other{welch2005b, title = {Verification and validation of Mars exploration Rover Survace capabilities}, author = {Welch, Richard V.}, year = {2005}, url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/37825} } - Chien, S. (2022). Formal Methods for Trusted Space Autonomy, Boon or Bane?
. NASA Formal Methods. Source
BibTeX
@inproceedings{chien2022formal, title = {Formal Methods for Trusted Space Autonomy, Boon or Bane?}, author = {Chien, Steve}, booktitle = {NASA Formal Methods}, series = {Lecture Notes in Computer Science}, pages = {3-13}, publisher = {Springer International Publishing}, year = {2022}, doi = {10.1007/978-3-031-06773-0_1} } - Pujari, S., Summer, J., Sereno, V. and Lumnah, A. (2024). Two Steps Forward: Evolving Psyche's Fault Protection Validation Post-Launch Slip
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{pujari2024two, title = {Two Steps Forward: Evolving Psyche's Fault Protection Validation Post-Launch Slip}, author = {Pujari, Swapnil and Summer, Jonathan and Sereno, Virginia and Lumnah, Alexander}, booktitle = {IEEE Aerospace Conference}, publisher = {JPL Open Repository}, year = {2024}, doi = {10.48577/jpl.grmhhb}, abstract = {On October 5th, 2023, the Psyche Mission is targeted to launch to the large metallic asteroid (16) Psyche that will be exploring the exposed core material of a differentiated planetesimal-class asteroid. This unique NASA Discovery-class Mission combines a Solar Electric Propulsion (SEP) Chassis from Maxar Technologies with JPL’s contribution of avionics, flight software, and fault protection design. The Psyche Fault Protection (FP), implemented as autonomous algorithms, can detect, isolate, respond to, and recover from off-nominal conditions that may prevent mission success.The FP Verification and Validation (V&V) campaign involved numerous types of testing including FP infrastructure, monitor & response end-to-end (E2E) functionality, scenario testing, and stress and robustness testing across both high and low-fidelity venues. Due to delays in system level software testing, Psyche had to reschedule its original launch window from July 2022 to October 2023. This paper focuses on how the V&V campaign was scoped and executed prior to the launch slip and then improved after the launch slip. Adaptations included testing a comprehensive E2E regression test suite by targeting updates made to the GNC FP design, a streamlined data execution that employed variance of initial conditions and a consistent test procedure, and a robust and efficient data review strategy including tooling to automate checks. By capitalizing on the additional time, resources, and insights gained from Psyche’s launch slip, the Fault Protection V&V campaign benefited from a more comprehensive and rigorous validation process. This paper captures the knowledge and processes employed to validating Psyche’s Fault Protection design such that future FP V&V campaigns can benefit from more robust testing.} } - Srivastava, P., Bates-Tarasewicz, H. A., Cloutier, K. D., De Soria-Santacruz Pich, M., Lumnah, A. J. and Wilkerson, M. (2024). Psyche's Payload Fault Protection Design and Strategy for Launch Readiness
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{srivastava2024psyche, title = {Psyche's Payload Fault Protection Design and Strategy for Launch Readiness}, author = {Srivastava, Priyanka and Bates-Tarasewicz, Haley A. and Cloutier, Kyle D. and De Soria-Santacruz Pich, Maria and Lumnah, Alexander J. and Wilkerson, Marcus}, booktitle = {IEEE Aerospace Conference}, publisher = {JPL Open Repository}, year = {2024}, doi = {10.48577/jpl.ivff4w}, abstract = {NASA's Psyche mission is a journey to the unique metal-rich (16) Psyche asteroid orbiting the Sun between Mars and Jupiter to get a better insight into the Earth's core and the core of other terrestrial planets. The mission will offer a unique scientific value by mapping the asteroid and studying its properties with the help of a robust Spacecraft that carries a highly capable Payload suite onboard. Psyche's Payload suite is a novel collection of heritage and technology demonstration instruments. Heritage instruments include a Multispectral Imager, a Magnetometer, and a Gamma Ray and Neutron Spectrometer. The Deep Space Optical Communications (DSOC) instrument is a Class D technology demonstration that will perform optical communications at AU-magnitude distances. The Spacecraft will need to survive in vaguely known environments and with a wide range of uncertainties from the time of Launch to the Psyche Science Orbital Operations period. Most of these instruments will be functioning at the same time in conjunction with the rest of the subsystems, and with limited contact with the ground operations team on Earth. To ensure that the Spacecraft and instruments are robust to possible anomalies and can remain healthy and functional to meet the science goals and overall mission success criteria, Psyche's Fault Protection (FP) and Payload teams designed and implemented an autonomous onboard fault management architecture and engine. This Flight Software-based autonomy will execute fault detection, isolation and perform health recovery actions to prevent propagation of anomalous behavior between the instruments and the rest of the Spacecraft. This paper details the history of how Psyche’s Payload Fault Protection design evolved and was fine-tuned through a rigorous testing and analysis campaign to address the needs and challenges of the diverse set of instruments. It describes the key lessons learned, including the ones that emerged post launch-slip in the year 2022, and how a shift in the FP Verification and Validation (V&V) strategy helped the team gain increased confidence in the instruments for Launch readiness in 2023.} } - Verma, V., Huang, J., Bailey, P., Carsten, J. and Klein, D. (2022). Perseverance Rover Collision Model for a range of Autonomous Behaviors
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{verma2022perseverance, title = {Perseverance Rover Collision Model for a range of Autonomous Behaviors}, author = {Verma, Vandi and Huang, Justin and Bailey, Philip and Carsten, Joseph and Klein, Douglas}, booktitle = {IEEE Aerospace Conference}, pages = {1-18}, publisher = {IEEE}, year = {2022}, doi = {10.1109/aero53065.2022.9843632} } - Siegfriedt, R. S., Girerd, A., Hazelrig, J., Gaines, D., Fosse, E., Appakonam, A., Waldram, N., Plave, A. and Rozek, M. (2023). Enabling Parallel Activities for Mars 2020 Rover Surface Operations
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{siegfriedt2023enabling, title = {Enabling Parallel Activities for Mars 2020 Rover Surface Operations}, author = {Siegfriedt, Rebekah S. and Girerd, Andre and Hazelrig, James and Gaines, Daniel and Fosse, Elyse and Appakonam, Ashwin and Waldram, Nicholas and Plave, Andrew and Rozek, Matthew}, booktitle = {IEEE Aerospace Conference}, address = {Big Sky, Montana}, year = {2023}, doi = {10.48577/jpl.obe5e4}, abstract = {Level 1 requirements levied on the Mars 2020 mission from NASA Headquarters challenged the Mars 2020 surface mission (also known as the Perseverance rover) to perform significantly better relative to the 2012 Mars Science Lab (MSL) surface mission (also known as the Curiosity rover). In one and a quarter Mars years, the Perseverance rover was expected to drive 50 percent farther and collect over twice as many samples as its predecessor, Curiosity. In order to reach these requirements, Mars 2020 came up with five guiding principles, two of which were “Be fast, be flexible: perform flight and ground functions more quickly” and “Do more science: increase the time that the vehicle is actively pursuing science on Mars''. These guiding principles required the mission to lay the groundwork early on for more surface activities to be performed in parallel, such as thinking while driving. Parallelism enabled both direct efficiency gains, such as completing multiple activities simultaneously during a given time frame, and indirect gains, such as increased available energy and better informed tactical planning due to more data being available sooner. These efficiency gains allowed the rover to complete more complex plans and better handle adversity such as data return disruptions. Although the mission planned to test and approve most of the parallel activities in the development phase, limitations on budget, time and personnel drove the majority of parallel activity testing and approvals to occur in the operations phase. Ideally, any activity can run concurrently with any other activity, but constraints such as Electromagnetic Interference (EMI), hardware safety, and resource utilization prevent some activities from running in parallel. In order to coordinate these behaviors on the rover, the mission needed an efficient and streamlined mechanism to manage them on the ground and maximize the amount of time the rover could pursue science on Mars. In response to this challenge, the Mars 2020 Surface Operations Team overhauled a legacy product from MSL called the Parallelism Matrix. Over the course of the primary mission, the Mars 2020 Parallelism Team made the following improvements: established a process to triage untested activity parallelism, enabled automatic updates to the matrix after downlinked data was analyzed, created a thorough testing campaign to investigate and approve untested parallel interactions, and provided a tool to autonomously track executed parallel activities as soon as data hit the ground.This paper will discuss: similarities and differences between the Perseverance and Curiosity missions with respect to maximizing science exploration time, Mars 2020’s new Flight Software (FSW) capabilities that enabled more parallelism, the Surface EMI testing campaign, the Surface Operations Parallelism Working Group efforts, Surface Operations parallelism tools and processes, the current state of parallel activity efforts on Mars 2020, the measurable benefits and achievements of parallelism, and the lessons learned and recommendations for the future.} } - Brooks, S., Townsend, J., Collins, C., Carsten, J., Frost, M., Reid, J., Robinson, M. and Warner, A. (2022). Docking the Mars 2020 Perseverance Robotic Arm
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{brooks2022docking, title = {Docking the Mars 2020 Perseverance Robotic Arm}, author = {Brooks, Sawyer and Townsend, Julie and Collins, Curtis and Carsten, Joseph and Frost, Matthew and Reid, Jason and Robinson, Matthew and Warner, Antonia}, booktitle = {IEEE Aerospace Conference}, pages = {1-12}, address = {Big Sky, Montana}, year = {2022}, doi = {10.1109/aero53065.2022.9843517} } - Butler, R. (2013). The evolution of software and its impact on complex system design in robotic spacecraft embedded systems
. Procedia Computer Science. doi.org/10.1016/j.procs.2013.01.078
BibTeX
@misc{butler2013evolution, title = {The evolution of software and its impact on complex system design in robotic spacecraft embedded systems}, author = {Butler, Roy}, journal = {Procedia Computer Science}, volume = {16}, pages = {747-756}, year = {2013}, doi = {10.1016/j.procs.2013.01.078}, abstract = {The growth in computer hardware performance, coupled with reduced energy requirements, has led to a rapid expansion of the resources available to software systems, driving them towards greater logical abstraction, flexibility, and complexity. This shift in focus from compacting functionality into a limited field towards developing layered, multi-state architectures in a grand field has both driven and been driven by the history of embedded processor design in the robotic spacecraft industry.The combinatorial growth of interprocess conditions is accompanied by benefits (concurrent development, situational autonomy, and evolution of goals) and drawbacks (late integration, non-deterministic interactions, and multifaceted anomalies) in achieving mission success, as illustrated by the case of the Mars Reconnaissance Orbiter. Approaches to optimizing the benefits while mitigating the drawbacks have taken the shape of the formalization of requirements, modular design practices, extensive system simulation, and spacecraft data trend analysis. The growth of hardware capability and software complexity can be expected to continue, with future directions including stackable commodity subsystems, computer-generated algorithms, runtime reconfigurable processors, and greater autonomy.} } - Bonitz, R., Shiraishi, L., Robinson, M., Carsten, J., Volpe, R., Trebi-Ollennu, A., Arvidson, R. E., Chu, P. C., Wilson, J. J. and Davis, K. R. (2009). The Phoenix Mars Lander Robotic Arm
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{bonitz2009phoenix, title = {The Phoenix Mars Lander Robotic Arm}, author = {Bonitz, Robert and Shiraishi, Lori and Robinson, Matthew and Carsten, Joseph and Volpe, Richard and Trebi-Ollennu, Ashitey and Arvidson, Raymond E. and Chu, P. C. and Wilson, J. J. and Davis, K. R.}, booktitle = {IEEE Aerospace Conference}, pages = {1-12}, organization = {Jet Propulsion Laboratory, California Institute of Technology}, address = {Big Sky, Montana}, year = {2009}, doi = {10.1109/aero.2009.4839306}, abstract = {The Phoenix Mars Lander Robotic Arm (RA) has operated for 149 sols since the Lander touched down on the north polar region of Mars on May 25, 2008. During its mission it has dug numerous trenches in the Martian regolith, acquired samples of Martian dry and icy soil, and delivered them to the Thermal Evolved Gas Analyzer (TEGA) and the Microscopy, Electrochemistry, and Conductivity Analyzer (MECA). The RA inserted the Thermal and Electrical Conductivity Probe (TECP) into the Martian regolith and positioned it at various heights above the surface for relative humidity measurements. The RA was used to point the Robotic Arm Camera to take images of the surface, trenches, samples within the scoop, and other objects of scientific interest within its workspace. Data from the RA sensors during trenching, scraping, and trench cave-in experiments have been used to infer mechanical properties of the Martian soil. This paper describes the design and operations of the RA as a critical component of the Phoenix Mars Lander necessary to achieve the scientific goals of the mission.} } - Khan, M. O., Sievers, M. and Standley, S. (2012). Model-based verification and validation of spacecraft avionics
. Infotech@Aerospace. Source
BibTeX
@inproceedings{khan2012model, title = {Model-based verification and validation of spacecraft avionics}, author = {Khan, M. Omair and Sievers, Michael and Standley, Shaun}, booktitle = {Infotech@Aerospace}, publisher = {American Institute of Aeronautics and Astronautics}, year = {2012}, doi = {10.2514/6.2012-2555}, abstract = {Verification and Validation (VV the norm for thermal and structural VV this experiment demonstrated such an approach is possible.} } - Clark, J. J., Lam, M., DuCharme, L., Lumnah, A. and Morantz, C. (2022). Validation of the Mars 2020 Fault Protection Design: Navigating the Infinity of the Off-Nominal
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{clark2022validation, title = {Validation of the Mars 2020 Fault Protection Design: Navigating the Infinity of the Off-Nominal}, author = {Clark, Jessica J and Lam, Mary and DuCharme, Lauren and Lumnah, Alex and Morantz, Chaz}, booktitle = {IEEE Aerospace Conference}, pages = {1-8}, publisher = {IEEE}, year = {2022}, doi = {10.1109/aero53065.2022.9843661}, abstract = {On July 30th 2020, the Mars 2020 mission successfully launched out of Cape Canaveral, Florida, passed through the Earth's shadow, and began its short cruise to Mars. Less than seven months later, the Perseverance rover touched down safely in Jezero Crater to begin its ambitious mission that includes looking for signs of ancient life and collecting samples for future return to Earth. Getting to the successful landing, or “Tango Delta Nominal,” could not have been achieved without also considering the off-nominal. One of the teams supporting this ambitious mission is the fault protection (FP) team. This team is tasked with assessing the various failures, or faults, that could prevent mission success and with ensuring that the autonomous behaviors built into the software and hardware can detect faults and recover the vehicle to a safe state. As part of its charter, the FP team designed a test campaign to provide confidence in the system's robustness to off-nominal scenarios across all of Mars 2020's mission phases. The greatest challenge associated with designing such a validation campaign was reducing the infinite number of anomalous scenarios into a finite test suite. In addition, the tests needed to be executed efficiently in order to utilize the team's limited test venue access, but still needed to maintain a level of rigor that guaranteed confidence in the test outcomes. Given that each test scenario generated massive amounts of data, the team also developed methods for quickly ascertaining whether the autonomous fault protection behaviors maintained vehicle safety in the presence of an anomaly. This paper summarizes the processes that the Mars 2020 fault protection team employed to execute its off-nominal validation campaign. It captures both the methods of generating a suite of off-nominal tests, as well as reducing it to a subset that can be realistically executed within schedule and resource constraints. It also describes the various processes and philosophies that the team utilized to execute the tests efficiently, including creating a standardized procedure template, keeping the test cases modular so that they could be easily interchanged, and capturing common fault injections in a change-controlled database. Finally, it will describe the tools and processes for assessing the test data, focusing in particular on a tool that evaluated vehicle state using “secondary” sources of data to validate that the software had truly configured the spacecraft to the expected safe state.} } - Lumnah, A., Pujari, S. and Sereno, V. (2024). Driving Down Risk in the Psyche Fault Protection Design Post Launch Slip
. International Symposium on Experimental Robotics. Source
BibTeX
@inproceedings{lumnah2024driving, title = {Driving Down Risk in the Psyche Fault Protection Design Post Launch Slip}, author = {Lumnah, Alexander and Pujari, Swapnil and Sereno, Virginia}, booktitle = {International Symposium on Experimental Robotics}, publisher = {JPL Open Repository}, year = {2024}, doi = {10.48577/jpl.wdvmh8}, abstract = {The Psyche Mission is a NASA Discovery-class Mission to explore (16) Psyche, a large metallic asteroid that orbits the sun between 2.5 and 3.3 au. The spacecraft bus is a repurposed earth-orbiting telecommunication satellite bus developed by Maxar, combined with Avionics and software developed by JPL. To ensure this earth-orbiting spacecraft bus can survive the lengthy journey into deep space, a robust fault protection function was designed, implemented, and tested. Originally scheduled for July 2022, the launch date was postponed to October 2023 due to delays during system-level testing of the spacecraft. By the time the launch slip was announced, the fault protection design was fully implemented in software, and the V&V program was nearly complete. Consequently, the team found themselves with an additional year of unplanned development time, which provided the unique opportunity to revisit parts of the fault protection design, implementation, and V&V strategy prior to launch, while incorporating lessons learned from the previous year of testing.This paper provides an overview of the efforts taken to drive down risk in the Psyche fault protection design after the launch slip. These efforts can be split into three categories: additional “deep dive” analyses targeting higher risk areas of the design, improvements to the design and implementation that resulted in software changes, and the development of a rich set of high-fidelity risk reduction test cases meant to validate the fault protection function. This paper discusses the motivating concerns that drove each of these efforts and explains how lessons learned were incorporated into the post-launch slip development work. It summarizes the analyses that were spawned and explains the methods that were used to constrain and complete each analysis. It captures the trades made when deciding which design changes and software bug fixes to implement, and describes the processes used to implement and verify those changes in an agile manner. Finally, the paper captures the brainstorming process for devising a set of risk reduction tests used to improve confidence in the fault protection design, and validate its performance over a wide range of scenarios.} } - (397K), S. P., Gaines, D., Kuhn, S. and Rabideau, G. (2024). Towards Trusted Mars Autonomy: V&V of the M2020 On Board Planner's Thermal Management Capabilities
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{k2024trusted, title = {Towards Trusted Mars Autonomy: V&V of the M2020 On Board Planner's Thermal Management Capabilities}, author = {{(397K), Shreya Parjan} and Gaines, Dan and Kuhn, Stephen and Rabideau, Gregg}, booktitle = {IEEE Aerospace Conference}, publisher = {JPL Open Repository}, year = {2024}, doi = {10.48577/jpl.mi9iun} } - Lowry, M. and Dvorak, D. (1998). Analytic Verification of Flight Software
. IEEE Intelligent Systems. Source
BibTeX
@article{lowry1998analytic, title = {Analytic Verification of Flight Software}, author = {Lowry, Michael and Dvorak, Daniel}, journal = {IEEE Intelligent Systems}, volume = {13}, pages = {45-49}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, year = {1998}, doi = {10.1109/5254.722359}, abstract = {To provide rigorous validation and verification of autonomous flight software for space vehicles, the authors apply two analytic verification approaches: design-time model checking and run-time behavior auditing.} } - Holzmann, G. J. (1997). The Model Checker SPIN
. IEEE Transactions on Software Engineering, 5. Source
BibTeX
@article{holzmann1997model, title = {The Model Checker SPIN}, author = {Holzmann, Gerard J.}, journal = {IEEE Transactions on Software Engineering}, volume = {23}, number = {5}, pages = {279--295}, year = {1997}, doi = {10.1109/32.588521}, abstract = {SPIN is an efficient verification system for models of distributed software systems. It has been used to detect design errors in applications ranging from high-level descriptions of distributed algorithms to detailed code for controlling telephone exchanges. The paper gives an overview of the design and structure of the verifier, reviews its theoretical foundation, and gives an overview of significant practical applications.} } - Rushby, J. (1995). Formal Methods and Their Role in Digital Systems Validation for Airborne Systems
. SRI International, prepared for NASA Langley Research Center, NASA Contractor Report 4673 (SRI CSL-95-1), Contract NAS1-18969. Source
BibTeX
@techreport{rushby1995formal, title = {Formal Methods and Their Role in Digital Systems Validation for Airborne Systems}, author = {Rushby, John}, number = {NASA Contractor Report 4673 (SRI CSL-95-1), Contract NAS1-18969}, institution = {SRI International, prepared for NASA Langley Research Center}, year = {1995}, url = {https://ntrs.nasa.gov/citations/19960008816}, abstract = {This report is based on one prepared as a chapter for the FAA Digital Systems Validation Handbook (a guide to assist FAA certification specialists with advanced technology issues). Its purpose is to explain the use of formal methods in the specification and verification of software and hardware requirements, designs, and implementations; to identify the benefits, weaknesses, and difficulties in applying these methods to digital systems used in critical applications; and to suggest factors for consideration when formal methods are offered in support of certification. The presentation concentrates on the rationale for formal methods and on their contribution to assurance for critical applications within a context such as that provided by DO-178B (the guidelines for software used on board civil aircraft); it is intended as an introduction for those to whom these topics are new.} } - Tunstel, E. W. (2003). Autonomous mobility software validation challenges for planetary surface missions
. Journal of Spacecraft and Rockets. Source
BibTeX
@article{tunstel2003autonomous, title = {Autonomous mobility software validation challenges for planetary surface missions}, author = {Tunstel, Edward W.}, journal = {Journal of Spacecraft and Rockets}, publisher = {JPL Open Repository}, year = {2003}, url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/7133} } - Whitehill, R. (2011). Wake cycle robustness of the Mars Science Laboratory Flight Software
. SPIE Defense & Commercial Sensing. Source
BibTeX
@inproceedings{whitehill2011wake, title = {Wake cycle robustness of the Mars Science Laboratory Flight Software}, author = {Whitehill, Robert}, booktitle = {SPIE Defense & Commercial Sensing}, publisher = {JPL Open Repository}, year = {2011}, url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/43715} } - Walker, L., Austin, J., Kennett, A. and Borne, D. (2022). Actuator and Motor Control End-to-End V&V on the Mars 2020 Rover
. JPL Open Repository. Source
BibTeX
@inproceedings{walker2022actuator, title = {Actuator and Motor Control End-to-End V&V on the Mars 2020 Rover}, author = {Walker, Luke and Austin, Jesse and Kennett, Andrew and Borne, Davis}, publisher = {JPL Open Repository}, year = {2022}, url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/55967} } - Matthes, C., Stumbo, M. and Foley, J. (2021). How to Build a Rover: An Overview of the Mars 2020 Mission's Vehicle System Testbed
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{matthes2021how, title = {How to Build a Rover: An Overview of the Mars 2020 Mission's Vehicle System Testbed}, author = {Matthes, Christopher and Stumbo, Matthew and Foley, Justin}, booktitle = {IEEE Aerospace Conference}, pages = {1-12}, year = {2021}, doi = {10.1109/aero53065.2022.9843658}, abstract = {While NASA's Mars rover Perseverance continues to make groundbreaking achievements on the Red Planet, its twin is hard at work here on Earth. The Operational Perseverance Twin for the Integration of Mechanisms and Instruments Sent to Mars, or OPTIMISM, is the Mars 2020 Vehicle System Testbed (VSTB) rover operated by NASA Jet Propulsion Laboratory (JPL) in Pasadena, California. OPTIMISM's home is the JPL Mars Yard; an outdoor field with red soil that simulates the terrain encountered by Perseverance. The VSTB is a full-scale engineering model of the flight rover, serving a number of functions to ensure mission operations can continue smoothly and on schedule. The VSTB possesses instrumentation, computers, mechanisms, cameras, and a Mobility subsystem that are nearly identical to its extraterrestrial twin. Its high fidelity allows the rover to be a highly effective tool to fully test system functionality and performance prior to commanding the flight rover. The early stages of building OPTIMISM began a few months prior to Perseverance departing JPL for Cape Canaveral, FL in early 2020. Electrical integration of the flight system avionics, and compatibility checkouts of the electrical ground support equipment ensured that the foundation of the electrical system was operational and in place. Next, the internal harnessing was installed and compatibility checks of the rover instrumentation and mechanisms were performed to confirm the system was prepared for full buildup. Finally, mechanical assembly of the rover chassis with its external components completed the integration of the system before it was moved to the Mars Yard for its initial phase of testing to perform verification & validation (V&V) of the Mobility subsystem requirements. By the time Perseverance landed at Jezero Crater in February 2021, the first phase of VSTB operations was underway. Surface guidance, navigation, and control (SGNC) testing for the Mobility subsystem ensured functionality and performance requirements were met for various capabilities such as visual odometry (VO), mapping, and automatic navigation (AutoNav). Subsequent integration of the robotic arm (RA) onto the VSTB enabled the V&V campaign for surface sampling operations (SSO) to commence. As the mission's engineering operations (EO) have gotten underway, the VSTB has been utilized for an array of purposes including troubleshooting software anomalies, and performing dry-runs for first time activities (FTAs) prior to sending the commands to Perseverance. OPTIMISM will continue to serve mission critical functions as long as Perseverance is roving the Red Planet.} } - Yen, J., Cooper, B., Hartman, F., Maxwell, S. and Wright, J. (2004). Sequence rehearsal and validation on surface operations of the Mars Exploration Rovers
. ["SPIE's International Symposium AeroSense. Source
BibTeX
@inproceedings{yen2004sequence, title = {Sequence rehearsal and validation on surface operations of the Mars Exploration Rovers}, author = {Yen, Jeng and Cooper, Brian and Hartman, Frank and Maxwell, Scott and Wright, John}, booktitle = {["SPIE's International Symposium AeroSense}, publisher = {JPL Open Repository}, year = {2004}, url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/37848} } - Johnson, A. E., Aaron, S., Ansari, H., Bergh, C., Bourdu, H., Butler, J., Chang, J., Cheng, R., Cheng, Y., Clark, K., Clouse, D., Donnelly, R., Gostelow, K., Jay, W., Jordan, M., Mohan, S., Montgomery, J. F., Morrison, J., Schroeder, S., Shenker, B., Sun, G., Trawny, N., Umsted, C., Vaughan, G., Ravine, M., Schaffner, J., Shamah, J. M. and Zheng, J. (2022). Mars 2020 Lander Vision System Flight Performance
. AIAA SCITECH Forum. Source
BibTeX
@inproceedings{johnson2022mars, title = {Mars 2020 Lander Vision System Flight Performance}, author = {Johnson, Andrew E. and Aaron, Seth and Ansari, Hugh and Bergh, Charles and Bourdu, Helene and Butler, Jim and Chang, Johnny and Cheng, Richard and Cheng, Yang and Clark, Ken and Clouse, Daniel and Donnelly, Robert and Gostelow, Kim and Jay, William and Jordan, Michael and Mohan, Swati and Montgomery, James F. and Morrison, Jack and Schroeder, Steven and Shenker, Boris and Sun, George and Trawny, Nikolas and Umsted, Carson and Vaughan, Geoffrey and Ravine, Michael and Schaffner, Jacob and Shamah, Joe M. and Zheng, Jason}, booktitle = {AIAA SCITECH Forum}, year = {2022}, doi = {10.2514/6.2022-1214}, abstract = {View Video Presentation: https://doi.org/10.2514/6.2022-1214.vid The Mars 2020 Entry Descent and Landing (EDL) system delivered the Perseverance rover to the surface of Mars on February 18th, 2021. A large fraction of the Jezero Crater landing site was covered with landing hazards including cliffs, inescapable dune fields and rocks. These hazards were identified or inferred using orbital imagery before launch so that they could be avoided using Terrain Relative Navigation (TRN) which was composed of two parts: the Lander Vision System (LVS) and Safe Target Selection (STS). During EDL, the LVS successfully estimated map relative position by fusing landmarks matched between descent imagery and a map of the landing site with Inertial Measurement Unit (IMU) data. This position estimate was used by STS to identify the safest target for landing that was also reachable given fuel and other constraints. The EDL system then used the powered descent phase to retarget to this location and land safely. The overall error between the targeted location and actual landing location was 5m which was an order of magnitude less than the 60m touchdown error requirement. This paper will describe the final tests of the LVS before launch, the checkout of the LVS during operations and the LVS performance during EDL.} } - Bailey, P., Klein, D., Zorn, T., Srinivasan, T., Schaler, E. W., Brooks, S., Beegle, L., Dolci, M., Hernandez, C., Sondheim, M., Kennedy, M. R. and Edgett, K. (2022). Perseverance Rover's Robotic Arm and Turret Mounted Instruments' Surface Commissioning
. IEEE Aerospace Conference. Source
BibTeX
@inproceedings{bailey2022perseverance, title = {Perseverance Rover's Robotic Arm and Turret Mounted Instruments' Surface Commissioning}, author = {Bailey, Philip and Klein, Douglas and Zorn, Torsten and Srinivasan, Thirupathi and Schaler, Ethan W. and Brooks, Sawyer and Beegle, Luther and Dolci, Marco and Hernandez, Christina and Sondheim, Michael and Kennedy, Megan R and Edgett, Kenneth}, booktitle = {IEEE Aerospace Conference}, publisher = {JPL Open Repository}, year = {2022}, url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/55994} } - Johnson, A. E., Trawny, N., Cheng, Y., Ansar, A. I., Montgomery, J. F., Matthies, L. H. and Thurman, S. W. (2023). Development of Terrain-Relative Guidance for Planetary Landing Applications
. Annual AAS Guidance, Navigation and Control Conference. Source
BibTeX
@inproceedings{johnson2023development, title = {Development of Terrain-Relative Guidance for Planetary Landing Applications}, author = {Johnson, Andrew E. and Trawny, Nikolas and Cheng, Yang and Ansar, Adnan I. and Montgomery, James F. and Matthies, Larry H. and Thurman, Sam W.}, booktitle = {Annual AAS Guidance, Navigation and Control Conference}, publisher = {JPL Open Repository}, year = {2023}, doi = {10.48577/jpl.i1uewa}, abstract = {The use of terrain sensing for navigation and guidance in planetary soft landing is a powerful capability for robotic space missions but one that took decades to mature across many different research tasks, technology developments, and field/flight test activities. This paper summarizes significant milestones in the development of this important capability for planetary exploration, which was first employed operationally in NASA’s Mars 2020 mission, as well as related applications that are just maturing and coming into use. A brief look at current work leading to future applications is included.} }