Skip to content
Space Robotics

Hazard Detection

Hazard detection converts a range measurement into a statement about where the vehicle can put itself. Stereo is not the only way to get that measurement. Two stereo-based families have flown on rovers. GESTALT, on Sojourner’s successors from 2004 through Curiosity, reduces the rover to a point and inflates obstacles by the rover radius [1]. Approximate Clearance Evaluation, on Perseverance, keeps the wheels and the suspension and bounds what the vehicle would actually do on the terrain [3]. A third, older approach avoids stereo correlation altogether: splitting a laser into co-planar beams and tilting the beam plane so each spot falls on its own scan line lets a single exposure with simple peak detection find obstacle heights without image differencing, which is what the Mars Pathfinder-derived laser hazard sensor design did, running a complete scan in 830 ms on a 486 processor against an estimated 4 s on the 8085 the Pathfinder microrover actually carried [8].

Where the height threshold and the driving cycle came from

Section titled “Where the height threshold and the driving cycle came from”

The criterion underneath every geometric hazard detector, fit a reference ground surface and call anything more than a height threshold above it an object, was established on stereo imagery from Viking Lander 1 [6]. Thresholds of 3, 5, 6 and 7 cm above the fitted ground surface were used on one image pair from the lander’s two facsimile cameras, 0.8187 m apart at 1.3 m above the reference plane, with the points above threshold grouped and approximated by ellipsoids [6]. Scene range in the test area was 3 to 4.5 m at one illumination, so the threshold was never established at rover standoff distances, and there was no independent ground truth for the rock positions, hence no detection or false alarm rate. The same work carried a per-pixel range variance and match probability out of the area correlator rather than a bare disparity, which is what makes a stereo range map usable against a height threshold at all.

The stop-perceive-plan-step cycle that MER, MSL and Mars 2020 still drive on is a consequence of perception cost, not of a control argument. The Stanford Cart took 10 to 15 minutes per meter, nine images per stop through an interest operator, a coarse-to-fine correlator, a camera solver and a path replan on a timeshared DEC KL-10 over a radio link, and completed 20 m indoors in about five hours [7]. Three of its five runs ended on obstacles the interest operator put no feature on, including one outdoor collision, which is the same failure class a modern goodness map addresses by fitting terrain rather than tracking isolated points.

GESTALT

Section titled “GESTALT”

Grid-based Estimation of Surface Traversability Applied to Local Terrain takes the 3D point cloud from stereo and produces a goodness map, also called a traversability map [1][2].

ParameterSpiritOpportunity
Map extent10 x 10 m12 x 12 m
Cell size0.2 m0.2 m
Fitted patchcircle of rover diameter, nominally 2.6 msame
Point clouds merged before an assessmentup to 10up to 10

Sources: [1], [2]. For each map cell in turn, the complete set of range points is fitted with a planar patch centered on that cell, the patch being a circle of the rover’s diameter. Four quantities from that fit produce the cell’s goodness: the surface normal, the RMS residual, and the minimum and maximum elevation difference from the best fit plane [1]. These correspond to the three named hazard classes: Step Obstacles, large elevation deltas from the fit plane; Tilt Hazards, large angle between the surface normal and the up vector; and Roughness hazards, the residual of the planar fit [2].

Goodness maps from successive range images are registered and accumulated using modulo map indexing, which bounds the map without scrolling; new data overwrites old where they overlap [1]. The merged map is then used to score a fixed set of candidate steering trajectories: 23 forward arcs, 23 backward arcs and two point turns, evaluated every driving cycle by summing goodness along each arc with nearby cells weighted more heavily. The votes go to an arbiter that also weighs operator-supplied waypoints, and the rover drives a fixed distance along the winning arc, set by operators anywhere from 35 cm to 1 m or more depending on terrain and the distance goal for the sol, before stopping to image again [1].

What it cost

Section titled “What it cost”

A GESTALT cycle takes about 70 s on the MER flight processor [1]. Peak driving speed is 5 cm/s and the rovers were typically driven at 3.75 cm/s for power reasons, but with computing time included the median net driving speed under autonomous navigation was about 0.6 cm/s. The response was the hybrid daily drive: operators plan a blind segment as far as they can verify safety in navigation and panoramic camera stereo, the rover drives that blind, then switches to autonomous navigation for whatever time remains. Spirit’s longest commanded drive to mid-June 2004 was 124 m on sol 125, 62 m directed and 62 m under Terrain Assessment with Local Path Selection [1][2]. Opportunity’s longest planned drive, a February 2005 holiday weekend, was 106 m of Local Path Selection with no vision processing followed by 284 m with Terrain Assessment over sols 383 to 385, in segments of 70, 104 and 110 m, with humans reviewing telemetry each sol but sending no further commands after the first day.

As of 15 August 2005, onboard terrain assessment had been used for 1354 m of Spirit’s 4798 m and 1379 m of Opportunity’s 5947 m, about 25 percent for both [2][9]. Nearly a third of Spirit’s more than 2 km trek to the Columbia Hills was driven under Terrain Assessment, which the mission credits with reaching the hills 50 percent sooner than directed driving alone would have allowed safely. Blind directed driving still dominated both totals, because the stereo and goodness-map evaluation behind GESTALT cost far more time than the drive itself [9].

GESTALT’s most consequential miss was not a rock. Purgatory Ripple mired Opportunity for over a month after 50 m of commanded driving produced 2 m of progress on a dune whose hazard was slip, not geometry, and so was invisible to a detector that only fits planes to a point cloud [9]. The fix was operational rather than perceptual: visual odometry was adopted as a slip gate that forbids commanding more than 5 m of driving without a confirmed motion measurement, and on Sol 603 it caught 44 percent slip on a similar dune and stopped the drive before it repeated Purgatory [9]. That gap between what GESTALT checks and what actually stops a rover is the same one that motivated slip prediction and terrain classification [5].

Failure behavior

Section titled “Failure behavior”

The asymmetry that governs the design is that a false positive, a safe path assessed as unsafe, is acceptable, and a false negative, an unsafe path assessed as safe, is not [3]. GESTALT is therefore built conservative, and its documented failure mode is the false positive. Representing the rover as a point and inflating obstacles by the rover radius means a rock that the vehicle could straddle with the belly pan clear is treated as impassable, and GESTALT is known to fail frequently in high rock density. In a MATLAB reimplementation driven over 80 randomly generated 30 by 40 m terrains at four cumulative fractional area rock coverages, the GESTALT-equivalent planner reached a goal 20 m away in only 40 percent of runs at 10 percent CFA, and failed to find any path at 15 and 20 percent CFA [3]. Its paths were also the most winding of the three planners compared.

The opposite failure appeared early in flight: GESTALT was tuned after landing to overcome excessively conservative behavior at slope changes [1]. On the Meridiani plains the only geometric obstacles were occasional hollows from small infilled craters, and inside Eagle and Endurance craters the binding hazards were slope and slip rather than obstacles at all [1].

Approximate Clearance Evaluation

Section titled “Approximate Clearance Evaluation”

ACE replaces the point-plus-inflation model with bounds on the vehicle’s actual state. From a terrain height map it estimates the lowest and highest height each wheel may reach at a given pose, then derives conservative upper and lower bounds on body clearance, roll, pitch and rocker and bogie angles from those extreme wheel heights, without iterative optimization [3]. The bounds are guaranteed conservative, so a pose ACE calls safe is safe; the gain over GESTALT is that a rock in a wheel track is no longer automatically an obstacle, because the belly pan clearance is evaluated rather than assumed away.

PlatformConfigurationTime per pose evaluation
RAD750, as on Curiosity and Mars 202010 cm resolution DEM10 to 15 ms
NVIDIA Jetson TK1, Athena roversingle pose11.2 microseconds
NVIDIA Jetson TK1, plane fitting for comparisonabout 100 points26.1 microseconds
NVIDIA Jetson TK1, plane fitting for comparisonabout 200 points68.2 microseconds
Intel Core i7 at 2.8 GHzsingle pose2 microseconds

Source: [3]. The RAD750 figure is given as typical, with the note that precise timing is hard given the specialized configuration of the flight software. Constant execution time is the property that matters operationally: because ACE is closed form, its cost does not depend on terrain pattern, where an iterative method needs more iterations on rough ground.

Measured against the same 80 terrain benchmark, ACE-based planning held a success rate of at least 95 percent up to 15 percent CFA, against 40 percent for the GESTALT equivalent at 10 percent CFA, and its success rate fell off sharply at 20 percent CFA [3]. All three planners used the same depth-five tree search with 1.5 m edges and a collision check every 0.25 m; only the collision check differed.

ACE’s own failure mode is residual conservatism. In a Jezero crater simulation with rocks populated at 10 percent CFA, the bounds always contained the ground truth, with the span between upper and lower bound within a few degrees most of the time, but at one point the upper bound on pitch reached about 10 degrees against an actual 1 degree [3]. These false alarms occur when a large rock lies in a wheel box that the rover did not in fact step on [3]. ACE was validated on the Athena rover, MER sized, and on Scarecrow, the MSL mobility testbed, in the JPL Mars Yard.

Hazard detection for a lander rather than a rover

Section titled “Hazard detection for a lander rather than a rover”

A rover re-images and replans every driving cycle; a lander gets one hazard map and one decision, made while still descending. The ALHAT Hazard Detection System flew on five free flights of the Morpheus rocket lander at the Kennedy Space Center simulated lunar hazard field in 2014 [10]. It mosaics an elevation map from a gimbaled flash lidar in a single 6 second scan, then scores leg-pad, effective-slope and probabilistic roughness maps against the vehicle’s own tolerances of 10 degrees of slope and 0.4 m of roughness, and returns five ranked safe sites within 11 to 13 seconds of the start of the scan [10]. The flown map was 60 by 60 m from about 460 m slant range rather than the 90 by 90 m from 750 m the system was designed for, a concession to engine performance rather than to the sensor.

Hazard detection worked on every flight, always placing the safest ranked site on the landing pad, within 0.4 to 2.6 m of the surveyed pad center [10]. Navigation was the harder half. Hazard-relative navigation, tracking the chosen site through final approach, is what struggled: across all five flights only 0 to 4 of 16 to 18 tracking attempts produced a valid measurement, because the tracked feature drifted outside its search region once its own measurements were fed back into the vehicle’s navigation filter [10]. The map itself was never the weak link; closing the loop between the map and the vehicle’s own state estimate was.

The cost of hazard checking inside a planning cycle

Section titled “The cost of hazard checking inside a planning cycle”

On Perseverance, Enhanced AutoNav ranks candidate paths and calls ACE to check the top-ranked ones. The number of ACE calls per planning cycle is therefore the compute budget for hazard detection. The overthink threshold is 275 calls, about 3 to 4 s at 10 to 20 ms per call, and exceeding it means the highest-ranked paths were all rejected and the rover may have to stop driving before a solution is found [4]. Evaluating the entire baseline tree of 1694 candidate paths at 25 cm intervals would take more than 22,000 calls and over 3 minutes of computation, which is why the ranking exists at all. In Monte Carlo simulation on complex terrain, replacing the ranking heuristic with a learned one cut the overthink rate from 20.0 percent to 7.1 percent and path inefficiency from 25.4 to 20.4 percent, with no trial failure caused by a violated safety constraint in any experiment [4].

Families of approach and what has flown

Section titled “Families of approach and what has flown”
FamilyWhat it measuresStatusSource
Stereo geometry, goodness map (GESTALT)Plane fit and elevation extremes under a rover-sized patchFlown on MER[1][2]
Stereo plus vehicle-state bounds (ACE)Wheel heights, clearance, roll, pitch, suspension anglesFlown on Mars 2020; simulation and ground testbeds in the cited work[3][4]
Laser triangulationObstacle height from co-planar beamsGround-tested design, not flown[8]
Structured-light bumperMissing or displaced dots on one image rowField test on a ground rover[18]
Appearance and learned terrain classesPixel labels (sand, rock, bedrock) from imageryOffline evaluation[19][20][21]
Wheel-motion and vibration sensingSlip, sinkage, terrain type from proprioceptionSimulation and Earth field tests[26][27]
Passive imaging for landersStereo slope plus shadow-based rock detectionModel predictions and field imagery[14]
Lidar for landersElevation map, safe-site rankingFree flights on a terrestrial lander; helicopter campaigns[10][12][16][22]
Segmentation of rocks and craters from RGB-DPositive and negative lunar obstaclesRendered imagery only[28]
Self-supervised costmaps from camera and lidarBird’s-eye-view traversabilityEarth rover, Mars-like simulation[29]
Probabilistic slip traversabilitySlip mean and variance from color and elevationSynthetic maps only[30]
Monocular landing-site detection for rotorcraftRocks against map cell sizeSimulation and offline UAV data[31][32]
Multi-level voxel traversability graphsPer-direction tip-over, slip, chassis collisionQualitative demonstration[35]
Camera-based obstacle detection at 50 m altitudeObstacle check during powered descentFlown once, on SLIM[23]

Stereo implementation and evaluation

Section titled “Stereo implementation and evaluation”

The flight stereo chain that fed GESTALT was documented two years before MER landed: image pyramid reduction from 1024 by 1024 to 256 by 256, rectification, Difference of Gaussians bias removal, a one-dimensional correlator with a 7 by 7 window, and several filters, with timing reported for the processors of the day [11]. That paper measures speed and reports no obstacle detection or false alarm rate. The statistical framework the correlator descends from, a maximum-likelihood disparity estimate with a membrane smoothness prior, was fielded on the JPL Robby testbed with qualitative depth maps only [24].

Detection performance is better characterized on terrestrial vehicles. Seven independent stereo detectors (positive, negative, tree trunk, slope, range density, water, low overhang) run off line on imagery from a rail cart at 1 m/s found 20 of 21 obstacles on a 170 m course with no false detections, at maximum ranges from 11.6 m for water to 25.2 m for tall thin poles [13]. The course was a vegetated Army site and the zero false-positive rate came from conservative tuning.

Active sensing on rovers

Section titled “Active sensing on rovers”

Laser triangulation [8] is described above. A structured-light virtual bumper built for permanently shadowed lunar craters, where stereo has no texture, projects 11 dots from a 26 cm baseline, pitched 28 degrees down, so a single image row is searched for missing or shifted dots [18]. It was tested over pea gravel on the K-REX2 rover, not regolith, and the gravel’s own reflectance variation was the dominant error source [18].

Terrain appearance, proprioception and learned traversability

Section titled “Terrain appearance, proprioception and learned traversability”

Geometry does not see slip or hard embedded rock. After wheel damage on Curiosity peaked over sols 450 to 515, a random forest was trained on 66 hand-labeled Navcam images to label five wheel-relevant classes and feed a risk-aware planner, with no accuracy figure reported [19]. A retrained SPOC classifier reached 97.0 percent pixel accuracy on MSL imagery using the AI4Mars labels, and 84.2 percent on Mars 2020 after adaptation with about 70 images, evaluated offline [20]. A learned bird’s-eye-view traversability network, trained self-supervised on pseudo labels from JPL’s X-Racer stack, was scored only against those same labels [21]. Wheel-motion comparison across the rocker-bogie joints detects slip and sinkage without extra sensors, in simulation only [26]; a wheel-mounted vibration sensor with a neural network separated five outdoor surfaces at 0.2 to 0.6 m/s on Earth wheeled vehicles, not planetary hardware [27]. The terrain-adaptive stack of [5] combined goodness maps, triage and slip prediction, averaging about 21 percent slip error across five terrains, with its 4 s per meter forward simulation run offboard. See terrain classification.

Recent learned work stays off planetary hardware. LuSeg segments positive (rock) and negative (crater) lunar obstacles from RGB-D input and reports 92.79 percent mIoU on its own 5,143-pair rendered dataset and 85.25 percent on a terrestrial road dataset, at 17.45 ms per frame on a desktop RTX 4090; no lunar surface imagery was used [28]. A self-supervised bird’s-eye-view costmap predictor fusing camera and lidar, labeled from IMU response, was trained on Mars-like simulation built from HiRISE terrain and run on an Earth rover in forest and rocky terrain, with no Mars data and rigid-terrain simulation [29]. A ten-network ensemble predicting slip mean and variance from color and elevation maps, with few-shot test-time adaptation, was evaluated on 400 synthetic planning problems with no rover hardware [30]. A multi-level sparse-voxel map with a weighted traversability graph scores tip-over, slip, occlusion and chassis collision per direction and level for 3D planning on point clouds, and reports no path-length, timing or success-rate figures [35].

A reactive alternative, repulsive potential fields evaluated inside the servo loop, was formulated for manipulators and mobile robots, and its author states that local minima can stably hold the robot short of its goal [25]. It supplies no map-based hazard judgment.

Landers: passive imaging, lidar and flight

Section titled “Landers: passive imaging, lidar and flight”

Passive imaging for landing fuses stereo slope estimation with monocular shadow-based rock detection: 85 percent of rocks with shadows of five pixels or more were found at about three false alarms per image across sun angles of 30 to 70 degrees, with stereo slope error below 3 degrees at one condition and landing-probability gains given as model predictions [14]. Lidar was scored against surveyed truth in a rocket-sled test at about 60 m/s, with surface reconstruction of 0.063 m standard deviation near the hazards using truth position and attitude and 0.1 m with a flight-like IMU [12].

The ALHAT compute element paired an FPGA for I/O and timestamping with a 64-core Tilera processor, and its paper reports only static gimbal pointing, not detection performance [15]. Flight timing was met on the Morpheus flights: durations of 11.332 to 12.856 s against a 14.1 s requirement, with the 60 by 60 m map the largest the allocation permitted [16]. The Simple Safe Site Selection algorithm skips the elevation model and scores a single flash lidar range image; on five hazardous images the chosen sites had true safety probability of 0.716 to 0.912 against about 0.997 for the best available [17]. SPLICE hazard detection software, run on helicopter-collected lidar maps over a hazard field in March 2025, achieved true positive rates of 50 to 70 percent and false positive rates of 0.4 to 0.6 percent at downsampling steps of 5 and 7 pixels, mostly from post-processed replay [22]. For rotorcraft, where lidar is unaffordable in size, weight and power, a monocular structure-from-motion pipeline builds a multi-resolution elevation map and ranks landing sites; rocks are reliably detected once their diameter exceeds about three times the finest cell size, with 100 percent detection of 0.3 m rocks on 8 cm cells and 31.8 percent at 0.1 m, from a simulation with synthetic disparity noise and confirmed on offline UAV flight data [31]. The companion system paper targets a Mars Science Helicopter and a 1 Hz detection rate on Snapdragon-class avionics, not yet run on that hardware [32]. China’s Chang’e-3 landing of 14 December 2013 included an imaging-sensor hazard detection and avoidance subphase of the powered descent; the review describing it quotes flight numbers from mission papers it could not check [33]. Hazard detection quality is also bounded by the reference maps: a lunar south pole LOLA DEM at 5 m per pixel has about a 10 percent fill factor, and LRO orbit errors of roughly 7 by 7 by 0.5 m produce streaky artifacts, in a plan whose benchmark maps and validation are future work [34]. On SLIM, which landed on January 19 to 20, 2024, vision-based crater matching and a 50 m altitude obstacle check were exercised in flight [23].

What is not established

Section titled “What is not established”

The MER mode-distance breakdown that anchors GESTALT’s usage numbers is derived from odometry rather than measured ground truth, so the non-visual-odometry rows can be overestimates and the 25 percent autonomous-driving figure is a bound, not a measurement [9]. GESTALT’s own CFA-versus-success figures come from a MATLAB reimplementation over synthetic terrain, not the flight software over real Mars terrain [3]. ACE’s guarantee that a called-safe pose is actually safe was checked in one Jezero crater simulation and on two ground testbeds, not in flight, and its own worst observed conservatism, a 10 degree pitch bound against a 1 degree actual, has no flight counterpart on record [3]. The ALHAT lander system demonstrated hazard detection itself on five terrestrial flights, but hazard-relative navigation, the half meant to steer the vehicle to the chosen site, produced a valid tracking measurement on well under a quarter of its attempts across those flights, and the closed-loop interaction between that tracking and the vehicle’s own navigation filter is named as the cause without being fully characterized [10].

Outside the two rover and lander systems above, no family has a flight-derived detection rate. The stereo detector evaluation reports one 170 m terrestrial course [13]. The learned classifiers are scored offline or against their own pseudo labels, and the Curiosity wheel-hazard classifier reports no accuracy at all [19][21]. Lander lidar false-positive and true-positive rates come from helicopter replay of three post-processed maps from one field, and the S4 result rests on five hand-selected images [17][22]. The SLIM result is one mission [23]. The rotorcraft detection rates are simulated or offline, and the learned segmentation and costmap results above rest on rendered, synthetic or Earth data [28][29][30][31].

References

  1. Maimone, M., Johnson, A., Cheng, Y., Willson, R. and Matthies, L. (2004). Autonomous Navigation Results from the Mars Exploration Rover (MER) Mission . International Symposium on Experimental Robotics. Source
    BibTeX
    @inproceedings{maimone2004autonomous,
  title = {Autonomous Navigation Results from the Mars Exploration Rover (MER) Mission},
  author = {Maimone, Mark and Johnson, Andrew and Cheng, Yang and Willson, Reg and Matthies, Larry},
  booktitle = {International Symposium on Experimental Robotics},
  address = {Singapore},
  year = {2004},
  url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/41077}
}
  2. Maimone, M. W., Leger, P. C. and Biesiadecki, J. J. (2007). Overview of the Mars Exploration Rovers' Autonomous Mobility and Vision Capabilities . IEEE International Conference on Robotics and Automation, Space Robotics Workshop. Source
    BibTeX
    @inproceedings{maimone2007overview,
  title = {Overview of the Mars Exploration Rovers' Autonomous Mobility and Vision Capabilities},
  author = {Maimone, Mark W. and Leger, P. Chris and Biesiadecki, Jeffrey J.},
  booktitle = {IEEE International Conference on Robotics and Automation, Space Robotics Workshop},
  address = {Rome, Italy},
  year = {2007},
  url = {https://www-robotics.jpl.nasa.gov/media/documents/mer_autonomy_icra_2007.pdf}
}
  3. Otsu, K., Matheron, G., Ghosh, S., Toupet, O. and Ono, M. (2020). Fast Approximate Clearance Evaluation for Rovers with Articulated Suspension Systems . Journal of Field Robotics, 5. Source
    BibTeX
    @article{otsu2020fast,
  title = {Fast Approximate Clearance Evaluation for Rovers with Articulated Suspension Systems},
  author = {Otsu, Kyohei and Matheron, Guillaume and Ghosh, Sourish and Toupet, Olivier and Ono, Masahiro},
  journal = {Journal of Field Robotics},
  volume = {37},
  number = {5},
  pages = {768--785},
  year = {2020},
  doi = {10.1002/rob.21892},
  abstract = {Abstract We present a light‐weight body‐terrain clearance evaluation algorithm for the automated path planning of NASA's Mars 2020 rover. Extraterrestrial path planning is challenging due to the combination of terrain roughness and severe limitation in computational resources. Path planning on cluttered and/or uneven terrains requires repeated safety checks on all the candidate paths at a small interval. Predicting the future rover state requires simulating the vehicle settling on the terrain, which involves an inverse‐kinematics problem with iterative nonlinear optimization under geometric constraints. However, such expensive computation is intractable for slow spacecraft computers, such as RAD750, which is used by the Curiosity Mars rover and upcoming Mars 2020 rover. We propose the approximate clearance evaluation (ACE) algorithm, which obtains conservative bounds on vehicle clearance, attitude, and suspension angles without iterative computation. It obtains those bounds by estimating the lowest and highest heights that each wheel may reach given the underlying terrain, and calculating the worst‐case vehicle configuration associated with those extreme wheel heights. The bounds are guaranteed to be conservative, hence ensuring vehicle safety during autonomous navigation. ACE is planned to be used as part of the new onboard path planner of the Mars 2020 rover. This paper describes the algorithm in detail and validates our claim of conservatism and fast computation through experiments.}
}
  4. Abcouwer, N., Daftry, S., Del Sesto, T., Toupet, O., Ono, M., Venkatraman, S., Lanka, R., Song, J. and Yue, Y. (2021). Machine Learning Based Path Planning for Improved Rover Navigation . IEEE Aerospace Conference. Source
    BibTeX
    @inproceedings{abcouwer2021machine,
  title = {Machine Learning Based Path Planning for Improved Rover Navigation},
  author = {Abcouwer, Neil and Daftry, Shreyansh and Del Sesto, Tyler and Toupet, Olivier and Ono, Masahiro and Venkatraman, Siddarth and Lanka, Ravi and Song, Jialin and Yue, Yisong},
  booktitle = {IEEE Aerospace Conference},
  pages = {1-9},
  address = {Big Sky, Montana},
  year = {2021},
  doi = {10.1109/aero50100.2021.9438337},
  abstract = {Enhanced AutoNav (ENav), the baseline surface navigation software for NASA's Perseverance rover, sorts a list of candidate paths for the rover to traverse, then uses the Approximate Clearance Evaluation (ACE) algorithm to evaluate whether the most highly ranked paths are safe. ACE is crucial for maintaining the safety of the rover, but is computationally expensive. If the most promising candidates in the list of paths are all found to be infeasible, ENav must continue to search the list and run time-consuming ACE evaluations until a feasible path is found. In this paper, we present two heuristics that, given a terrain heightmap around the rover, produce cost estimates that more effectively rank the candidate paths before ACE evaluation. The first heuristic uses Sobel operators and convolution to incorporate the cost of traversing high-gradient terrain. The second heuristic uses a machine learning (ML) model to predict areas that will be deemed untraversable by ACE. We used physics simulations to collect training data for the ML model and to run Monte Carlo trials to quantify navigation performance across a variety of terrains with various slopes and rock distributions. Compared to ENav's baseline performance, integrating the heuristics can lead to a significant reduction in ACE evaluations and average computation time per planning cycle, increase path efficiency, and maintain or improve the rate of successful traverses. This strategy of targeting specific bottlenecks with ML while maintaining the original ACE safety checks provides an example of how ML can be infused into planetary science missions and other safety-critical software.}
}
  5. Helmick, D. M., Angelova, A., Livianu, M. and Matthies, L. H. (2007). Terrain Adaptive Navigation for Mars Rovers . IEEE Aerospace Conference. Source
    BibTeX
    @inproceedings{helmick2007terrain,
  title = {Terrain Adaptive Navigation for Mars Rovers},
  author = {Helmick, Daniel M. and Angelova, Anelia and Livianu, Matthew and Matthies, Larry H.},
  booktitle = {IEEE Aerospace Conference},
  pages = {1-11},
  address = {Big Sky, Montana},
  year = {2007},
  doi = {10.1109/aero.2007.352684},
  abstract = {A navigation system for Mars rovers in very rough terrain has been designed, implemented, and tested on a research rover in Mars analog terrain. This navigation system consists of several technologies that are integrated to increase the capabilities compared to current rover navigation algorithms. These technologies include: goodness maps and terrain triage, terrain classification, remote slip prediction, path planning, high-fidelity traversability analysis (HFTA), and slip-compensated path following. The focus of this paper is not on the component technologies, but rather on the integration of these components. Results from the onboard integration of several of the key technologies described here are shown. Additionally, the results from independent demonstrations of several of these technologies are shown. Future work will include the demonstration of the entire integrated system.}
}
  6. Gennery, D. B. (1980). Modelling the Environment of an Exploring Vehicle by Means of Stereo Vision, AIM-339 / STAN-CS-80-805. Source
    BibTeX
    @phdthesis{gennery1980modelling,
  title = {Modelling the Environment of an Exploring Vehicle by Means of Stereo Vision},
  author = {Gennery, Donald B.},
  number = {AIM-339 / STAN-CS-80-805},
  school = {Stanford University},
  type = {Ph.D. dissertation},
  year = {1980},
  url = {https://apps.dtic.mil/sti/citations/ADA091081}
}
  7. Moravec, H. P. (1980). Obstacle Avoidance and Navigation in the Real World by a Seeing Robot Rover . Carnegie Mellon University, CMU-RI-TR-3 / STAN-CS-80-813 / AIM-340. Source
    BibTeX
    @phdthesis{moravec1980obstacle,
  title = {Obstacle Avoidance and Navigation in the Real World by a Seeing Robot Rover},
  author = {Moravec, Hans P.},
  journal = {Carnegie Mellon University},
  number = {CMU-RI-TR-3 / STAN-CS-80-813 / AIM-340},
  school = {Stanford University},
  type = {Ph.D. dissertation},
  year = {1980},
  doi = {10.1184/r1/6557033.v1}
}
  8. Matthies, L., Balch, T. and Wilcox, B. (1997). Fast Optical Hazard Detection for Planetary Rovers Using Multiple Spot Laser Triangulation . International Conference on Robotics and Automation. Source
    BibTeX
    @inproceedings{matthies1997fast,
  title = {Fast Optical Hazard Detection for Planetary Rovers Using Multiple Spot Laser Triangulation},
  author = {Matthies, L. and Balch, T. and Wilcox, B.},
  booktitle = {International Conference on Robotics and Automation},
  volume = {1},
  pages = {859-866},
  publisher = {IEEE},
  year = {1997},
  doi = {10.1109/robot.1997.620142},
  abstract = {A new laser-based optical sensor system that provides hazard detection for planetary rovers is presented. The sensor can support safe travel at speeds up to 12 cm/second for large (1 m) rovers in full sunlight on Earth or Mars. This is at least a 5 times improvement over the sensor aboard NASA's Mars Pathfinder rover. The system overcomes limitations in the older design that require image differencing to detect a laser stripe in full sun. The new system ensures the projected laser light is detectable in a single image, eliminating the requirement for additional difference images. The improvement is significant since any reduction in image gathering or processing time provides for faster rover motion. The savings are even more important in the case of a Mars rover since power and radiation-hardening requirements lead to severely constrained computational resources. The paper includes a thorough discussion of design details and tradeoffs for optical hazard sensing that will benefit future efforts in this area.}
}
  9. Biesiadecki, J. J. and Maimone, M. W. (2006). The Mars Exploration Rover surface mobility flight software : driving ambition . IEEE Aerospace Conference. Source
    BibTeX
    @inproceedings{biesiadecki2006mars,
  title = {The Mars Exploration Rover surface mobility flight software : driving ambition},
  author = {Biesiadecki, Jeffrey J. and Maimone, Mark W.},
  booktitle = {IEEE Aerospace Conference},
  pages = {1-15},
  publisher = {IEEE},
  year = {2006},
  doi = {10.1109/aero.2006.1655723},
  abstract = {NASA's Mars exploration rovers' (MER) onboard mobility flight software was designed to provide robust and flexible operation. The MER vehicles can be commanded directly, or given autonomous control over multiple aspects of mobility: which motions to drive, measurement of actual motion, terrain interpretation, even the selection of targets of interest (although this mode remains largely underused). Vehicle motion can be commanded using multiple layers of control: motor control, direct drive operations (arc, turn in place), and goal-based driving (goto waypoint). Multiple layers of safety checks ensure vehicle performance: command limits (command timeout, time of day limit, software enable, activity constraints), reactive checks (e.g., motor current limit, vehicle tilt limit), and predictive checks (e.g., step, tilt, roughness hazards). From January 2004 through October 2005, Spirit accumulated over 5000 meters and Opportunity 6000 meters of odometry, often covering more than 100 meters in a single day. In this paper we describe the software that has driven these rovers more than a combined 11,000 meters over the Martian surface, including its design and implementation, and summarize current mobility performance results from Mars}
}
  10. Trawny, N., Huertas, A., Luna, M. E., Villalpando, C. Y., Martin, K. E., Carson, J. M., Johnson, A. E., Restrepo, C. and Roback, V. E. (2015). Flight testing a real-time Hazard Detection System for safe lunar landing on the rocket-powered Morpheus vehicle . AIAA Guidance, Navigation, and Control Conference. Source
    BibTeX
    @inproceedings{trawny2015flight,
  title = {Flight testing a real-time Hazard Detection System for safe lunar landing on the rocket-powered Morpheus vehicle},
  author = {Trawny, Nikolas and Huertas, Andres and Luna, Michael E. and Villalpando, Carlos Y. and Martin, Keith E. and Carson, John M. and Johnson, Andrew E. and Restrepo, Carolina and Roback, Vincent E.},
  booktitle = {AIAA Guidance, Navigation, and Control Conference},
  publisher = {American Institute of Aeronautics and Astronautics},
  year = {2015},
  doi = {10.2514/6.2015-0326},
  abstract = {The Hazard Detection System (HDS) is a component of the ALHAT (Autonomous Landing and Hazard Avoidance Technology) sensor suite, which together provide a lander Guidance, Navigation and Control (GN&C) system with the relevant measurements necessary to enable safe precision landing under any lighting conditions. The HDS consists of a stand-alone compute element (CE), an Inertial Measurement Unit (IMU), and a gimbaled flash LIDAR sensor that are used, in real-time, to generate a Digital Elevation Map (DEM) of the landing terrain, detect candidate safe landing sites for the vehicle through Hazard Detection (HD), and generate hazard-relative navigation (HRN) measurements used for safe precision landing. Following an extensive ground and helicopter test campaign, ALHAT was integrated onto the Morpheus rocket-powered terrestrial test vehicle in March 2014. Morpheus and ALHAT then performed five successful free flights at the simulated lunar hazard field constructed at the Shuttle Landing Facility (SLF) at Kennedy Space Center, for the first time testing the full system on a lunar-like approach geometry in a relevant dynamic environment. During these flights, the HDS successfully generated DEMs, correctly identified safe landing sites and provided HRN measurements to the vehicle, marking the first autonomous landing of a NASA rocket-powered vehicle in hazardous terrain. This paper provides a brief overview of the HDS architecture and describes its in-flight performance.}
}
  11. Goldberg, S. B., Maimone, M. W. and Matthies, L. (2002). Stereo Vision and Rover Navigation Software for Planetary Exploration . IEEE Aerospace Conference. Source
    BibTeX
    @inproceedings{goldberg2002stereo,
  title = {Stereo Vision and Rover Navigation Software for Planetary Exploration},
  author = {Goldberg, Steven B. and Maimone, Mark W. and Matthies, Larry},
  booktitle = {IEEE Aerospace Conference},
  volume = {5},
  pages = {2025--2036},
  address = {Big Sky, Montana},
  year = {2002},
  doi = {10.1109/aero.2002.1035370},
  abstract = {NASA's Mars Exploration Rover (MER) missions will land twin rovers on the surface of Mars in 2004. These rovers will have the ability to navigate safely through unknown and potentially hazardous terrain, using autonomous passive stereo vision to detect potential terrain hazards before driving into them. Unfortunately, the computational power of currently available radiation hardened processors limits the amount of distance (and therefore science) that can be safely achieved by any rover in a given time frame. We present overviews of our current rover vision and navigation systems, to provide context for the types of computation that are required to navigate safely. We also present baseline timing results that represent a lower bound in achievable performance (useful for systems engineering studies of future missions), and describe ways to improve that performance using commercial grade (as opposed to radiation hardened) processors. In particular, we document speedups to our stereo vision system that were achieved using the vectorized operations provided by Pentium MMX technology. Timing data were derived from implementations on several platforms: a prototype Mars rover with flight-like electronics (the Athena Software Development Model (SDM) rover), a RAD6000 computing platform (as will be used in the 2003 MER missions), and research platforms with commercial Pentium III and Sparc processors. Finally, we summarize the radiation effects analysis that suggests that commercial grade processors are likely to be adequate for Mars surface missions, and discuss the level of speedup that may accrue from using these instead of radiation hardened parts.}
}
  12. Johnson, A. E. and Skulsky, E. D. (2002). Descent-speed testing of a hazard detection system for safe landing on Mars . Nuclear and Emerging Technologies for Space. Source
    BibTeX
    @inproceedings{johnson2002descent,
  title = {Descent-speed testing of a hazard detection system for safe landing on Mars},
  author = {Johnson, A. E. and Skulsky, E. D.},
  booktitle = {Nuclear and Emerging Technologies for Space},
  publisher = {JPL Open Repository},
  address = {Breckenridge, Colorado},
  year = {2002},
  url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/37089}
}
  13. Rankin, A., Huertas, A. and Matthies, L. (2005). Evaluation of stereo vision obstacle detection algorithms for off-road autonomous navigation . AUVSIs Unmanned Systems North America. Source
    BibTeX
    @inproceedings{rankin2005evaluation,
  title = {Evaluation of stereo vision obstacle detection algorithms for off-road autonomous navigation},
  author = {Rankin, Arturo and Huertas, Andres and Matthies, Larry},
  booktitle = {AUVSIs Unmanned Systems North America},
  publisher = {JPL Open Repository},
  year = {2005},
  url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/38362}
}
  14. Huertas, A., Cheng, Y. and Matthies, L. H. (2008). Automatic hazard detection for landers . International Symposium on Artificial Intelligence, R and A in Space. Source
    BibTeX
    @inproceedings{huertas2008automatic,
  title = {Automatic hazard detection for landers},
  author = {Huertas, Andres and Cheng, Yang and Matthies, Larry H.},
  booktitle = {International Symposium on Artificial Intelligence, R and A in Space},
  publisher = {JPL Open Repository},
  year = {2008},
  url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/41368}
}
  15. Villalpando, C. Y., Werner, R. A., Carson III, J. M., Khanoyan, G., Stern, R. A. and Trawny, N. (2013). A hybrid FPGA/Tilera compute element for autonomous hazard detection and navigation . IEEE Aerospace Conference. Source
    BibTeX
    @inproceedings{villalpando2013hybrid,
  title = {A hybrid FPGA/Tilera compute element for autonomous hazard detection and navigation},
  author = {Villalpando, Carlos Y. and Werner, Robert A. and Carson III, John M. and Khanoyan, Garen and Stern, Ryan A. and Trawny, Nikolas},
  booktitle = {IEEE Aerospace Conference},
  pages = {1--9},
  publisher = {IEEE},
  year = {2013},
  doi = {10.1109/aero.2013.6496977},
  abstract = {To increase safety for future missions landing on other planetary or lunar bodies, the Autonomous Landing and Hazard Avoidance Technology (ALHAT) program is developing an integrated sensor for autonomous surface analysis and hazard determination. The ALHAT Hazard Detection System (HDS) consists of a Flash LIDAR for measuring the topography of the landing site, a gimbal to scan across the terrain, and an Inertial Measurement Unit (IMU), along with terrain analysis algorithms to identify the landing site and the local hazards. An FPGA and Manycore processor system was developed to interface all the devices in the HDS, to provide high-resolution timing to accurately measure system state, and to run the surface analysis algorithms quickly and efficiently. In this paper, we will describe how we integrated COTS components such as an FPGA evaluation board, a TILExpress64, and multi-threaded/multi-core aware software to build the HDS Compute Element (HDSCE). The ALHAT program is also working with the NASA Morpheus Project and has integrated the HDS as a sensor on the Morpheus Lander. This paper will also describe how the HDS is integrated with the Morpheus lander and the results of the initial test flights with the HDS installed. We will also describe future improvements to the HDSCE.}
}
  16. Luna, M. E., Huertas, A., Trawny, N., Villalpando, C. Y., Martin, K. E., Wilson, W. and Restrepo, C. I. (2015). Simulations of the Hazard Detection System for approach trajectories of the Morpheus Lunar Lander . AIAA Guidance, Navigation, and Control Conference. Source
    BibTeX
    @inproceedings{luna2015simulations,
  title = {Simulations of the Hazard Detection System for approach trajectories of the Morpheus Lunar Lander},
  author = {Luna, Michael E. and Huertas, Andres and Trawny, Nikolas and Villalpando, Carlos Y. and Martin, Keith E. and Wilson, William and Restrepo, Carolina I.},
  booktitle = {AIAA Guidance, Navigation, and Control Conference},
  publisher = {American Institute of Aeronautics and Astronautics},
  year = {2015},
  doi = {10.2514/6.2015-0327}
}
  17. Luna, M. E., Almeida, E., Spiers, G., Villalpando, C. Y., Johnson, A. E. and Trawny, N. (2017). Evaluation of the Simple Safe Site Selection (S4) hazard detection algorithm using helicopter field test data . AIAA Guidance, Navigation, and Control Conference. Source
    BibTeX
    @inproceedings{luna2017evaluation,
  title = {Evaluation of the Simple Safe Site Selection (S4) hazard detection algorithm using helicopter field test data},
  author = {Luna, Michael E. and Almeida, Eduardo and Spiers, Gary and Villalpando, Carlos Y. and Johnson, Andrew E. and Trawny, Nikolas},
  booktitle = {AIAA Guidance, Navigation, and Control Conference},
  publisher = {American Institute of Aeronautics and Astronautics},
  year = {2017},
  doi = {10.2514/6.2017-1497}
}
  18. Nefian, A. V., Wong, U. Y., Dille, M., Bouyssounouse, X., Edwards, L. and To, V. (2017). Structured Light-Based Hazard Detection for Planetary Surface Navigation . IEEE Winter Conference on Applications of Computer Vision, 20170009200. Source
    BibTeX
    @inproceedings{nefian2017structured,
  title = {Structured Light-Based Hazard Detection for Planetary Surface Navigation},
  author = {Nefian, Ara V. and Wong, Uland Y. and Dille, Michael and Bouyssounouse, Xavier and Edwards, Laurence and To, Vinh},
  booktitle = {IEEE Winter Conference on Applications of Computer Vision},
  number = {20170009200},
  pages = {2665-2671},
  institution = {NASA Ames Research Center},
  year = {2017},
  doi = {10.1109/iros.2017.8206090},
  abstract = {This paper describes a structured light-based sensor for hazard avoidance in planetary environments. The system presented here can also be used in terrestrial applications constrained by reduced onboard power and computational complexity and low illumination conditions. The sensor consists on a calibrated camera and laser dot projector system. The onboard hazard avoidance system determines the position of the projected dots in the image and through a triangulation process detects potential hazards. The paper presents the design parameters for this sensor and describes the image based solution for hazard avoidance. The system presented here was tested extensively in day and night conditions in Lunar analogue environments. The current system achieves over 97% detection rate with 1.7% false alarms over 2000 images.}
}
  19. Ono, M., Fuchs, T. J., Steffy, A., Maimone, M. and Yen, J. (2015). Risk-aware planetary rover operation: autonomous terrain classification and path planning . IEEE Aerospace Conference. Source
    BibTeX
    @inproceedings{ono2015risk,
  title = {Risk-aware planetary rover operation: autonomous terrain classification and path planning},
  author = {Ono, Masahiro and Fuchs, Thoams J. and Steffy, Amanda and Maimone, Mark and Yen, Jeng},
  booktitle = {IEEE Aerospace Conference},
  pages = {1-10},
  publisher = {IEEE},
  year = {2015},
  doi = {10.1109/aero.2015.7119022},
  abstract = {Identifying and avoiding terrain hazards (e.g., soft soil and pointy embedded rocks) are crucial for the safety of planetary rovers. This paper presents a newly developed ground-based Mars rover operation tool that mitigates risks from terrain by automatically identifying hazards on the terrain, evaluating their risks, and suggesting operators safe paths options that avoids potential risks while achieving specified goals. The tool will bring benefits to rover operations by reducing operation cost, by reducing cognitive load of rover operators, by preventing human errors, and most importantly, by significantly reducing the risk of the loss of rovers. The risk-aware rover operation tool is built upon two technologies. The first technology is a machine learning-based terrain classification that is capable of identifying potential hazards, such as pointy rocks and soft terrains, from images. The second technology is a risk-aware path planner based on rapidly-exploring random graph (RRG) and the A* search algorithms, which is capable of avoiding hazards identified by the terrain classifier with explicitly considering wheel placement. We demonstrate the integrated capability of the proposed risk-aware rover operation tool by using the images taken by the Curiosity rover.}
}
  20. Atha, D., Swan, R. M., Didier, A., Hasnain, Z. and Ono, M. (2022). Multi-Mission Terrain Classifier for Safe Rover Navigation and Automated Science . IEEE Aerospace Conference. Source
    BibTeX
    @inproceedings{atha2022multi,
  title = {Multi-Mission Terrain Classifier for Safe Rover Navigation and Automated Science},
  author = {Atha, Deegan and Swan, R. Michael and Didier, Annie and Hasnain, Zaki and Ono, Masahiro},
  booktitle = {IEEE Aerospace Conference},
  pages = {1-13},
  publisher = {IEEE},
  year = {2022},
  doi = {10.1109/aero53065.2022.9843615},
  abstract = {We previously presented Soil Property and Object Classification (SPOC), a machine learning-based terrain classi-fier for Mars rovers, for automatically segmenting rover images by its surface type such as sand and bedrock. This paper presents a number of practical improvements to pave the way for potential future onboard deployment. First, we achieved 97.0 % overall pixel accuracy, evaluated against the classification generated by human experts on images from Mars Science Lab-oratory (MSL) missions. The substantial increase in accuracy was primarily enabled by the sheer volume of data used for training; we created a new large-scale dataset of Martian terrain labels, namely AI4Mars, which contains more than 400k labels contributed by citizen scientists for ~50k images taken by the Mars Exploration Rovers (MER) and Mars Science Laboratory (MSL) rover. Second, we demonstrated that SPOC can quickly adapt to a new mission landed on a previously unseen site. Specifically, we pretrained a model with MER and MSL data from the AI4Mars dataset and then adapted to the Mars 2020 Rover (M2020) by feeding a small volume of data between Sol 0 and 157; the adapted model was tested on Sol 200–203 and resulted in 84.2 % overall pixel accuracy and 93.4 % reliability (recall) for detecting sand, the most concerning class for rover's traversability. Third, we found that pretraining can substan-tially mitigate the decline of accuracy over time. We showed that the performance of a SPOC model pretrained with the ImageNet dataset and then trained by MSL images only up to Sol 390 remains comparable to a model trained by images up to Sol 1689 on the test data after Sol 1689. Fourth, we reimplemented SPOC with a light-weight convolutional neural network (CNN), MobileNetV2, which typically runs within tens of milliseconds (ms) on mobile processors such as Qualcomm's Snapdragon. Finally, we released the AI4Mars dataset to the public to encourage open innovation.}
}
  21. Frey, J., Patel, M., Atha, D., Nubert, J., Fan, D., Agha, A., Padgett, C., Spieler, P., Hutter, M. and Khattak, S. (2024). RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving . arXiv. Source
    BibTeX
    @article{frey2024roadrunner,
  title = {RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving},
  author = {Frey, Jonas and Patel, Manthan and Atha, Deegan and Nubert, Julian and Fan, David and Agha, Ali and Padgett, Curtis and Spieler, Patrick and Hutter, Marco and Khattak, Shehryar},
  journal = {arXiv},
  year = {2024},
  doi = {10.1109/tfr.2024.3464369},
  abstract = {Autonomous navigation at high speeds in off-road environments necessitates robots to comprehensively understand their surroundings using onboard sensing only. The extreme conditions posed by the off-road setting can cause degraded camera image quality due to poor lighting and motion blur, as well as limited sparse geometric information available from LiDAR sensing when driving at high speeds. In this work, we present RoadRunner, a novel framework capable of predicting terrain traversability and an elevation map directly from camera and LiDAR sensor inputs. RoadRunner enables reliable autonomous navigation, by fusing sensory information, handling of uncertainty, and generation of contextually informed predictions about the geometry and traversability of the terrain while operating at low latency. In contrast to existing methods relying on classifying handcrafted semantic classes and using heuristics to predict traversability costs, our method is trained end-to-end in a self-supervised fashion. The RoadRunner network architecture builds upon popular sensor fusion network architectures from the autonomous driving domain, which embed LiDAR and camera information into a common Bird's Eye View perspective. Training is enabled by utilizing an existing traversability estimation stack to generate training data in hindsight in a scalable manner from real-world off-road driving datasets. Furthermore, RoadRunner improves the system latency by a factor of roughly 4, from 500 ms to 140 ms, while improving the accuracy for traversability costs and elevation map predictions. We demonstrate the effectiveness of RoadRunner in enabling safe and reliable off-road navigation at high speeds in multiple real-world driving scenarios through unstructured desert environments.}
}
  22. Chen, P.-T., Liu, J. Y., Bocchino, R. L. J., Mendeck, G. F. and Vaughan, G. M. (2026). Hazard Detection Flight Software for Processing LIDAR Generated DEM . AIAA SciTech. Source
    BibTeX
    @inproceedings{chen2026hazard,
  title = {Hazard Detection Flight Software for Processing LIDAR Generated DEM},
  author = {Chen, Po-Ting and Liu, John Y. and Bocchino, Robert L., Jr. and Mendeck, Gavin F. and Vaughan, Geoffrey M.},
  booktitle = {AIAA SciTech},
  publisher = {JPL Open Repository},
  year = {2026},
  doi = {10.48577/jpl.d3e9iv},
  abstract = {Hazard Detection (HD) is an enabling technology for safe landing on planetary bodies when the lander hazard tolerance capability is at the resolution limit of orbital imagery. This situation is common to many planetary bodies in the Solar System including Moon and Mars. At these locations, an autonomous lander can utilize a Light Detection and Ranging (LIDAR) sensor to map the landing area and construct a Digital Elevation Map (DEM) in real-time. The HD software developed by the Safe and Precise Landing Integrated Capabilities Evolution (SPLICE) project is designed to process a DEM and identify safe landing locations in real-time. The hazard detection method is applicable to 4-legged lander geometry and is adaptable to different hazard tolerance and run time requirements. This paper discusses the hazard detection performance of the SPLICE HD software using DEMs generated by the SPLICE Hazard Detection LIDAR.}
}
  23. Ishida, T., Fukuda, S., Kariya, K., Kamata, H., Takadama, K., Kojima, H., Sawai, S. and Sakai, S. (2025). Vision-based navigation and obstacle detection flight results in SLIM lunar landing . Acta Astronautica. Source
    BibTeX
    @article{ishida2025vision,
  title = {Vision-based navigation and obstacle detection flight results in SLIM lunar landing},
  author = {Ishida, Takayuki and Fukuda, Seisuke and Kariya, Kazuki and Kamata, Hiroyuki and Takadama, Keiki and Kojima, Hirohisa and Sawai, Shujiro and Sakai, Shinichiro},
  journal = {Acta Astronautica},
  volume = {226},
  pages = {772-781},
  publisher = {Elsevier BV},
  year = {2025},
  doi = {10.1016/j.actaastro.2024.11.002},
  abstract = {On January 20, 2024, Smart Lander for Investigating Moon (SLIM) landed on the Moon. Vision-Based Navigation (VBN) was used to estimate the position of the spacecraft accurately and autonomously during the descent phase, and it was successfully used in each of the seven regions. Obstacle detection was also performed 50 m above the lunar surface, successfully identifying safe points within the field of view of the navigation camera. As a result, SLIM was the first mission to realize pinpoint lunar landing technology with 100 m accuracy. This paper details the VBN operational concept and developed components, as well as the flight results of VBN in SLIM’s lunar landing operations. • SLIM successfully landed on the Moon, achieving the first pinpoint landing. • Vision-based navigation provided precise position information to the probe. • Obstacle detection successfully identified a safer site within the field of view. • The landing accuracy 50 m above the lunar surface was 3 to 4 m.}
}
  24. Matthies, L. H. (1992). Stereo vision for planetary rovers: Stochastic modeling to near real-time implementation . International Journal of Computer Vision, 1. Source
    BibTeX
    @article{matthies1992stereo,
  title = {Stereo vision for planetary rovers: Stochastic modeling to near real-time implementation},
  author = {Matthies, Larry H.},
  journal = {International Journal of Computer Vision},
  volume = {8},
  number = {1},
  pages = {71-91},
  year = {1992},
  doi = {10.1007/bf00126401},
  abstract = {JPL has achieved the first autonomous cross-country robotic traverses to use stereo vision, with all computing onboard the vehicle. This paper describes the stereo vision system, including the underlying statistical model and the details of the implementation. It is argued that the overall approach provides a unifying paradigm for practical domain-independent stereo ranging.}
}
  25. Khatib, O. (1986). Real-Time Obstacle Avoidance for Manipulators and Mobile Robots . Earth and Space Science, 1. Source
    BibTeX
    @article{khatib1986real,
  title = {Real-Time Obstacle Avoidance for Manipulators and Mobile Robots},
  author = {Khatib, Oussama},
  journal = {Earth and Space Science},
  volume = {5},
  number = {1},
  pages = {90-98},
  publisher = {SAGE Publications},
  year = {1986},
  doi = {10.1177/027836498600500106},
  abstract = {This paper presents a unique real-time obstacle avoidance approach for manipulators and mobile robots based on the artificial potential field concept. Collision avoidance, tradi tionally considered a high level planning problem, can be effectively distributed between different levels of control, al lowing real-time robot operations in a complex environment. This method has been extended to moving obstacles by using a time-varying artificial patential field. We have applied this obstacle avoidance scheme to robot arm mechanisms and have used a new approach to the general problem of real-time manipulator control. We reformulated the manipulator con trol problem as direct control of manipulator motion in oper ational space—the space in which the task is originally described—rather than as control of the task's corresponding joint space motion obtained only after geometric and kine matic transformation. Outside the obstacles' regions of influ ence, we caused the end effector to move in a straight line with an upper speed limit. The artificial potential field ap proach has been extended to collision avoidance for all ma nipulator links. In addition, a joint space artificial potential field is used to satisfy the manipulator internal joint con straints. This method has been implemented in the COSMOS system for a PUMA 560 robot. Real-time collision avoidance demonstrations on moving obstacles have been performed by using visual sensing.}
}
  26. Wilcox, B. (1994). Non-Geometric Hazard Detection for a Mars Microrover. ntrs.nasa.gov/citations/19950005120
    BibTeX
    @misc{wilcox1994non,
  title = {Non-Geometric Hazard Detection for a Mars Microrover},
  author = {Wilcox, B.},
  year = {1994},
  url = {https://ntrs.nasa.gov/citations/19950005120},
  abstract = {The sinkage and slippage detection methodologies employed by a MESUR Pathfinder microrover, Rocky 3.2, are discussed. Results from a simulation are presented.}
}
  27. Bai, C., Guo, J., Guo, L. and Song, J. (2019). Deep Multi-Layer Perception Based Terrain Classification for Planetary Exploration Rovers . Sensors, 14. Source
    BibTeX
    @article{bai2019deep,
  title = {Deep Multi-Layer Perception Based Terrain Classification for Planetary Exploration Rovers},
  author = {Bai, Chengchao and Guo, Jifeng and Guo, Linli and Song, Junlin},
  journal = {Sensors},
  volume = {19},
  number = {14},
  pages = {3102},
  publisher = {MDPI AG},
  year = {2019},
  doi = {10.3390/s19143102},
  abstract = {Accurate classification and identification of the detected terrain is the basis for the long-distance patrol mission of the planetary rover. But terrain measurement based on vision and radar is subject to conditions such as light changes and dust storms. In this paper, under the premise of not increasing the sensor load of the existing rover, a terrain classification and recognition method based on vibration is proposed. Firstly, the time-frequency domain transformation of vibration information is realized by fast Fourier transform (FFT), and the characteristic representation of vibration information is given. Secondly, a deep neural network based on multi-layer perception is designed to realize classification of different terrains. Finally, combined with the Jackal unmanned vehicle platform, the XQ unmanned vehicle platform, and the vibration sensor, the terrain classification comparison test based on five different terrains was completed. The results show that the proposed algorithm has higher classification accuracy, and different platforms and running speeds have certain influence on the terrain classification at the same time, which provides support for subsequent practical applications.}
}
  28. Jiao, S., Zeng, Z., Su, Z., Chen, X., Zhou, Z. and Lu, H. (2025). LuSeg: Efficient Negative and Positive Obstacles Segmentation via Contrast-Driven Multi-Modal Feature Fusion on the Lunar . IEEE/RSJ International Conference on Intelligent Robots and Systems. Source
    BibTeX
    @inproceedings{jiao2025luseg,
  title = {LuSeg: Efficient Negative and Positive Obstacles Segmentation via Contrast-Driven Multi-Modal Feature Fusion on the Lunar},
  author = {Jiao, Shuaifeng and Zeng, Zhiwen and Su, Zhuoqun and Chen, Xieyuanli and Zhou, Zongtan and Lu, Huimin},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  pages = {5962-5969},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246916},
  abstract = {As lunar exploration missions grow increasingly complex, ensuring safe and autonomous rover-based surface exploration has become one of the key challenges in lunar exploration tasks. In this work, we have developed a lunar surface simulation system called the Lunar Exploration Simulator System (LESS) and the LunarSeg dataset, which provides RGB-D data for lunar obstacle segmentation that includes both positive and negative obstacles. Additionally, we propose a novel two-stage segmentation network called LuSeg. Through contrastive learning, it enforces semantic consistency between the RGB encoder from Stage I and the depth encoder from Stage II. Experimental results on our proposed LunarSeg dataset and additional public real-world NPO road obstacle dataset demonstrate that LuSeg achieves state-of-the-art segmentation performance for both positive and negative obstacles while maintaining a high inference speed of approximately 57 Hz. We have released the implementation of our LESS system, LunarSeg dataset, and the code of LuSeg at: https://github.com/nubot-nudt/LuSeg.}
}
  29. Xie, Z., Yun, K., Liu, Y., Ji, Y. and Li, H. (2025). Mars Traversability Prediction: A Multi-modal Self-supervised Approach for Costmap Generation . arXiv. Source
    BibTeX
    @article{xie2025mars,
  title = {Mars Traversability Prediction: A Multi-modal Self-supervised Approach for Costmap Generation},
  author = {Xie, Zongwu and Yun, Kaijie and Liu, Yang and Ji, Yiming and Li, Han},
  journal = {arXiv},
  year = {2025},
  doi = {10.48550/arxiv.2509.11082},
  abstract = {We present a robust multi-modal framework for predicting traversability costmaps for planetary rovers. Our model fuses camera and LiDAR data to produce a bird's-eye-view (BEV) terrain costmap, trained self-supervised using IMU-derived labels. Key updates include a DINOv3-based image encoder, FiLM-based sensor fusion, and an optimization loss combining Huber and smoothness terms. Experimental ablations (removing image color, occluding inputs, adding noise) show only minor changes in MAE/MSE (e.g. MAE increases from ~0.0775 to 0.0915 when LiDAR is sparsified), indicating that geometry dominates the learned cost and the model is highly robust. We attribute the small performance differences to the IMU labeling primarily reflecting terrain geometry rather than semantics and to limited data diversity. Unlike prior work claiming large gains, we emphasize our contributions: (1) a high-fidelity, reproducible simulation environment; (2) a self-supervised IMU-based labeling pipeline; and (3) a strong multi-modal BEV costmap prediction model. We discuss limitations and future work such as domain generalization and dataset expansion.}
}
  30. Endo, M., Taniai, T. and Ishigami, G. (2024). Deep Probabilistic Traversability with Test-time Adaptation for Uncertainty-aware Planetary Rover Navigation. arxiv.org/abs/2409.00641v1
    BibTeX
    @misc{endo2024deep,
  title = {Deep Probabilistic Traversability with Test-time Adaptation for Uncertainty-aware Planetary Rover Navigation},
  author = {Endo, Masafumi and Taniai, Tatsunori and Ishigami, Genya},
  year = {2024},
  url = {https://arxiv.org/abs/2409.00641v1}
}
  31. Schoppman, P., Proenca, P. F., Delaune, J., Pantic, M., Hinzmann, T., Matthies, L., Siegwart, R. and Brockers, R. (2021). Multi-Resolution Elevation Mapping and Safe Landing Site Detection with Applications to Planetary Rotorcraft . IEEE/RSJ International Conference on Intelligent Robots and Systems. Source
    BibTeX
    @inproceedings{schoppman2021multi,
  title = {Multi-Resolution Elevation Mapping and Safe Landing Site Detection with Applications to Planetary Rotorcraft},
  author = {Schoppman, Pascal and Proenca, Pedro F. and Delaune, Jeff and Pantic, Michael and Hinzmann, Timo and Matthies, Larry and Siegwart, Roland and Brockers, Roland},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  pages = {1990-1997},
  publisher = {IEEE},
  year = {2021},
  doi = {10.1109/iros51168.2021.9636507}
}
  32. Brockers, R., Delaune, J., Proenca, P., Schoppmann, P., Domnik, M., Kubiak, G. and Tzanetos, T. (2021). Autonomous Safe Landing Site Detection for a Future Mars Science Helicopter . IEEE Aerospace Conference. Source
    BibTeX
    @inproceedings{brockers2021autonomous,
  title = {Autonomous Safe Landing Site Detection for a Future Mars Science Helicopter},
  author = {Brockers, Roland and Delaune, Jeff and Proenca, Pedro and Schoppmann, Pascal and Domnik, Matthias and Kubiak, Gerik and Tzanetos, Teddy},
  booktitle = {IEEE Aerospace Conference},
  pages = {1-8},
  publisher = {IEEE},
  year = {2021},
  doi = {10.1109/aero50100.2021.9438289}
}
  33. Li, S., Jiang, X. and Tao, T. (2016). Guidance Summary and Assessment of the Chang’e-3 Powered Descent and Landing . Journal of Spacecraft and Rockets. Source
    BibTeX
    @article{li2016guidance,
  title = {Guidance Summary and Assessment of the Chang’e-3 Powered Descent and Landing},
  author = {Li, Shuang and Jiang, Xiuqiang and Tao, Ting},
  journal = {Journal of Spacecraft and Rockets},
  volume = {53},
  pages = {258-277},
  year = {2016},
  doi = {10.2514/1.a33208},
  abstract = {On 14 December 2013, Chang’e-3 successfully touched down on the surface of the moon, setting off its roaming exploration activities. Before this significant step, years of design, development, and testing of the Chang’e-3 powered descent and landing guidance, navigation, and control system were conducted to prepare for the most challenging robotic spacecraft landing endeavor attempted on the moon to date. In this paper, the mission and system design of Chang’e-3 are first described. Then, the guidance algorithms of powered descent and soft landing are summarized and assessed in detail. The powered descent and landing guidance technologies developed for Chang’e-3, including autonomous hazard detection and avoidance, lay the critical foundation for future lunar pinpoint soft-landing missions.}
}
  34. Restrepo, C. I., Petro, N. E., Barker, M. K., Mazarico, E., Liounis, A., Gnam, C., Scheidt, S., Richardson, J., Ansar, A. I., Cheng, Y., Morgan, Z., Iwashita, Y. and Beyer, R. A. (2022). Building Lunar Maps for Terrain Relative Navigation and Hazard Detection Applications . AIAA SCITECH Forum. Source
    BibTeX
    @inproceedings{restrepo2022building,
  title = {Building Lunar Maps for Terrain Relative Navigation and Hazard Detection Applications},
  author = {Restrepo, Carolina I. and Petro, Noah E. and Barker, Michael K. and Mazarico, Erwan and Liounis, Andrew and Gnam, Christopher and Scheidt, Stephen and Richardson, Jacob and Ansar, Adnan I. and Cheng, Yang and Morgan, Zachary and Iwashita, Yumi and Beyer, Ross A.},
  booktitle = {AIAA SCITECH Forum},
  publisher = {American Institute of Aeronautics and Astronautics},
  year = {2022},
  doi = {10.2514/6.2022-0356},
  abstract = {View Video Presentation: https://doi.org/10.2514/6.2022-0356.vid Terrain Relative Navigation (TRN) systems that localize a spacecraft with respect to a map of the surface by comparing descent imagery to that reference map can only be as accurate as the reference map itself. Accurate map products that are based on orbital reconnaissance data must be validated for navigation applications to ensure that all relevant error sources are minimized. Currently available map products have been generated for scientific applications, which may have different requirements from those of TRN. Therefore, the need for accurate TRN maps remains a gap to be filled for upcoming lunar lander missions, in particular missions to the South Pole region. Additionally, representative high-resolution maps that contain lander-scale features are needed for successful development and testing of Hazard Detection (HD) systems. This paper describes one of NASA’s current efforts to develop benchmark data sets that can be used for developing and testing TRN and HD algorithms. These data sets can also help to define processes and metrics for generating and validating lunar maps that can be used for navigation and hazard detection.}
}
  35. Tang, Y., Li, Q., Geng, H., Xie, Y., Shi, H. and Yang, Y. (2025). Path Planning on Multi-level Point Cloud with a Weighted Traversability Graph . arXiv. Source
    BibTeX
    @article{tang2025path,
  title = {Path Planning on Multi-level Point Cloud with a Weighted Traversability Graph},
  author = {Tang, Yujie and Li, Quan and Geng, Hao and Xie, Yangmin and Shi, Hang and Yang, Yusheng},
  journal = {arXiv},
  year = {2025},
  doi = {10.48550/arxiv.2504.21622},
  abstract = {This article proposes a new path planning method for addressing multi-level terrain situations. The proposed method includes innovations in three aspects: 1) the pre-processing of point cloud maps with a multi-level skip-list structure and data-slimming algorithm for well-organized and simplified map formalization and management, 2) the direct acquisition of local traversability indexes through vehicle and point cloud interaction analysis, which saves work in surface fitting, and 3) the assignment of traversability indexes on a multi-level connectivity graph to generate a weighted traversability graph for generally search-based path planning. The A* algorithm is modified to utilize the traversability graph to generate a short and safe path. The effectiveness and reliability of the proposed method are verified through indoor and outdoor experiments conducted in various environments, including multi-floor buildings, woodland, and rugged mountainous regions. The results demonstrate that the proposed method can properly address 3D path planning problems for ground vehicles in a wide range of situations.}
}