Skip to content
Space Robotics

Terrain Classification

Geometric hazard detection sees rocks and slopes. It cannot see sand. A patch of cohesionless soil is flat, smooth and low in the goodness map, and it is also where a rover sinks. Terrain classification exists to type the surface material so that a slip model can be applied to it, turning a non-geometric hazard into a cost the path planner can avoid [1]. Slip is learned from previous experience rather than derived from a soil model, because measuring the mechanical properties of the terrain ahead is not feasible from a moving vehicle, and a single imaging step for onboard hazard assessment already costs minutes on flight-class hardware, leaving no budget for a physical characterization pass [2, 11]. It has not flown on any rover; the flight record is field-tested research, and the mechanical counterpart is what has actually been deployed.

The problem the geometry misses

Section titled “The problem the geometry misses”

The idea of reading traction from an image rather than measuring it predates the JPL slip-prediction architecture. A 2001 rover safety module paired a vision-based neural network, trained to classify the terrain ahead as low or high traction, with a fuzzy controller that throttled commanded speed continuously rather than stopping at a threshold [3]. The classifier’s own accuracy was never reported in that work, and the demonstrated runs used hand-assigned traction values rather than the network’s output, but the shape of the idea, appearance predicts traction before the wheels get there, is the shape every later system keeps [3]. The next step was predicting slip itself rather than a traction category, from a single learned model relating appearance to slip magnitude [2].

On the Meridiani plains the only geometric obstacles were occasional hollows from small infilled craters, and inside Eagle and Endurance craters the binding hazards were slope and slip rather than obstacles [4]. Opportunity measured slip rates of 98.9 to 99.5 percent while embedded in the Purgatory ripple, and Spirit recorded slip up to 125 percent in the Columbia Hills, meaning it moved backwards during a commanded forward drive. A large slip region is an obstacle that no elevation, tilt or roughness test detects [4].

Flight slip data sorted by terrain class shows what a classifier has to separate. On Husband Hill, Spirit’s accumulated slip peaked at 83.86 m on sol 648, 5.3 percent of 1.5 km driven since sol 154, and normalized slip was 15 percent upslope against 14 percent downslope [17]. Longitudinal slip at 0 to 5 degrees of slope was 10.3 percent on clast terrain and 4.0 percent on sand-sized terrain, and on fine-grain terrain longitudinal slip reached 100 percent at 10 to 15 degrees [17]. Terrain class in that study was assigned manually from rover images, so it gives the target a classifier must reproduce, not a classifier’s output.

Visual typing and remote slip prediction

Section titled “Visual typing and remote slip prediction”

The JPL architecture built for MSL-class navigation predicts slip at a distance from imagery, in four stages [1]:

  1. The goodness map is built from navigation camera stereo, then terrain triage bins every cell into definitely traversable, definitely not traversable, or uncertain. Triage runs twice, once before slip prediction to avoid predicting slip in cells already ruled out, and once after to fold the slip cost back into the map.
  2. For a cell needing prediction, the terrain classifier is applied to the cells in its neighborhood and a majority vote gives the class at that location.
  3. A locally linear fit over the neighborhood gives longitudinal and lateral slope under the prospective rover footprint.
  4. Those slope angles enter a pre-learned nonlinear slip model specific to the classified terrain type, learned by receptive field regression, giving predicted slip and a slip cost for the cell.

The classifier itself is appearance-based: a texton representation over color and texture, in which frequent color features in small pixel neighborhoods are selected, a histogram of their occurrence within the image patch corresponding to a map cell is built, and a nearest neighbor classifier compares it against a database of training patches [1]. The texton representation treats texture as a union of features with specific appearances and no regard to their location, with random feature sampling over large patches for speed [5].

The architectural decision that makes this affordable is classifying per map cell rather than per image. Pixels above the horizon are never processed, the classifier is not invoked at all in cells already marked as obstacles or otherwise uninteresting, and near-range cells that cover a large part of the image can be processed selectively. A map cell stores only its 3D location and pointers to the images that observed it; when a class is needed, the cell is projected into an image and the corresponding patch retrieved [1]. The same structure lets stereo imagery arrive asynchronously or intermittently, since only the most recent patch for a cell is used when a classification is invoked.

The journal version of this architecture, Terrain Adaptive Navigation, was run end to end onboard the 60 kg Pluto rover in the JPL Mars Yard with a three-class classifier (bedrock, soil, rock) trained on Mars Yard imagery, per-class slip models, and 14 random candidate paths costed by high-fidelity traversability analysis [20]. The planned path crossed higher-slope bedrock and avoided a sand patch that stalled the geometry-only baseline. The evidence is one start-goal pair and one traverse per condition, with slip measured by visual odometry rather than ground truth, so it shows the mechanism works and gives no statistics [20].

What it costs

Section titled “What it costs”

Classifier accuracy and cost trade directly against each other on the same LAGR data set of six off-road terrains, soil, sand, gravel, asphalt, grass and woodchips, over 100 runs on 200 randomly sampled test patches [5]:

RepresentationErrorTime per patch
Average red channel40.9 +/- 1.7 percent0.06 +/- 0.01 s
Average color17.5 +/- 2.6 percent0.06 +/- 0.01 s
Color histogram14.0 +/- 3.3 percent0.57 +/- 0.03 s
Texton based8.1 +/- 1.5 percent4.21 +/- 0.32 s
Texton based, slow variant7.9 +/- 2.4 percent6.26 +/- 0.42 s

Source: [5], Table 1, on a research workstation rather than flight hardware; the authors note the times are machine dependent and should be read in relative terms. The texton classifier’s accuracy is judged satisfactory and its computation time explicitly not acceptable for a real-time system.

The response is a hierarchical classifier that spends effort where the discrimination is hard: with a 3 s budget the selection algorithm picks average color, color histogram and texton in sequence, discarding the average red channel as no cheaper than average color and much worse, and discarding the slow texton variant as too expensive, for an expected error of 11.7 percent [5]. In practice one class, grass, is discarded after the first level and the remainder split into sand, soil and woodchip against gravel and asphalt at the second, so five of six classes still reach the bottom level; splitting alone reduces time without hurting accuracy, and the larger saving comes from abandoning high-confidence examples early, which introduces some error. Image patches are 100 pixels across for cells at 1 to 2 m and 10 to 15 pixels across for cells at 5 to 6 m [5], which is the resolution limit behind the range failure described below [1].

Map resolution in the integrated demonstration was 10 cm cells for both the goodness and triage maps, with paths planned by D* [1]. The high-fidelity traversability analysis that runs on cells triage marks uncertain, a full kinematic and dynamic forward simulation resolving 18 contact forces, three at each of six wheels, costs about 4 s per meter of path analyzed [1]. On the actual MER hardware two decades earlier, the equivalent tradeoff was starker still: a single onboard visual odometry or hazard-avoidance imaging step took 2 to 3 minutes on a 20 MHz flight processor already running more than 90 tasks, which is why blind driving at about 120 m/hr fell to 5 m/hr with both hazard avoidance and visual odometry engaged [11]. Every terrain-classification budget since has been set against that kind of ceiling, not against a workstation’s.

Measured performance and where it fails

Section titled “Measured performance and where it fails”

On the LAGR vehicle across five terrains, sand, soil, gravel, asphalt and woodchips, with slip models and classifier trained on 3000 frames and tested on 2000 non-intersecting frames, average slip prediction error was about 21 percent [1]. If the terrain type is classified correctly the error falls to about 11 percent, so misclassification contributes the larger share. Misclassification concentrates between visually similar terrains, specifically sand against soil, and worst against soil areas covered with dust. On Mars that is the failure case that matters, because dust covers everything; Opportunity’s hazard camera stereo was defeated at Meridiani by exactly this class of surface, whose fine texture gave inadequate correlation [4].

The second failure mode is range. Most terrain classification errors in the integrated runs occur beyond 6 m, where the image patch corresponding to a map cell is very small for the LAGR camera configuration [1]. Classification accuracy therefore degrades exactly where the map is least constrained by other evidence.

Slip prediction from visual appearance alone reached about 15 percent error on the earlier single-terrain-model formulation [2], which is comparable to the measurement error of slip itself.

Mechanical typing, which did fly

Section titled “Mechanical typing, which did fly”

Curiosity’s Traction Control estimates the wheel-terrain contact angles in real time from a rocker-bogie kinematics model driven by suspension encoders and measured attitude rates, and commands the idealized no-slip angular rate for each of the six wheels [7]. It was written because wheel puncture damage appeared in October 2013 at an unexpected rate, driven by five wheels pushing a sixth over an embedded sharp rock when all six turn at one speed. It does not classify the terrain by name; it measures the vehicle’s mechanical response to it, which is the information the visual classifier is trying to predict.

It flew as a hot patch on top of R12 rather than as a flight software release, and it adds less than 3 percent to total CPU usage while driving [7]. Wheel speed ratios are evaluated at 8 Hz; the average Traction Control speed ratio over the first six months of operation, sols 1646 through 1822, was 0.899. Checkout was a 5 m drive on sol 1646 and a 20 m drive on sol 1662, both logging drive telemetry at 64 Hz, with nominal use from sol 1678 [7]. Ground testing on engineered terrain also found an unexpected mode, a wheelie in which the middle wheel lifts off under suspension tension in high-friction terrain, which required a suppression behavior in the algorithm.

The same reasoning drove Approximate Clearance Evaluation: rather than classify the terrain, bound what the suspension would do on it [8]. A more direct alternative sits between the two: a sensorized wheel that reads soil hydration, composition, slip and sink from pressure-grid and electromagnetic-induction sensing in the wheel itself, in no-light conditions the camera cannot use at all [6]. It has been demonstrated only on a stationary bench rig with an unspecified regolith simulant, not on a driving vehicle, so it answers a different part of the problem than the visual classifier without yet replacing it [6].

Other sensing routes

Section titled “Other sensing routes”

Terrestrial ground vehicle programs treated classification as one layer of a stereo terrain map. JPL’s work for the Demo III, PerceptOR and RCTA programs encodes elevation, classification, roughness, traversability cost and confidence per cell, merged over time into a 100 m by 100 m world map at 40 cm cells [13]. With a 50 cm baseline and a 1.53 m sensor height, one-sigma range error at 100 m is about 3.27 m while height error is about 5 cm, so height rather than range is the usable long-range cue [13]. The same programs used cooled mid-wave and uncooled long-wave thermal infrared cameras to separate soil from vegetation and to detect water, mud and negative obstacles, at the cost of reporting few detection rates [14]. These are terrestrial results with no planetary hardware.

Contact and proprioceptive sensing avoids the range limit by typing the surface the wheel is already on. On a bench-mounted test foot, a microphone classified six terrains (three sands, gravel, carpet and tile) at over 97 percent accuracy against 82 to 91 percent for a six-axis force-torque sensor [12]. Piezoelectric sensors on a 1:4 scale hook-shaped spoke of a Curiosity-style wheel identified five lab terrains at 90 percent with a support vector machine [15]. Both are laboratory results on small trays with no regolith simulant, dust or vacuum [12, 15].

Vibration sensing was validated earlier than either. A contact microphone on a FIDO wheel classified gravel at 302 of 312 one-second segments, Mars-1 simulant at 208 of 277 and beach sand at 139 of 276, with 86 sand segments returning no class, and a four-wheeled rover outdoors separated sand, packed dirt and gravel from 3 s segments [19]. All data are terrestrial, at 1 g and in air.

Learned image segmentation has moved toward onboard use. A segmentation classifier trained on the AI4Mars set of over 400000 citizen-scientist labels reached 97.0 percent pixel accuracy on Curiosity imagery, 84.2 percent with 93.4 percent sand recall after adaptation to Mars 2020 from about 70 images, and runs in 24 ms per image with MobileNetV2 on a mobile processor [18]. The evaluation is offline and the timing is cited from other work, so no onboard result exists [18].

Learned traversability has also been extended to carry uncertainty. An ensemble of ten U-Net networks predicts slip mean and variance from color and elevation maps, and adapts at test time from slip the rover actually experiences. On 400 synthetic planning problems, success on maps with unfamiliar appearance and geometry was 87 percent against 66 to 69 percent for two baselines [21]. All results are simulation on synthetic maps with prescribed slip functions, with no rover hardware or regolith simulant [21].

Foundation-model classification has appeared most recently: GPT-4V rated terrain complexity from images to select among three rover navigation modes, in simulation only, with five positions per scenario [16].

Datasets, learned segmentation and wheel-hazard typing

Section titled “Datasets, learned segmentation and wheel-hazard typing”

Operations-side typing for wheel damage came before the datasets. After the MSL wheel damage peaked over sols 450 to 515, a Random Forest classifier labeled every pixel of a Navcam image into five wheel-relevant classes (sand, loose rock, bedrock, embedded pointy rock, embedded round rock) from 66 hand-labeled images, to feed a risk-aware path tool [30]. The paper reports no classifier accuracy and no driven path, so it is a proposal with a demonstration of the tooling [30]. SPOC, a dilated-convolution network, labeled six terrain classes on Curiosity Navcam images and 17 classes on orbital HiRISE mosaics; sand recall on the rover images was 50.5 percent, with 44.4 percent of sand labeled as small rocks [23].

AI4Mars supplied the label set that later work trains on: about 326000 crowdsourced labels over 35000 Curiosity, Opportunity and Spirit images, reduced to four traversability classes, with about 1500 expert-labeled validation images [22]. It is a benchmark and reports no onboard runtime [22]. With pretraining on unlabeled images, a segmentation network reached 91.1 percent test accuracy from 1 percent of the labels (161 images) against 81.9 percent for random initialization, and 97.4 against 95.3 percent with all labels [24]. The network has 171 million parameters, far above flight processor capacity, and was run offline only [24].

Multimodal input is the current direction on the ground. OmniUnet segments terrain from RGB, depth and thermal channels recorded by a half-scale ExoMars-class rover in the Bardenas semi-desert; total pixel accuracy was 80.37 percent but mean pixel accuracy was 46.59 percent and rock reached 18.40 percent, and no ablation separates the thermal channel’s contribution [25]. A virtual infrared network predicts a thermal image from RGB so a rover without an IR camera can use it, but its own table shows the added Sun angle and solar radiation inputs change frequency-weighted IoU by 0.001 [29]. Thermal inertia itself separates soils only at Mars pressure: in a Mars chamber four soils were distinguishable at 8 mbar and essentially not at 1000 mbar [31]. For slip, adding thermal inertia to a slope-only model on 135 Curiosity visual odometry measurements cut root mean squared error from 22.0 to 16.8 percent with the rover ground temperature sensor and 15.9 percent with orbital THEMIS data [28].

Costmap predictors skip the class label. RoadRunner predicts traversability cost and elevation directly from camera and LiDAR in a bird’s eye view, trained on pseudo ground truth from JPL’s X-Racer stack on terrestrial off-road driving, so its accuracy is measured against the heuristic that produced its labels [27]. A Mars-like simulation-trained fusion of camera and LiDAR, labeled from IMU response, was tested only on an Earth rover [26].

Proprioceptive typing has a learned branch as well. A self-supervised scheme used wheel vibration or a traction-force model to label terrain a four-wheeled rover had driven, then trained a stereo classifier on those labels to predict terrain ahead; the data are from one beach with sand, grass and rock, at 3 cm/s [33]. A multilayer perceptron on FFT features of wheel vibration separated five terrestrial surfaces at walking pace, with confusion between sand and soil attributed to weeds in the soil plot [34]. LSTM, convolutional and SVM classifiers on wheel current, velocity and IMU signals were compared on a Husky in one vineyard at 0.5 m/s [35]. The Barefoot Rover wheel, with a pressure grid and impedance sensing, estimates slip and hydration with gradient boosted trees on a lab test cart; buried rock accuracy was 7 percent [32]. None of these has flown.

What is not established

Section titled “What is not established”

The visual architecture described above has not flown, and the gap between it and flight is not just qualification. A recent attempt to remove the terrain-classification step entirely, predicting a per-segment slip cost map directly from a learned terrain segmentation and in-situ wheel telemetry, reports its numbers from a single simulated rover model in a physics engine, not from a physical vehicle or a characterized regolith simulant, and its own baseline comparison loses on one of five held-out test samples [9]. Whether a segmentation-to-slip mapping generalizes off a simulator has not been shown.

The mechanical alternative to camera-based typing has the same evidentiary shape in a different place: force-torque sensors mounted above the wheels of a half-scale rover testbed classify terrain better than the chassis inertial measurement unit, 95.6 versus 85.8 percent in one comparison, but the same field campaign could not extract a usable drawbar-pull signal from the sensors at all, because vibration swamped it, and the authors recommend against the mounting position that produced the classification result [10]. No published work has run the camera-based classifier and a wheel-based sensor on the same drive over the same ground truth, so which one degrades first under dust, and whether they fail on correlated terrain or independent terrain, is unanswered. The economics that would decide a flight choice between them, the CPU and power cost of vision-based triage against the mass and wiring cost of instrumenting six wheels, have not been compared either.

References

  1. Helmick, D. M., Angelova, A., Livianu, M. and Matthies, L. H. (2007). Terrain Adaptive Navigation for Mars Rovers . IEEE Aerospace Conference. Source
    BibTeX
    @inproceedings{helmick2007terrain,
  title = {Terrain Adaptive Navigation for Mars Rovers},
  author = {Helmick, Daniel M. and Angelova, Anelia and Livianu, Matthew and Matthies, Larry H.},
  booktitle = {IEEE Aerospace Conference},
  pages = {1-11},
  address = {Big Sky, Montana},
  year = {2007},
  doi = {10.1109/aero.2007.352684},
  abstract = {A navigation system for Mars rovers in very rough terrain has been designed, implemented, and tested on a research rover in Mars analog terrain. This navigation system consists of several technologies that are integrated to increase the capabilities compared to current rover navigation algorithms. These technologies include: goodness maps and terrain triage, terrain classification, remote slip prediction, path planning, high-fidelity traversability analysis (HFTA), and slip-compensated path following. The focus of this paper is not on the component technologies, but rather on the integration of these components. Results from the onboard integration of several of the key technologies described here are shown. Additionally, the results from independent demonstrations of several of these technologies are shown. Future work will include the demonstration of the entire integrated system.}
}
  2. Angelova, A., Matthies, L., Helmick, D., Sibley, G. and Perona, P. (2006). Learning to Predict Slip for Ground Robots . IEEE International Conference on Robotics and Automation. Source
    BibTeX
    @inproceedings{angelova2006learning,
  title = {Learning to Predict Slip for Ground Robots},
  author = {Angelova, Anelia and Matthies, Larry and Helmick, Daniel and Sibley, Gabe and Perona, Pietro},
  booktitle = {IEEE International Conference on Robotics and Automation},
  pages = {3324-3331},
  year = {2006},
  doi = {10.1109/robot.2006.1642209},
  abstract = {In this paper we predict the amount of slip an exploration rover would experience using stereo imagery by learning from previous examples of traversing similar terrain. To do that, the information of terrain appearance and geometry regarding some location is correlated to the slip measured by the rover while this location is being traversed. This relationship is learned from previous experience, so slip can be predicted later at a distance from visual information only. The advantages of the approach are: 1) learning from examples allows the system to adapt to unknown terrains rather than using fixed heuristics or predefined rules; 2) the feedback about the observed slip is received from the vehicle's own sensors which can fully automate the process; 3) learning slip from previous experience can replace complex mechanical modeling of vehicle or terrain, which is time consuming and not necessarily feasible. Predicting slip is motivated by the need to assess the risk of getting trapped before entering a particular terrain. For example, a planning algorithm can utilize slip information by taking into consideration that a slippery terrain is costly or hazardous to traverse. A generic nonlinear regression framework is proposed in which the terrain type is determined from appearance and then a nonlinear model of slip is learned for a particular terrain type. In this paper we focus only on the latter problem and provide slip learning and prediction results for terrain types, such as soil, sand, gravel, and asphalt. The slip prediction error achieved is about 15% which is comparable to the measurement errors for slip itself}
}
  3. Tunstel, E., Howard, A. and Seraji, H. (2001). Fuzzy rule-based reasoning for rover safety and survivability . IEEE International Conference on Robotics and Automation. Source
    BibTeX
    @inproceedings{tunstel2001fuzzy,
  title = {Fuzzy rule-based reasoning for rover safety and survivability},
  author = {Tunstel, E. and Howard, A. and Seraji, Homayoun},
  booktitle = {IEEE International Conference on Robotics and Automation},
  volume = {2},
  pages = {1413-1420},
  publisher = {IEEE},
  year = {2001},
  doi = {10.1109/robot.2001.932808}
}
  4. Maimone, M., Johnson, A., Cheng, Y., Willson, R. and Matthies, L. (2004). Autonomous Navigation Results from the Mars Exploration Rover (MER) Mission . International Symposium on Experimental Robotics. Source
    BibTeX
    @inproceedings{maimone2004autonomous,
  title = {Autonomous Navigation Results from the Mars Exploration Rover (MER) Mission},
  author = {Maimone, Mark and Johnson, Andrew and Cheng, Yang and Willson, Reg and Matthies, Larry},
  booktitle = {International Symposium on Experimental Robotics},
  address = {Singapore},
  year = {2004},
  url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/41077}
}
  5. Angelova, A., Matthies, L., Helmick, D. and Perona, P. (2007). Fast Terrain Classification Using Variable-Length Representation for Autonomous Navigation . IEEE Conference on Computer Vision and Pattern Recognition. Source
    BibTeX
    @inproceedings{angelova2007learning,
  title = {Fast Terrain Classification Using Variable-Length Representation for Autonomous Navigation},
  author = {Angelova, Anelia and Matthies, Larry and Helmick, Daniel and Perona, Pietro},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
  pages = {1-8},
  address = {Minneapolis, Minnesota},
  year = {2007},
  doi = {10.1109/cvpr.2007.383024},
  abstract = {We propose a method for learning using a set of feature representations which retrieve different amounts of information at different costs. The goal is to create a more efficient terrain classification algorithm which can be used in real-time, onboard an autonomous vehicle. Instead of building a monolithic classifier with uniformly complex representation for each class, the main idea here is to actively consider the labels or misclassification cost while constructing the classifier. For example, some terrain classes might be easily separable from the rest, so very simple representation will be sufficient to learn and detect these classes. This is taken advantage of during learning, so the algorithm automatically builds a variable-length visual representation which varies according to the complexity of the classification task. This enables fast recognition of different terrain types during testing. We also show how to select a set of feature representations so that the desired terrain classification task is accomplished with high accuracy and is at the same time efficient. The proposed approach achieves a good trade-off between recognition performance and speedup on data collected by an autonomous robot.}
}
  6. Kennedy, B., Ma, R., Junkins, E., Lightholder, J., Mandrake, L., Marchetti, Y. and Tavallali, P. (2019). The Barefoot Rover. dataverse.jpl.nasa.gov/api/access/datafile/63636
    BibTeX
    @misc{kennedy2019barefoot,
  title = {The Barefoot Rover},
  author = {Kennedy, Brett and Ma, Raymond and Junkins, Eric and Lightholder, Jack and Mandrake, Lukas and Marchetti, Yuliya and Tavallali, Peyman},
  year = {2019},
  url = {https://dataverse.jpl.nasa.gov/api/access/datafile/63636}
}
  7. Toupet, O., Biesiadecki, J., Rankin, A., Steffy, A., Meirion-Griffith, G., Levine, D., Schadegg, M. and Maimone, M. (2020). Traction Control on the Curiosity Mars Rover: Algorithm and Flight Results . Journal of Field Robotics. Source
    BibTeX
    @article{toupet2020traction,
  title = {Traction Control on the Curiosity Mars Rover: Algorithm and Flight Results},
  author = {Toupet, Olivier and Biesiadecki, Jeffrey and Rankin, Arturo and Steffy, Amanda and Meirion-Griffith, Gareth and Levine, Dan and Schadegg, Maximilian and Maimone, Mark},
  journal = {Journal of Field Robotics},
  year = {2020},
  doi = {10.48577/jpl.hkzuqs},
  abstract = {No abstract available.}
}
  8. Otsu, K., Matheron, G., Ghosh, S., Toupet, O. and Ono, M. (2020). Fast Approximate Clearance Evaluation for Rovers with Articulated Suspension Systems . Journal of Field Robotics, 5. Source
    BibTeX
    @article{otsu2020fast,
  title = {Fast Approximate Clearance Evaluation for Rovers with Articulated Suspension Systems},
  author = {Otsu, Kyohei and Matheron, Guillaume and Ghosh, Sourish and Toupet, Olivier and Ono, Masahiro},
  journal = {Journal of Field Robotics},
  volume = {37},
  number = {5},
  pages = {768--785},
  year = {2020},
  doi = {10.1002/rob.21892},
  abstract = {Abstract We present a light‐weight body‐terrain clearance evaluation algorithm for the automated path planning of NASA's Mars 2020 rover. Extraterrestrial path planning is challenging due to the combination of terrain roughness and severe limitation in computational resources. Path planning on cluttered and/or uneven terrains requires repeated safety checks on all the candidate paths at a small interval. Predicting the future rover state requires simulating the vehicle settling on the terrain, which involves an inverse‐kinematics problem with iterative nonlinear optimization under geometric constraints. However, such expensive computation is intractable for slow spacecraft computers, such as RAD750, which is used by the Curiosity Mars rover and upcoming Mars 2020 rover. We propose the approximate clearance evaluation (ACE) algorithm, which obtains conservative bounds on vehicle clearance, attitude, and suspension angles without iterative computation. It obtains those bounds by estimating the lowest and highest heights that each wheel may reach given the underlying terrain, and calculating the worst‐case vehicle configuration associated with those extreme wheel heights. The bounds are guaranteed to be conservative, hence ensuring vehicle safety during autonomous navigation. ACE is planned to be used as part of the new onboard path planner of the Mars 2020 rover. This paper describes the algorithm in detail and validates our claim of conservatism and fast computation through experiments.}
}
  9. Yakubu, M., Zweiri, Y., Abubakar, A., Azzam, R., Alhammadi, R. and Seneviratne, L. D. (2024). SlipNet: Enhancing Slip Cost Mapping for Autonomous Navigation on Heterogeneous and Deformable Terrains . arXiv. Source
    BibTeX
    @article{yakubu2024slipnet,
  title = {SlipNet: Enhancing Slip Cost Mapping for Autonomous Navigation on Heterogeneous and Deformable Terrains},
  author = {Yakubu, Mubarak and Zweiri, Yahya and Abubakar, Ahmad and Azzam, Rana and Alhammadi, Ruqayya and Seneviratne, Lakmal D.},
  journal = {arXiv},
  year = {2024},
  doi = {10.48550/arxiv.2409.02273},
  abstract = {Autonomous space rovers face significant challenges when navigating deformable and heterogeneous terrains due to variability in soil properties, which can lead to severe wheel slip, compromising navigation efficiency and increasing the risk of entrapment. To address this problem, we introduce SlipNet, a novel approach for predicting wheel slip in segmented regions of diverse terrain surfaces without relying on prior terrain classification. SlipNet employs dynamic terrain segmentation and slip assignment techniques on previously unseen data, enhancing rover navigation capabilities in uncertain environments. We developed a synthetic data generation framework using the high-fidelity Vortex Studio simulator to create realistic datasets that replicate a wide range of deformable terrain conditions for training and evaluation. Extensive simulation results demonstrate that our model, combining DeepLab v3+ with SlipNet, significantly outperforms the state-of-the-art TerrainNet method, achieving lower mean absolute error (MAE) across five distinct terrain samples. These findings highlight the effectiveness of SlipNet in improving rover navigation in challenging terrains.}
}
  10. Gerdes, L., Pérez del Pulgar, C., Castilla Arquillo, R. and Azkarate, M. (2025). Field Assessment of Force Torque Sensors for Planetary Rover Navigation . Journal of Intelligent and Robotic Systems. Source
    BibTeX
    @article{gerdes2025field,
  title = {Field Assessment of Force Torque Sensors for Planetary Rover Navigation},
  author = {Gerdes, Levin and Pérez del Pulgar, Carlos and Castilla Arquillo, Raúl and Azkarate, Martin},
  journal = {Journal of Intelligent and Robotic Systems},
  volume = {111},
  year = {2025},
  doi = {10.1007/s10846-025-02324-2},
  abstract = {Abstract Proprioceptive sensors on planetary rovers serve for state estimation and for understanding terrain and locomotion performance. While inertial measurement units (IMUs) are widely used to this effect, force-torque sensors are less explored for planetary navigation despite their potential to directly measure interaction forces and provide insights into traction performance. This paper presents an evaluation of the performance and use cases of force-torque sensors based on data collected from a six-wheeled rover during tests over varying terrains, speeds, and slopes. We discuss challenges, such as sensor signal reliability and terrain response accuracy, and identify opportunities regarding the use of these sensors. The data is openly accessible and includes force-torque measurements from each of the six-wheel assemblies as well as IMU data from within the rover chassis. This paper aims to inform the design of future studies and rover upgrades, particularly in sensor integration and control algorithms, to improve navigation capabilities.}
}
  11. Maimone, M. W. (2009). Planetary Rover Systems: Sojourner to MSL, 20150011982. ntrs.nasa.gov/citations/20150011982
    BibTeX
    @misc{maimone2009planetary,
  title = {Planetary Rover Systems: {Sojourner} to {MSL}},
  author = {Maimone, Mark W.},
  howpublished = {Presentation at the Jet Propulsion Laboratory, California Institute of Technology},
  number = {20150011982},
  year = {2009},
  url = {https://ntrs.nasa.gov/citations/20150011982},
  abstract = {No abstract available}
}
  12. Xue, F., Hu, L., Yao, C., Liu, Z., Zhu, Z. and Jia, Z. (2022). Sound-Based Terrain Classification for Multi-modal Wheel-Leg Robots . International Conference on Advanced Robotics. Source
    BibTeX
    @inproceedings{xue2022sound,
  title = {Sound-Based Terrain Classification for Multi-modal Wheel-Leg Robots},
  author = {Xue, Feng and Hu, Longteng and Yao, Chen and Liu, Zhengtao and Zhu, Zheng and Jia, Zhenzhong},
  booktitle = {International Conference on Advanced Robotics},
  pages = {174-179},
  publisher = {IEEE},
  year = {2022},
  doi = {10.1109/icarm54641.2022.9959511}
}
  13. Rankin, A. L., Huertas, A. and Matthies, L. H. (2009). Stereo vision based terrain mapping for off-road autonomous navigation . Unmanned Systems Technology XI. Source
    BibTeX
    @inproceedings{rankin2009stereo,
  title = {Stereo vision based terrain mapping for off-road autonomous navigation},
  author = {Rankin, Arturo L. and Huertas, Andres and Matthies, Larry H.},
  booktitle = {Unmanned Systems Technology XI},
  series = {SPIE Proceedings},
  volume = {7332},
  pages = {733210},
  publisher = {SPIE},
  year = {2009},
  doi = {10.1117/12.819099},
  abstract = {Successful off-road autonomous navigation by an unmanned ground vehicle (UGV) requires reliable perception and representation of natural terrain. While perception algorithms are used to detect driving hazards, terrain mapping algorithms are used to represent the detected hazards in a world model a UGV can use to plan safe paths. There are two primary ways to detect driving hazards with perception sensors mounted to a UGV: binary obstacle detection and traversability cost analysis. Binary obstacle detectors label terrain as either traversable or non-traversable, whereas, traversability cost analysis assigns a cost to driving over a discrete patch of terrain. In uncluttered environments where the non-obstacle terrain is equally traversable, binary obstacle detection is sufficient. However, in cluttered environments, some form of traversability cost analysis is necessary. The Jet Propulsion Laboratory (JPL) has explored both approaches using stereo vision systems. A set of binary detectors has been implemented that detect positive obstacles, negative obstacles, tree trunks, tree lines, excessive slope, low overhangs, and water bodies. A compact terrain map is built from each frame of stereo images. The mapping algorithm labels cells that contain obstacles as nogo regions, and encodes terrain elevation, terrain classification, terrain roughness, traversability cost, and a confidence value. The single frame maps are merged into a world map where temporal filtering is applied. In previous papers, we have described our perception algorithms that perform binary obstacle detection. In this paper, we summarize the terrain mapping capabilities that JPL has implemented during several UGV programs over the last decade and discuss some challenges to building terrain maps with stereo range data.}
}
  14. Rankin, A., Huertas, A., Matthies, L., Bajracharya, M., Assad, C., Brennan, S., Bellutta, P. and Sherwin, G. W. (2011). Unmanned ground vehicle perception using thermal infrared cameras . Unmanned Systems Technology XIII. Source
    BibTeX
    @inproceedings{rankin2011unmanned,
  title = {Unmanned ground vehicle perception using thermal infrared cameras},
  author = {Rankin, Arturo and Huertas, Andres and Matthies, Larry and Bajracharya, Max and Assad, Christopher and Brennan, Shane and Bellutta, Paolo and Sherwin, Gary W.},
  booktitle = {Unmanned Systems Technology XIII},
  series = {SPIE Proceedings},
  volume = {8045},
  pages = {804503},
  publisher = {SPIE},
  year = {2011},
  doi = {10.1117/12.884349},
  abstract = {The ability to perform off-road autonomous navigation at any time of day or night is a requirement for some unmanned ground vehicle (UGV) programs. Because there are times when it is desirable for military UGVs to operate without emitting strong, detectable electromagnetic signals, a passive only terrain perception mode of operation is also often a requirement. Thermal infrared (TIR) cameras can be used to provide day and night passive terrain perception. TIR cameras have a detector sensitive to either mid-wave infrared (MWIR) radiation (3-5μm) or long-wave infrared (LWIR) radiation (7-14μm). With the recent emergence of high-quality uncooled LWIR cameras, TIR cameras have become viable passive perception options for some UGV programs. The Jet Propulsion Laboratory (JPL) has used a stereo pair of TIR cameras under several UGV programs to perform stereo ranging, terrain mapping, tree-trunk detection, pedestrian detection, negative obstacle detection, and water detection based on object reflections. In addition, we have evaluated stereo range data at a variety of UGV speeds, evaluated dual-band TIR classification of soil, vegetation, and rock terrain types, analyzed 24 hour water and 12 hour mud TIR imagery, and analyzed TIR imagery for hazard detection through smoke. Since TIR cameras do not currently provide the resolution available from megapixel color cameras, a UGV's daytime safe speed is often reduced when using TIR instead of color cameras. In this paper, we summarize the UGV terrain perception work JPL has performed with TIR cameras over the last decade and describe a calibration target developed by General Dynamics Robotic Systems (GDRS) for TIR cameras and other sensors.}
}
  15. Jin, X., Wang, Z., Yu, Z., Choi, C., Carpenter, K. and Nanayakkara, T. (2025). Physical Reservoir Computing in Hook-Shaped Rover Wheel Spokes for Real-Time Terrain Identification. arxiv.org/abs/2504.13348
    BibTeX
    @misc{jin2025physical,
  title = {Physical Reservoir Computing in Hook-Shaped Rover Wheel Spokes for Real-Time Terrain Identification},
  author = {Jin, Xiao and Wang, Zihan and Yu, Zhenhua and Choi, Changrak and Carpenter, Kalind and Nanayakkara, Thrishantha},
  year = {2025},
  url = {https://arxiv.org/abs/2504.13348}
}
  16. Cheng, S., Zhou, R., Feng, W., Yang, H., Gao, H., Deng, Z. and Ding, L. (2025). VLM-Empowered Multi-Mode System for Efficient and Safe Planetary Navigation . IEEE/RSJ International Conference on Intelligent Robots and Systems. Source
    BibTeX
    @inproceedings{cheng2025vlm,
  title = {VLM-Empowered Multi-Mode System for Efficient and Safe Planetary Navigation},
  author = {Cheng, Sinuo and Zhou, Ruyi and Feng, Wenhao and Yang, Huaiguang and Gao, Haibo and Deng, Zongquan and Ding, Liang},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  pages = {1487-1494},
  year = {2025},
  doi = {10.1109/iros60139.2025.11247594},
  abstract = {The increasingly complex and diverse planetary exploration environment requires more adaptable and flexible rover navigation strategy. In this study, we propose a VLM-empowered multi-mode system to achieve efficient while safe autonomous navigation for planetary rovers. Vision-Language Model (VLM) is used to parse scene information by image inputs to achieve a human-level understanding of terrain complexity. Based on the complexity classification, the system switches to the most suitable navigation mode, composing of perception, mapping and planning modules designed for different terrain types, to traverse the terrain ahead before reaching the next waypoint. By integrating the local navigation system with a map server and a global waypoint generation module, the rover is equipped to handle long-distance navigation tasks in complex scenarios. The navigation system is evaluated in various simulation environments. Compared to the single-mode conservative navigation method, our multi-mode system is able to bootstrap the time and energy efficiency in a long-distance traversal with varied type of obstacles, enhancing efficiency by 79.5%, while maintaining its avoidance capabilities against terrain hazards to guarantee rover safety. More system information is shown at https://chengsn1234.github.io/multi-mode-planetary-navigation/.}
}
  17. Li, R., Wu, B., Di, K., Angelova, A., Arvidson, R. E., Lee, I.-C., Maimone, M., Matthies, L. H., Richer, L., Sullivan, R., Sims, M. H., Greenberger, R. and Squyres, S. W. (2008). Characterization of traverse slippage experienced by Spirit rover on Husband Hill at Gusev crater . Journal of Geophysical Research: Planets. Source
    BibTeX
    @article{li2008characterization,
  title = {Characterization of traverse slippage experienced by Spirit rover on Husband Hill at Gusev crater},
  author = {Li, Rongxing and Wu, Bo and Di, Kaichang and Angelova, Anelia and Arvidson, Raymond E. and Lee, I-Chieh and Maimone, Mark and Matthies, Larry H. and Richer, Lutz and Sullivan, Robert and Sims, Michael H. and Greenberger, Rebecca and Squyres, Steven W.},
  journal = {Journal of Geophysical Research: Planets},
  volume = {113},
  year = {2008},
  doi = {10.1029/2008je003097},
  abstract = {Spirit rover experienced significant slips traversing Husband Hill. This paper analyzes the slippage Spirit experienced from Sol 154 to Sol 737. Slippage with respect to terrain type and slope is computed using data downlinked from the rover, rover position, and orientation estimations from visual odometry (VO) and photogrammetry based bundle adjustment (BA) method. Accumulated slippage reached a maximum of 83.86 m on Sol 648. However, as Spirit descended into the Inner Basin, the direction of slippage reversed, and accumulated slippage approached zero by the end of the entire traverse. Eight local regions with significant slips and nineteen traverse segments have been analyzed. Slippage was found to be highly correlated to slope direction and magnitude; the reverse of slope directions in the ascending and descending portions of the traverse proves to be the main contributor to the observed cancellation of slippage. While the horizontal component of the slippage almost canceled out, the difference in elevation continually accumulated, mainly during the ascent. In general, long traverse segments created more slips than short ones. This is reflected in both the accumulated and individual slippages. In considering the four major Mars terrain types, Spirit performed best on bedrock, managing to drive on slopes close to 30°. Fine‐grain surfaces were the most challenging; though progress was made on slopes up to 15°, slippages of over 100% (more slippage than distance traveled) occurred for short segments. The results of this work can be incorporate into a traverse planning framework in which rover slippage is minimized. Results can be employed in landed planetary missions for precision navigation to avoid potentially dangerous regions by considering expected slippage.}
}
  18. Atha, D., Swan, R. M., Didier, A., Hasnain, Z. and Ono, M. (2022). Multi-Mission Terrain Classifier for Safe Rover Navigation and Automated Science . IEEE Aerospace Conference. Source
    BibTeX
    @inproceedings{atha2022multi,
  title = {Multi-Mission Terrain Classifier for Safe Rover Navigation and Automated Science},
  author = {Atha, Deegan and Swan, R. Michael and Didier, Annie and Hasnain, Zaki and Ono, Masahiro},
  booktitle = {IEEE Aerospace Conference},
  pages = {1-13},
  publisher = {IEEE},
  year = {2022},
  doi = {10.1109/aero53065.2022.9843615},
  abstract = {We previously presented Soil Property and Object Classification (SPOC), a machine learning-based terrain classi-fier for Mars rovers, for automatically segmenting rover images by its surface type such as sand and bedrock. This paper presents a number of practical improvements to pave the way for potential future onboard deployment. First, we achieved 97.0 % overall pixel accuracy, evaluated against the classification generated by human experts on images from Mars Science Lab-oratory (MSL) missions. The substantial increase in accuracy was primarily enabled by the sheer volume of data used for training; we created a new large-scale dataset of Martian terrain labels, namely AI4Mars, which contains more than 400k labels contributed by citizen scientists for ~50k images taken by the Mars Exploration Rovers (MER) and Mars Science Laboratory (MSL) rover. Second, we demonstrated that SPOC can quickly adapt to a new mission landed on a previously unseen site. Specifically, we pretrained a model with MER and MSL data from the AI4Mars dataset and then adapted to the Mars 2020 Rover (M2020) by feeding a small volume of data between Sol 0 and 157; the adapted model was tested on Sol 200–203 and resulted in 84.2 % overall pixel accuracy and 93.4 % reliability (recall) for detecting sand, the most concerning class for rover's traversability. Third, we found that pretraining can substan-tially mitigate the decline of accuracy over time. We showed that the performance of a SPOC model pretrained with the ImageNet dataset and then trained by MSL images only up to Sol 390 remains comparable to a model trained by images up to Sol 1689 on the test data after Sol 1689. Fourth, we reimplemented SPOC with a light-weight convolutional neural network (CNN), MobileNetV2, which typically runs within tens of milliseconds (ms) on mobile processors such as Qualcomm's Snapdragon. Finally, we released the AI4Mars dataset to the public to encourage open innovation.}
}
  19. Brooks, C. A. and Iagnemma, K. (2005). Vibration-Based Terrain Classification for Planetary Exploration Rovers . IEEE Transactions on Robotics, 6. Source
    BibTeX
    @article{brooks2005vibration,
  title = {Vibration-Based Terrain Classification for Planetary Exploration Rovers},
  author = {Brooks, Christopher A. and Iagnemma, Karl},
  journal = {IEEE Transactions on Robotics},
  volume = {21},
  number = {6},
  pages = {1185--1191},
  year = {2005},
  doi = {10.1109/tro.2005.855994},
  abstract = {Safe, autonomous mobility in rough terrain is an important requirement for planetary exploration rovers. Knowledge of local terrain properties is critical to ensure a rover's safety on slopes and uneven surfaces. Visual features are often used to classify terrain; however, vision can be sensitive to lighting variations and other effects. This paper presents a method to classify terrain based on vibrations induced in the rover structure by wheel-terrain interaction during driving. This sensing mode is robust to lighting variations. Vibrations are measured using an accelerometer mounted on the rover structure. The classifier is trained using labeled vibration data during an offline learning phase. Linear discriminant analysis is used for online identification of terrain classes, such as sand, gravel, or clay. This approach has been experimentally validated on a laboratory testbed and on a four-wheeled rover in outdoor conditions.}
}
  20. Helmick, D., Angelova, A. and Matthies, L. (2009). Terrain Adaptive Navigation for planetary rovers . Journal of Field Robotics. Source
    BibTeX
    @article{helmick2009terrain,
  title = {Terrain Adaptive Navigation for planetary rovers},
  author = {Helmick, Daniel and Angelova, Anelia and Matthies, Larry},
  journal = {Journal of Field Robotics},
  volume = {26},
  pages = {391-410},
  year = {2009},
  doi = {10.1002/rob.20292},
  abstract = {Abstract This paper describes the design, implementation, and experimental results of a navigation system for planetary rovers called Terrain Adaptive Navigation (TANav). This system was designed to enable greater access to and more robust operations within terrains of widely varying slippage. The system achieves this goal by using onboard stereo cameras to remotely classify surrounding terrain, predict the slippage of that terrain, and use this information in the planning of a path to the goal. This navigation system consists of several integrated techniques: goodness map generation, terrain triage, terrain classification, remote slip prediction, path planning, High‐Fidelity Traversability Analysis, and slip‐compensated path following. Results from experiments with an end‐to‐end onboard implementation of the TANav system in a Mars analog environment are shown and compared to results from experiments with a more traditional navigation system that does not account for terrain properties. © 2009 Wiley Periodicals, Inc.}
}
  21. Endo, M., Taniai, T. and Ishigami, G. (2024). Deep Probabilistic Traversability with Test-time Adaptation for Uncertainty-aware Planetary Rover Navigation. arxiv.org/abs/2409.00641v1
    BibTeX
    @misc{endo2024deep,
  title = {Deep Probabilistic Traversability with Test-time Adaptation for Uncertainty-aware Planetary Rover Navigation},
  author = {Endo, Masafumi and Taniai, Tatsunori and Ishigami, Genya},
  year = {2024},
  url = {https://arxiv.org/abs/2409.00641v1}
}
  22. Swan, R. M., Atha, D., Leopold, H. A., Gildner, M., Oij, S., Chiu, C., Hong, X. and Ono, H. (2021). AI4MARS: A Dataset for Terrain-Aware Autonomy on Mars . JPL Open Repository. Source
    BibTeX
    @inproceedings{swan2021ai4mars,
  title = {AI4MARS: A Dataset for Terrain-Aware Autonomy on Mars},
  author = {Swan, R. Michael and Atha, Deegan and Leopold, Henry A. and Gildner, Matthew and Oij, Stephanie and Chiu, Cindy and Hong, Xiangyu and Ono, Hiro},
  publisher = {JPL Open Repository},
  year = {2021},
  url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/54773}
}
  23. Rothrock, B., Kennedy, R., Cunningham, C., Papon, J., Heverly, M. and Ono, M. (2016). SPOC: Deep Learning-based Terrain Classification for Mars Rover Missions . AIAA Space. Source
    BibTeX
    @inproceedings{rothrock2016spoc,
  title = {SPOC: Deep Learning-based Terrain Classification for Mars Rover Missions},
  author = {Rothrock, Brandon and Kennedy, Ryan and Cunningham, Chris and Papon, Jeremie and Heverly, Matthew and Ono, Masahiro},
  booktitle = {AIAA Space},
  year = {2016},
  doi = {10.2514/6.2016-5539}
}
  24. Goh, E., Chen, J. and Wilson, B. (2022). Mars Terrain Segmentation with Less Labels . IEEE Aerospace Conference. Source
    BibTeX
    @inproceedings{goh2022mars,
  title = {Mars Terrain Segmentation with Less Labels},
  author = {Goh, Edwin and Chen, Jingdao and Wilson, Brian},
  booktitle = {IEEE Aerospace Conference},
  pages = {1-10},
  year = {2022},
  doi = {10.1109/aero53065.2022.9843245},
  abstract = {Planetary rover systems need to perform terrain seg-mentation to identify drivable areas as well as identify specific types of soil for sample collection. The latest Martian terrain segmentation methods rely on supervised learning which is very data hungry and difficult to train where only a small number of labeled samples are available. Moreover, the semantic classes are defined differently for different applications (e.g., rover traversal vs. geological) and as a result the network has to be trained from scratch each time, which is an inefficient use of resources. This research proposes a semi-supervised learning framework for Mars terrain segmentation where a deep segmen-tation network trained in an unsupervised manner on unlabeled images is transferred to the task of terrain segmentation trained on few labeled images. The network incorporates a backbone module which is trained using a contrastive loss function and an output atrous convolution module which is trained using a pixel-wise cross-entropy loss function. Evaluation results using the metric of segmentation accuracy show that the proposed method with contrastive pretraining outperforms plain super-vised learning by 2%-10%. Moreover, the proposed model is able to achieve a segmentation accuracy of 91.1% using only 161 training images (1% of the original dataset) compared to 81.9% with plain supervised learning.}
}
  25. Castilla-Arquillo, R., Perez-del-Pulgar, C., Gerdes, L., Garcia-Cerezo, A. and Olivares-Mendez, M. A. (2025). OmniUnet: A Multimodal Network for Unstructured Terrain Segmentation on Planetary Rovers Using RGB, Depth, and Thermal Imagery . International Conference on Space Robotics. Source
    BibTeX
    @inproceedings{castillaarquillo2025omniunet,
  title = {OmniUnet: A Multimodal Network for Unstructured Terrain Segmentation on Planetary Rovers Using RGB, Depth, and Thermal Imagery},
  author = {Castilla-Arquillo, Raul and Perez-del-Pulgar, Carlos and Gerdes, Levin and Garcia-Cerezo, Alfonso and Olivares-Mendez, Miguel A.},
  booktitle = {International Conference on Space Robotics},
  pages = {161-167},
  publisher = {IEEE},
  year = {2025},
  doi = {10.1109/isparo66239.2025.11436158},
  abstract = {Robot navigation in unstructured environments requires multimodal perception systems that can support safe navigation. Multimodality enables the integration of complementary information collected by different sensors. However, this information must be processed by machine learning algorithms specifically designed to leverage heterogeneous data. Furthermore, it is necessary to identify which sensor modalities are most informative for navigation in the target environment. In Martian exploration, thermal imagery has proven valuable for assessing terrain safety due to differences in thermal behaviour between soil types. This work presents OmniUnet, a transformer-based neural network architecture for semantic segmentation using RGB, depth, and thermal (RGB-D-T) imagery. A custom multimodal sensor housing was developed using 3D printing and mounted on the Martian Rover Testbed for Autonomy (MaRTA) to collect a multimodal dataset in the Bardenas semi-desert in northern Spain. This location serves as a representative environment of the Martian surface, featuring terrain types such as sand, bedrock, and compact soil. A subset of this dataset was manually labeled to support supervised training of the network. The model was evaluated both quantitatively and qualitatively, achieving a pixel accuracy of $80.37 \%$ and demonstrating strong performance in segmenting complex unstructured terrain. Inference tests yielded an average prediction time of 673 ms on a resource-constrained computer (Jetson Orin Nano), confirming its suitability for onrobot deployment. The software implementation of the network and the labeled dataset have been made publicly available to support future research in multimodal terrain perception for planetary robotics.}
}
  26. Xie, Z., Yun, K., Liu, Y., Ji, Y. and Li, H. (2025). Mars Traversability Prediction: A Multi-modal Self-supervised Approach for Costmap Generation . arXiv. Source
    BibTeX
    @article{xie2025mars,
  title = {Mars Traversability Prediction: A Multi-modal Self-supervised Approach for Costmap Generation},
  author = {Xie, Zongwu and Yun, Kaijie and Liu, Yang and Ji, Yiming and Li, Han},
  journal = {arXiv},
  year = {2025},
  doi = {10.48550/arxiv.2509.11082},
  abstract = {We present a robust multi-modal framework for predicting traversability costmaps for planetary rovers. Our model fuses camera and LiDAR data to produce a bird's-eye-view (BEV) terrain costmap, trained self-supervised using IMU-derived labels. Key updates include a DINOv3-based image encoder, FiLM-based sensor fusion, and an optimization loss combining Huber and smoothness terms. Experimental ablations (removing image color, occluding inputs, adding noise) show only minor changes in MAE/MSE (e.g. MAE increases from ~0.0775 to 0.0915 when LiDAR is sparsified), indicating that geometry dominates the learned cost and the model is highly robust. We attribute the small performance differences to the IMU labeling primarily reflecting terrain geometry rather than semantics and to limited data diversity. Unlike prior work claiming large gains, we emphasize our contributions: (1) a high-fidelity, reproducible simulation environment; (2) a self-supervised IMU-based labeling pipeline; and (3) a strong multi-modal BEV costmap prediction model. We discuss limitations and future work such as domain generalization and dataset expansion.}
}
  27. Frey, J., Patel, M., Atha, D., Nubert, J., Fan, D., Agha, A., Padgett, C., Spieler, P., Hutter, M. and Khattak, S. (2024). RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving . arXiv. Source
    BibTeX
    @article{frey2024roadrunner,
  title = {RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving},
  author = {Frey, Jonas and Patel, Manthan and Atha, Deegan and Nubert, Julian and Fan, David and Agha, Ali and Padgett, Curtis and Spieler, Patrick and Hutter, Marco and Khattak, Shehryar},
  journal = {arXiv},
  year = {2024},
  doi = {10.1109/tfr.2024.3464369},
  abstract = {Autonomous navigation at high speeds in off-road environments necessitates robots to comprehensively understand their surroundings using onboard sensing only. The extreme conditions posed by the off-road setting can cause degraded camera image quality due to poor lighting and motion blur, as well as limited sparse geometric information available from LiDAR sensing when driving at high speeds. In this work, we present RoadRunner, a novel framework capable of predicting terrain traversability and an elevation map directly from camera and LiDAR sensor inputs. RoadRunner enables reliable autonomous navigation, by fusing sensory information, handling of uncertainty, and generation of contextually informed predictions about the geometry and traversability of the terrain while operating at low latency. In contrast to existing methods relying on classifying handcrafted semantic classes and using heuristics to predict traversability costs, our method is trained end-to-end in a self-supervised fashion. The RoadRunner network architecture builds upon popular sensor fusion network architectures from the autonomous driving domain, which embed LiDAR and camera information into a common Bird's Eye View perspective. Training is enabled by utilizing an existing traversability estimation stack to generate training data in hindsight in a scalable manner from real-world off-road driving datasets. Furthermore, RoadRunner improves the system latency by a factor of roughly 4, from 500 ms to 140 ms, while improving the accuracy for traversability costs and elevation map predictions. We demonstrate the effectiveness of RoadRunner in enabling safe and reliable off-road navigation at high speeds in multiple real-world driving scenarios through unstructured desert environments.}
}
  28. Cunningham, C., Whittaker, W. L. and Nesnas, I. A. (2017). Improving Slip Prediction on Mars Using Thermal Inertia Measurements . Robotics: Science and Systems XIII. Source
    BibTeX
    @inproceedings{cunningham2017improving,
  title = {Improving Slip Prediction on Mars Using Thermal Inertia Measurements},
  author = {Cunningham, Christopher and Whittaker, William L. and Nesnas, Issa A.},
  booktitle = {Robotics: Science and Systems XIII},
  publisher = {Robotics: Science and Systems Foundation},
  year = {2017},
  doi = {10.15607/rss.2017.xiii.038},
  abstract = {Rovers operating on Mars have been delayed, diverted, and trapped by loose granular materials.Vision-based mobility prediction approaches often fail because hazardous sand is difficult to distinguish from safe sand based on surface appearance alone.Unlike surface appearance, the thermal inertia of terrain is directly correlated to the same geophysical properties that control slip.This paper presents a quantitative analysis showing that considering thermal inertia improves rover slip prediction on Mars using in-situ data from the Curiosity rover.Thermal inertia is estimated for each slip measurement in sand using both on-board and orbital instruments.Slip models are learned using a mixture of experts approach where the experts are identified using thermal inertia.Two-expert models are compared to a single-expert, vision-only model to show that slip predictions are improved by separating high-slip, low thermal inertia sand from low-slip, high thermal inertia sand.These results support the hypothesis that the consideration of thermal inertia improves mobility estimates for rovers on Mars.}
}
  29. Iwashita, Y., Nakashima, K., Gatto, J., Higa, S., Khoo, N., Kurazume, R. and Stoica, A. (2020). Virtual IR Sensing for Planetary Rovers: Improved Terrain Classification and Thermal Inertia Estimation . International Symposium on Deep Space Communication and Navigation. Source
    BibTeX
    @inproceedings{iwashita2020virtual,
  title = {Virtual IR Sensing for Planetary Rovers: Improved Terrain Classification and Thermal Inertia Estimation},
  author = {Iwashita, Yumi and Nakashima, Kazuto and Gatto, Joseph and Higa, Shoya and Khoo, Norris and Kurazume, Ryo and Stoica, Adrian},
  booktitle = {International Symposium on Deep Space Communication and Navigation},
  publisher = {JPL Open Repository},
  year = {2020},
  doi = {10.48577/jpl.93jfiw}
}
  30. Ono, M., Fuchs, T. J., Steffy, A., Maimone, M. and Yen, J. (2015). Risk-aware planetary rover operation: autonomous terrain classification and path planning . IEEE Aerospace Conference. Source
    BibTeX
    @inproceedings{ono2015risk,
  title = {Risk-aware planetary rover operation: autonomous terrain classification and path planning},
  author = {Ono, Masahiro and Fuchs, Thoams J. and Steffy, Amanda and Maimone, Mark and Yen, Jeng},
  booktitle = {IEEE Aerospace Conference},
  pages = {1-10},
  publisher = {IEEE},
  year = {2015},
  doi = {10.1109/aero.2015.7119022},
  abstract = {Identifying and avoiding terrain hazards (e.g., soft soil and pointy embedded rocks) are crucial for the safety of planetary rovers. This paper presents a newly developed ground-based Mars rover operation tool that mitigates risks from terrain by automatically identifying hazards on the terrain, evaluating their risks, and suggesting operators safe paths options that avoids potential risks while achieving specified goals. The tool will bring benefits to rover operations by reducing operation cost, by reducing cognitive load of rover operators, by preventing human errors, and most importantly, by significantly reducing the risk of the loss of rovers. The risk-aware rover operation tool is built upon two technologies. The first technology is a machine learning-based terrain classification that is capable of identifying potential hazards, such as pointy rocks and soft terrains, from images. The second technology is a risk-aware path planner based on rapidly-exploring random graph (RRG) and the A* search algorithms, which is capable of avoiding hazards identified by the terrain classifier with explicitly considering wheel placement. We demonstrate the integrated capability of the proposed risk-aware rover operation tool by using the images taken by the Curiosity rover.}
}
  31. Castilla-Arquillo, R., Mandow, A., Pérez-del-Pulgar, C. J., Álvarez-Llamas, C., Vadillo, J. M. and Laserna, J. (2024). Thermal Imagery for Rover Soil Assessment Using a Multipurpose Environmental Chamber Under Simulated Mars Conditions . IEEE Transactions on Instrumentation and Measurement. Source
    BibTeX
    @article{castillaarquillo2024thermal,
  title = {Thermal Imagery for Rover Soil Assessment Using a Multipurpose Environmental Chamber Under Simulated Mars Conditions},
  author = {Castilla-Arquillo, Raúl and Mandow, Anthony and Pérez-del-Pulgar, Carlos J. and Álvarez-Llamas, César and Vadillo, José M. and Laserna, Javier},
  journal = {IEEE Transactions on Instrumentation and Measurement},
  volume = {73},
  pages = {1-12},
  publisher = {Institute of Electrical and Electronics Engineers (IEEE)},
  year = {2024},
  doi = {10.1109/tim.2023.3346528},
  abstract = {Planetary rover missions on Mars have suffered entrapments and serious mobility incidents due to soil assessment limitations of stereo RGB cameras, which cannot characterize relevant physical phenomena such as thermal behavior that depend on granularity and cohesion. In particular, thermal inertia estimations are already being used to assess geophysical properties from one-dimensional low-resolution measurements by onboard thermopiles. However, no high-resolution measurements are currently available to characterize Martian soils for safer navigation in future missions, so new experimental methods are required to capture and analyze thermal images with planetary conditions in Earth-based experiments. In this work, we propose a novel measurement system configuration and experimental methodology to capture thermal images using isolated multipurpose environmental chambers to replicate the temperature and pressure conditions of Mars. Furthermore, the system has allowed to measure diurnal cycles for four soil types of known physical characteristics under Martian and Earth pressures to perform a unique quantitative analysis and comparison of thermal behavior and thermal inertia for soil assessment. Even if no actual Martian infrared (IR) images are available for comparison, results indicate a correlation between granularity and thermal inertia that is consistent with available thermopile measurements recorded by rovers onsite. Furthermore, the set of measurements acquired in the experiments has been made available to the scientific community.}
}
  32. Marchetti, Y., Lightholder, J., Junkins, E., Cross, M., Mandrake, L. and Yates, D. (2021). Barefoot Rover: A Sensor-Embedded Rover Wheel Demonstrating In-Situ Engineering and Science Extractions Using Machine Learning . IEEE International Conference on Robotics and Automation. Source
    BibTeX
    @inproceedings{marchetti2021barefoot,
  title = {Barefoot Rover: A Sensor-Embedded Rover Wheel Demonstrating In-Situ Engineering and Science Extractions Using Machine Learning},
  author = {Marchetti, Yuliya and Lightholder, Jack and Junkins, Eric and Cross, Matthew and Mandrake, Lukas and Yates, Devon},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://ntrs.nasa.gov/citations/20220004834},
  abstract = {No abstract provided}
}
  33. Brooks, C. A. and Iagnemma, K. (2012). Self-supervised terrain classification for planetary surface exploration rovers . Journal of Field Robotics. Source
    BibTeX
    @article{brooks2012self,
  title = {Self-supervised terrain classification for planetary surface exploration rovers},
  author = {Brooks, Christopher A. and Iagnemma, Karl},
  journal = {Journal of Field Robotics},
  volume = {29},
  pages = {445-468},
  year = {2012},
  doi = {10.1002/rob.21408},
  abstract = {Abstract In future planetary exploration missions, improvements in autonomous rover mobility have the potential to increase scientific data return by providing safe access to geologically interesting sites that lie in rugged terrain, far from landing areas. To improve rover‐based terrain sensing, this paper proposes a self‐supervised learning framework that will enable a robotic system to learn to predict mechanical properties of distant terrain, based on measurements of mechanical properties of similar terrain that has been traversed previously. In this framework, a proprioceptive terrain classifier is used to distinguish terrain classes based on features derived from rover–terrain interaction, and labels from this classifier are used to train an exteroceptive (i.e., vision‐based) terrain classifier. Once trained, the vision‐based classifier is able to recognize similar terrain classes in stereo imagery. This paper presents two distinct proprioceptive classifiers—a novel approach based on optimization of a traction force model and a previously described approach based on wheel vibration—as well as a vision‐based terrain classification approach suitable for environments with unexpected appearances. The high accuracy of the self‐supervised learning framework and its supporting algorithms is demonstrated using experimental data from a four‐wheeled robot in an outdoor Mars‐analogue environment. © 2012 Wiley Periodicals, Inc.}
}
  34. Bai, C., Guo, J., Guo, L. and Song, J. (2019). Deep Multi-Layer Perception Based Terrain Classification for Planetary Exploration Rovers . Sensors, 14. Source
    BibTeX
    @article{bai2019deep,
  title = {Deep Multi-Layer Perception Based Terrain Classification for Planetary Exploration Rovers},
  author = {Bai, Chengchao and Guo, Jifeng and Guo, Linli and Song, Junlin},
  journal = {Sensors},
  volume = {19},
  number = {14},
  pages = {3102},
  publisher = {MDPI AG},
  year = {2019},
  doi = {10.3390/s19143102},
  abstract = {Accurate classification and identification of the detected terrain is the basis for the long-distance patrol mission of the planetary rover. But terrain measurement based on vision and radar is subject to conditions such as light changes and dust storms. In this paper, under the premise of not increasing the sensor load of the existing rover, a terrain classification and recognition method based on vibration is proposed. Firstly, the time-frequency domain transformation of vibration information is realized by fast Fourier transform (FFT), and the characteristic representation of vibration information is given. Secondly, a deep neural network based on multi-layer perception is designed to realize classification of different terrains. Finally, combined with the Jackal unmanned vehicle platform, the XQ unmanned vehicle platform, and the vibration sensor, the terrain classification comparison test based on five different terrains was completed. The results show that the proposed algorithm has higher classification accuracy, and different platforms and running speeds have certain influence on the terrain classification at the same time, which provides support for subsequent practical applications.}
}
  35. Vulpi, F., Milella, A., Marani, R. and Reina, G. (2021). Recurrent and convolutional neural networks for deep terrain classification by autonomous robots . The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Source
    BibTeX
    @article{vulpi2021recurrent,
  title = {Recurrent and convolutional neural networks for deep terrain classification by autonomous robots},
  author = {Vulpi, Fabio and Milella, Annalisa and Marani, Roberto and Reina, Giulio},
  journal = {The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences},
  volume = {96},
  pages = {119-131},
  publisher = {Elsevier BV},
  year = {2021},
  doi = {10.1016/j.jterra.2020.12.002}
}