Terrain Classification

Geometric hazard detection sees rocks and slopes. It cannot see sand. A patch of cohesionless soil is flat, smooth and low in the goodness map, and it is also where a rover sinks. Terrain classification exists to type the surface material so that a slip model can be applied to it, turning a non-geometric hazard into a cost the path planner can avoid [1]. Slip is learned from previous experience rather than derived from a soil model, because measuring the mechanical properties of the terrain ahead is not feasible from a moving vehicle, and a single imaging step for onboard hazard assessment already costs minutes on flight-class hardware, leaving no budget for a physical characterization pass [2, 11]. It has not flown on any rover; the flight record is field-tested research, and the mechanical counterpart is what has actually been deployed.

The problem the geometry misses

The idea of reading traction from an image rather than measuring it predates the JPL slip-prediction architecture. A 2001 rover safety module paired a vision-based neural network, trained to classify the terrain ahead as low or high traction, with a fuzzy controller that throttled commanded speed continuously rather than stopping at a threshold [3]. The classifier’s own accuracy was never reported in that work, and the demonstrated runs used hand-assigned traction values rather than the network’s output, but the shape of the idea, appearance predicts traction before the wheels get there, is the shape every later system keeps [3]. The next step was predicting slip itself rather than a traction category, from a single learned model relating appearance to slip magnitude [2].

On the Meridiani plains the only geometric obstacles were occasional hollows from small infilled craters, and inside Eagle and Endurance craters the binding hazards were slope and slip rather than obstacles [4]. Opportunity measured slip rates of 98.9 to 99.5 percent while embedded in the Purgatory ripple, and Spirit recorded slip up to 125 percent in the Columbia Hills, meaning it moved backwards during a commanded forward drive. A large slip region is an obstacle that no elevation, tilt or roughness test detects [4].

Visual typing and remote slip prediction

The JPL architecture built for MSL-class navigation predicts slip at a distance from imagery, in four stages [1]:

1. The goodness map is built from navigation camera stereo, then terrain triage bins every cell into definitely traversable, definitely not traversable, or uncertain. Triage runs twice, once before slip prediction to avoid predicting slip in cells already ruled out, and once after to fold the slip cost back into the map.
2. For a cell needing prediction, the terrain classifier is applied to the cells in its neighborhood and a majority vote gives the class at that location.
3. A locally linear fit over the neighborhood gives longitudinal and lateral slope under the prospective rover footprint.
4. Those slope angles enter a pre-learned nonlinear slip model specific to the classified terrain type, learned by receptive field regression, giving predicted slip and a slip cost for the cell.

The classifier itself is appearance-based: a texton representation over color and texture, in which frequent color features in small pixel neighborhoods are selected, a histogram of their occurrence within the image patch corresponding to a map cell is built, and a nearest neighbor classifier compares it against a database of training patches [1]. The texton representation treats texture as a union of features with specific appearances and no regard to their location, with random feature sampling over large patches for speed [5].

The architectural decision that makes this affordable is classifying per map cell rather than per image. Pixels above the horizon are never processed, the classifier is not invoked at all in cells already marked as obstacles or otherwise uninteresting, and near-range cells that cover a large part of the image can be processed selectively. A map cell stores only its 3D location and pointers to the images that observed it; when a class is needed, the cell is projected into an image and the corresponding patch retrieved [1]. The same structure lets stereo imagery arrive asynchronously or intermittently, since only the most recent patch for a cell is used when a classification is invoked.

What it costs

Classifier accuracy and cost trade directly against each other on the same LAGR data set of six off-road terrains, soil, sand, gravel, asphalt, grass and woodchips, over 100 runs on 200 randomly sampled test patches [5]:

Representation	Error	Time per patch
Average red channel	40.9 +/- 1.7 percent	0.06 +/- 0.01 s
Average color	17.5 +/- 2.6 percent	0.06 +/- 0.01 s
Color histogram	14.0 +/- 3.3 percent	0.57 +/- 0.03 s
Texton based	8.1 +/- 1.5 percent	4.21 +/- 0.32 s
Texton based, slow variant	7.9 +/- 2.4 percent	6.26 +/- 0.42 s

Source: [5], Table 1, on a research workstation rather than flight hardware; the authors note the times are machine dependent and should be read in relative terms. The texton classifier’s accuracy is judged satisfactory and its computation time explicitly not acceptable for a real-time system.

The response is a hierarchical classifier that spends effort where the discrimination is hard: with a 3 s budget the selection algorithm picks average color, color histogram and texton in sequence, discarding the average red channel as no cheaper than average color and much worse, and discarding the slow texton variant as too expensive, for an expected error of 11.7 percent [5]. In practice one class, grass, is discarded after the first level and the remainder split into sand, soil and woodchip against gravel and asphalt at the second, so five of six classes still reach the bottom level; splitting alone reduces time without hurting accuracy, and the larger saving comes from abandoning high-confidence examples early, which introduces some error. Image patches are 100 pixels across for cells at 1 to 2 m and 10 to 15 pixels across for cells at 5 to 6 m [5], which is the resolution limit behind the range failure described below [1].

Map resolution in the integrated demonstration was 10 cm cells for both the goodness and triage maps, with paths planned by D* [1]. The high-fidelity traversability analysis that runs on cells triage marks uncertain, a full kinematic and dynamic forward simulation resolving 18 contact forces, three at each of six wheels, costs about 4 s per meter of path analyzed [1]. On the actual MER hardware two decades earlier, the equivalent tradeoff was starker still: a single onboard visual odometry or hazard-avoidance imaging step took 2 to 3 minutes on a 20 MHz flight processor already running more than 90 tasks, which is why blind driving at about 120 m/hr fell to 5 m/hr with both hazard avoidance and visual odometry engaged [11]. Every terrain-classification budget since has been set against that kind of ceiling, not against a workstation’s.

Measured performance and where it fails

On the LAGR vehicle across five terrains, sand, soil, gravel, asphalt and woodchips, with slip models and classifier trained on 3000 frames and tested on 2000 non-intersecting frames, average slip prediction error was about 21 percent [1]. If the terrain type is classified correctly the error falls to about 11 percent, so misclassification contributes the larger share. Misclassification concentrates between visually similar terrains, specifically sand against soil, and worst against soil areas covered with dust. On Mars that is the failure case that matters, because dust covers everything; Opportunity’s hazard camera stereo was defeated at Meridiani by exactly this class of surface, whose fine texture gave inadequate correlation [4].

The second failure mode is range. Most terrain classification errors in the integrated runs occur beyond 6 m, where the image patch corresponding to a map cell is very small for the LAGR camera configuration [1]. Classification accuracy therefore degrades exactly where the map is least constrained by other evidence.

Slip prediction from visual appearance alone reached about 15 percent error on the earlier single-terrain-model formulation [2], which is comparable to the measurement error of slip itself.

Mechanical typing, which did fly

Curiosity’s Traction Control estimates the wheel-terrain contact angles in real time from a rocker-bogie kinematics model driven by suspension encoders and measured attitude rates, and commands the idealized no-slip angular rate for each of the six wheels [7]. It was written because wheel puncture damage appeared in October 2013 at an unexpected rate, driven by five wheels pushing a sixth over an embedded sharp rock when all six turn at one speed. It does not classify the terrain by name; it measures the vehicle’s mechanical response to it, which is the information the visual classifier is trying to predict.

It flew as a hot patch on top of R12 rather than as a flight software release, and it adds less than 3 percent to total CPU usage while driving [7]. Wheel speed ratios are evaluated at 8 Hz; the average Traction Control speed ratio over the first six months of operation, sols 1646 through 1822, was 0.899. Checkout was a 5 m drive on sol 1646 and a 20 m drive on sol 1662, both logging drive telemetry at 64 Hz, with nominal use from sol 1678 [7]. Ground testing on engineered terrain also found an unexpected mode, a wheelie in which the middle wheel lifts off under suspension tension in high-friction terrain, which required a suppression behavior in the algorithm.

The same reasoning drove Approximate Clearance Evaluation: rather than classify the terrain, bound what the suspension would do on it [8]. A more direct alternative sits between the two: a sensorized wheel that reads soil hydration, composition, slip and sink from pressure-grid and electromagnetic-induction sensing in the wheel itself, in no-light conditions the camera cannot use at all [6]. It has been demonstrated only on a stationary bench rig with an unspecified regolith simulant, not on a driving vehicle, so it answers a different part of the problem than the visual classifier without yet replacing it [6].

What is not established

The visual architecture described above has not flown, and the gap between it and flight is not just qualification. A recent attempt to remove the terrain-classification step entirely, predicting a per-segment slip cost map directly from a learned terrain segmentation and in-situ wheel telemetry, reports its numbers from a single simulated rover model in a physics engine, not from a physical vehicle or a characterized regolith simulant, and its own baseline comparison loses on one of five held-out test samples [9]. Whether a segmentation-to-slip mapping generalizes off a simulator has not been shown.

The mechanical alternative to camera-based typing has the same evidentiary shape in a different place: force-torque sensors mounted above the wheels of a half-scale rover testbed classify terrain better than the chassis inertial measurement unit, 95.6 versus 85.8 percent in one comparison, but the same field campaign could not extract a usable drawbar-pull signal from the sensors at all, because vibration swamped it, and the authors recommend against the mounting position that produced the classification result [10]. No published work has run the camera-based classifier and a wheel-based sensor on the same drive over the same ground truth, so which one degrades first under dust, and whether they fail on correlated terrain or independent terrain, is unanswered. The economics that would decide a flight choice between them, the CPU and power cost of vision-based triage against the mass and wiring cost of instrumenting six wheels, have not been compared either.