Perception

Perception turns sensor data into a model of the surroundings that planning can act on: a range map, a statement of where the vehicle can put itself, and a typing of the surface it would put itself on. On the surface of Mars the processor sets the terms. The Mars Exploration Rovers ran a 20 MHz RAD6000, stereo matching at 256x256 took 24 to 30 s per pair, a full obstacle avoidance cycle about 70 s, and a visual odometry step about 160 s, which pulled median net driving speed to roughly 0.6 cm/s against a 5 cm/s peak [1]. At most 75 percent of that processor was available to any one capability [2]. Perception on these vehicles is a compute budget before it is an algorithm.

Design space and why flown systems converged

Three families of sensing have been tried. Passive stereo is the only one that has flown on rovers. Binocular depth from disparity alone was demonstrated with random-dot stereograms in 1960 [10], and dense two-frame matchers have since been compared by matching cost, aggregation and optimization, on terrestrial benchmark scenes rather than rover imagery [11]. The planetary line starts with the Stanford Cart, which crossed a 20 m indoor course in about five hours [7], and with a 1980 Mars-oriented stereo chain that fitted a ground surface, called anything above a height threshold an object, and was exercised on Viking Lander 1 imagery with thresholds of 3 to 7 cm [6]. That criterion is still the basis of geometric hazard detection.

Active ranging is the second family. A Pathfinder-derived laser hazard sensor split a laser into co-planar beams and found obstacle heights in one exposure with peak detection, running a scan in 830 ms on a 486 against an estimated 4 s on the 8085 the Pathfinder microrover carried [9]. For landing, the ALHAT hazard detection system flew a gimbaled flash lidar on the Morpheus lander in 2014, mosaicking a 60 m by 60 m elevation map from about 460 m slant range and returning ranked safe sites within 11 to 13 s [8]. That was a terrestrial rocket flight; no flash lidar has flown on a planetary rover. The SPLICE hazard detection software, tested on lidar elevation maps from a March 2025 helicopter campaign, reached true positive rates of 50 to 70 percent and false positive rates of 0.4 to 0.6 percent at downsampling steps of 5 and 7 pixels, from three post-processed maps at one site [21]. Terrain-relative navigation, which supports hazard avoidance by fixing position before the hazard map is built, flew on Mars 2020: the Lander Vision System cut a 3.2 km initial position error to under 40 m within 10 s from 4200 m altitude, in one landing [26]. For lunar landing, a two-stage Gaussian process map that learns spatially varying noise from orbital elevation confidence was evaluated offline on 961 South Pole tiles and 500 synthetic tiles, with no flight sensor or onboard compute tested [22].

Flown rover systems converged on small-window correlation at reduced image size because the budget left no alternative. The Perseverance campaign moved that correlator into dedicated hardware, which is what ended the stereo-limited drive rate of earlier missions [12]. Learned stereo is the newest family and has not left the laboratory: a wavelet-enhanced depth network reported an absolute relative error of 0.089 against 0.103 for a baseline network, on simulated Zhurong-like imagery [13]. An analysis of 16,768 Perseverance NavCam stereo pairs found that the flight block matcher leaves navigation-stopping holes at solar phase angles under 20 degrees and on low-texture terrain; RAFT-Stereo and IGEV++ run zero shot matched flight accuracy (Bad1.0 of 0.74 and 0.94 percent against 0.80 percent) while raising disparity coverage by 22.8 percent at low phase, with no true ground truth behind the comparison [20].

What the maps are used for

Hazard detection consumes the range map. The MER method, GESTALT, treats the rover as a point and inflates obstacles by its radius. Autonomous terrain assessment and path selection covered 1354 of Spirit’s 4798 m and 1379 of Opportunity’s 5947 m, about 25 percent of all distance, and visual odometry covered 792 and 682 m [2]. Approximate Clearance Evaluation, flown on Perseverance, keeps the wheels and suspension in the check and bounds clearance analytically, at 10 to 15 ms per pose on a RAD750 and 11.2 microseconds on a Jetson TK1 in test [3]. Because those calls dominate the planning cycle, a learned ranking of candidate paths cut the overthink rate from 20.0 to 7.1 percent in simulation; its cost on flight hardware was not measured [4].

Terrain classification addresses what geometry cannot see. Sand is flat and smooth and is where a rover sinks. A 2001 safety module paired a vision-based network that classified terrain ahead as low or high traction with a fuzzy speed controller; the network’s accuracy was never reported [15]. The JPL line predicts slip at a distance from stereo appearance and slope: about 21 percent average error on a research vehicle across five terrains, about 11 percent when the class was right, and most errors beyond 6 m [5]. The classifier trade is explicit: a texton representation reached 8.1 percent error at 4.21 s per patch on a workstation, against 17.5 percent at 0.06 s for average color [14]. Segmentation networks extend this to obstacle typing: LuSeg separates rocks and craters at 92.79 percent mIoU on a rendered lunar dataset, at 17.45 ms per frame on a desktop GPU, so neither real regolith nor flight hardware is represented [23]. RoadRunner predicts traversability cost and elevation from camera and lidar input, but is scored against the pseudo labels of the stack that trained it [24]. AI4Mars supplies about 326,000 crowdsourced labels over 35,000 rover images in four terrain classes, a benchmark and not a flight result [25]. RELLIS-3D, an off-road campus dataset with 6,235 annotated images and 13,556 annotated lidar scans, gave baseline mIoU of 52.92 and 51.55 percent for two image networks and 43.07 and 19.97 percent for two lidar networks, on terrain with no planetary analog [30]. None of the visual classifiers has flown.

Perception also supports absolute localization. A crater detection front end for lunar rovers, tested on simulated imagery, gave 3 sigma crater center errors of 0.28 m at 5 m diameter to 1.94 m at 20 m for lidar, and 1.02 to 4.48 m for stereo over the same range; the real Chang’e imagery supports only qualitative statements [28]. It is a proposal and has not flown. Field data for such work is scarce: BASEPROD records about 1.7 km over 24 teleoperated traverses of a half-scale ExoMars model in a Spanish semi-desert, with RTK GNSS ground truth, and exercises no autonomous navigation [29].

Perception also directs manipulation. The Chang’e-6 sampler used deep-learning stereo reconstruction from the lander binocular camera to choose scoop points against slope, relief and obstacle criteria, and point cloud registration for a 6D sampler pose in place of the manual fine-tuning used on Chang’e-5 [27]. Mission imagery shows the ten planned sample points and the resulting pits, but every accuracy figure, including 3.8 mm average scooping depth error and 1.1 mm pose precision, comes from an Earth testbed with balloon offload, and no on-orbit accuracy was reported [27].

What flew in the mobility loop is mechanical. Curiosity’s traction control estimates wheel-terrain contact angles from suspension kinematics and adds under 3 percent to CPU load [17]. A sensorized wheel reading soil composition and slip has run only on a stationary bench rig [16].

What changed

Period	Range and hazard method	Where it ran
1980	Ground-surface fit, height-threshold objects [6]	Viking Lander 1 imagery, offline
2004	SAD stereo, GESTALT [1]	Flight, 20 MHz RAD6000
2020	Analytic clearance bound, ACE [3]	Flight, RAD750
2014	Flash lidar hazard map [8]	Terrestrial rocket lander only
2024	Learned slip cost map [18]	Simulation only

The row order follows the subsystem’s families, not strict chronology.

What is not established

Flight evidence is thin outside stereo, hazard checking and mechanical traction control. The MER usage percentages show how often operators trusted a capability, not how well it worked, and carry no false hazard rates [2]. The planner speedups for learned ranking are arithmetic conversions from evaluation counts, with no inference cost on a RAD750 [4]. The integrated terrain-adaptive stack was never run as a whole system [5]. A learned slip cost map reports results from one simulated rover model [18], and force-torque terrain classification on a half-scale testbed (95.6 against 85.8 percent for the inertial unit) came from a campaign that could not recover a usable drawbar-pull signal [19]. No published work compares the camera-based and wheel-based approaches on the same drive.