Stereo Vision

Stereo vision is the range sensor for every Mars rover flown. There is no lidar and no radar on any of them: the terrain model that hazard detection, path selection and arm placement all consume is a point cloud produced by correlating two camera images [1]. Its cost on flight processors set the drive rate of every mission before Mars 2020.

The algorithm

The flight pipeline is an area-based correlator using the sum of absolute differences over a rectangular window, typically 7x7, searched along the epipolar line [3][5]. The steps are rectification using a geometric lens calibration, bandpass filtering, the SAD correlation over the disparity search range, consistency checks, subpixel disparity by quadratic interpolation of the SAD scores, and projection to XYZ range images [1][3]. A left-right uniqueness check and a threshold on the disparity gradient remove matches that are not supported by both views [5].

The variant used from 2011 onward adds SAD5: the score at each disparity is the SAD1 score of the center pixel plus the lowest two SAD1 scores of the four corner pixels at that disparity level, which improves range at object edges and range discontinuities [5].

Image sizes and what sets them

Full frame readout from the MER cameras is about 5 s per 1024x1024 12 bit image, so images for navigation are binned vertically in the CCD and read out at 256x1024, then averaged down to 256x256, and stereo matching runs at 256x256 [1]. The output is a 256x256 disparity image in which each pixel either encodes the 3D location of the terrain seen at that pixel in the rectified left image, or is unknown [2].

Vehicle and camera	Field of view	Typical 3D points per pair
Spirit, front and rear hazard cameras	125 degrees	about 15,000
Opportunity, navigation cameras	45 degrees	about 48,000

Source: [2]. Instrument placement uses a different configuration: one or two front hazard camera pairs at 1024x1024, downsampled to 512x512 and cropped to 420x412 for a coarse view, followed by a full resolution 241x241 subframe of the patch of terrain the arm will touch.

Camera geometry decides what the stereo can see. The wide field of view hazard cameras were sized to see more than the full width of the rover a short distance ahead, which is what obstacle avoidance and turn-in-place safety checks need, but their wide field of view and narrow baseline limit useful lookahead to 3 to 4 m. The navigation cameras see further with a narrower field of view and wider baseline, but their field of view is only wide enough to verify one candidate path several meters ahead [1]. The panoramic cameras are not used for autonomous navigation [1].

Range accuracy against distance

Range error is set by the accuracy of the subpixel disparity estimate, which is why the correlator’s subpixel stage rather than its integer search decides how good the range is [3]. Evaluated for the flight camera geometries at 0.25 pixel correlation accuracy [7]:

	MER and MSL navigation cameras	MER panoramic cameras	MSL panoramic cameras
Baseline	0.20 m	0.30 m	0.20 m
Resolution	1024x1024	1024x1024	1200x1200
Field of view	45 x 45 degrees	16 x 16 degrees	6 x 6 to 50 x 50 degrees
Range error at 50 m	2.5 m	0.60 m	0.20 m at maximum zoom

Source: [7], Table 1. Range error grows steeply with standoff, which is why the hazard camera lookahead limit is a geometry limit rather than a software one, and why traversability assessment weights nearby map cells more heavily than distant ones [1].

Cost on flight hardware

Platform	Configuration	Time per stereo pair
MER flight processor, 20 MHz RAD6000	256x256 SAD	24 to 30 s
Xeon 5160 at 3 GHz	rectification, bilateral subtraction filter, disparity	7024 ms total, of which 74 ms disparity
Core 2 Quad at 2.4 GHz	same pipeline	8827 ms total, of which 87 ms disparity
Virtex-4 LX160 at 66 MHz	rectification, bilateral filter, SAD1 at 1024x768	15 Hz fed directly from Camera Link, 12 Hz when raw imagery is loaded over the PCI bus
Perseverance Vision Compute Element, Virtex5QV	stereo correlation and visual odometry	22 million disparities per second

Sources: [1] for MER, [5] for the FPGA and desktop comparison, [4] for Perseverance. The FPGA figure in [5] is for a 1024x768 SAD1 system consuming 134 block RAMs and 49,924 slices on a Virtex-4 LX160 class part; a 512x384 SAD5 system takes 162 block RAMs and 38,581 slices. On the general purpose processors in that comparison, the bilateral subtraction filter rather than the correlation dominates, at 6947 ms of the 7024 ms total.

The per-step cost across missions tracks the pixel count rather than the clock alone [4]:

	Sojourner	Spirit and Opportunity	Curiosity	Perseverance
Stereo pixels processed per step	20	10,000 to 50,000	40,000 to 200,000	240,000 to 1,200,000
Image pairs per step	1	1 to 2	4	1
Autonomous navigation pause per step	not reported	about 120 s	about 120 s	typically 0 if not steering

Source: [4], Table 1. Perseverance processes about six times as many pixels per step as Curiosity in less time because the correlation runs in the Virtex-5 FPGA on the Vision Compute Element rather than on the RAD750.

The imaging software that feeds the correlator is itself a scheduling problem. MER Imaging Services holds requests in priority queues, two per priority level, and hands each image to a separate post-processing task as soon as the pixels are read so the camera control task can start the next exposure. Image buffers are a limited shared pool, and ground imaging commands queue when none is free [6].

Failure modes

Texture starvation. Opportunity could not get acceptable range data from its hazard cameras at Meridiani: the fine texture of the rock-free soil gave inadequate texture for correlation at hazard camera resolution and baseline. The workaround was to use navigation camera stereo instead, and because Meridiani is largely obstacle free, it was sufficient to check the traversability of the nominal path forward and stop the vehicle if a hazard was detected [1]. This is the same terrain property that starves visual odometry, and it is the dominant flight failure mode for both.

Range discontinuities. Correlation windows that straddle a depth edge produce a blended disparity. SAD5 exists specifically to improve range at object edges and range discontinuities, and the disparity gradient threshold exists to discard the results that remain wrong [5].

Calibration. Range depends on the camera model. The MER images are labeled with geometric camera models shortly after acquisition, mapping pixel space to the body-fixed frame shared by all subsystems, so a calibration change propagates into every consumer of the range data [6].