Stereo Vision

Stereo vision is the range sensor for every Mars rover flown. There is no lidar and no radar on any of them: the terrain model that hazard detection, path selection and arm placement all consume is a point cloud produced by correlating two camera images [1]. Its cost on flight processors set the drive rate of every mission before Mars 2020.

The algorithm

The flight pipeline is an area-based correlator using the sum of absolute differences over a rectangular window, typically 7x7, searched along the epipolar line [3][5]. The steps are rectification using a geometric lens calibration, bandpass filtering, the SAD correlation over the disparity search range, consistency checks, subpixel disparity by quadratic interpolation of the SAD scores, and projection to XYZ range images [1][3]. A left-right uniqueness check and a threshold on the disparity gradient remove matches that are not supported by both views [5].

The variant used from 2011 onward adds SAD5: the score at each disparity is the SAD1 score of the center pixel plus the lowest two SAD1 scores of the four corner pixels at that disparity level, which improves range at object edges and range discontinuities [5].

Image sizes and what sets them

Full frame readout from the MER cameras is about 5 s per 1024x1024 12 bit image, so images for navigation are binned vertically in the CCD and read out at 256x1024, then averaged down to 256x256, and stereo matching runs at 256x256 [1]. The output is a 256x256 disparity image in which each pixel either encodes the 3D location of the terrain seen at that pixel in the rectified left image, or is unknown [2].

Vehicle and camera	Field of view	Typical 3D points per pair
Spirit, front and rear hazard cameras	125 degrees	about 15,000
Opportunity, navigation cameras	45 degrees	about 48,000

Source: [2]. Instrument placement uses a different configuration: one or two front hazard camera pairs at 1024x1024, downsampled to 512x512 and cropped to 420x412 for a coarse view, followed by a full resolution 241x241 subframe of the patch of terrain the arm will touch.

Camera geometry decides what the stereo can see. The wide field of view hazard cameras were sized to see more than the full width of the rover a short distance ahead, which is what obstacle avoidance and turn-in-place safety checks need, but their wide field of view and narrow baseline limit useful lookahead to 3 to 4 m. The navigation cameras see further with a narrower field of view and wider baseline, but their field of view is only wide enough to verify one candidate path several meters ahead [1]. The panoramic cameras are not used for autonomous navigation [1].

Range accuracy against distance

Range error is set by the accuracy of the subpixel disparity estimate, which is why the correlator’s subpixel stage rather than its integer search decides how good the range is [3]. Evaluated for the flight camera geometries at 0.25 pixel correlation accuracy [7]:

	MER and MSL navigation cameras	MER panoramic cameras	MSL panoramic cameras
Baseline	0.20 m	0.30 m	0.20 m
Resolution	1024x1024	1024x1024	1200x1200
Field of view	45 x 45 degrees	16 x 16 degrees	6 x 6 to 50 x 50 degrees
Range error at 50 m	2.5 m	0.60 m	0.20 m at maximum zoom

Source: [7], Table 1. Range error grows steeply with standoff, which is why the hazard camera lookahead limit is a geometry limit rather than a software one, and why traversability assessment weights nearby map cells more heavily than distant ones [1].

Cost on flight hardware

Platform	Configuration	Time per stereo pair
MER flight processor, 20 MHz RAD6000	256x256 SAD	24 to 30 s
Xeon 5160 at 3 GHz	rectification, bilateral subtraction filter, disparity	7024 ms total, of which 74 ms disparity
Core 2 Quad at 2.4 GHz	same pipeline	8827 ms total, of which 87 ms disparity
Virtex-4 LX160 at 66 MHz	rectification, bilateral filter, SAD1 at 1024x768	15 Hz fed directly from Camera Link, 12 Hz when raw imagery is loaded over the PCI bus
Perseverance Vision Compute Element, Virtex5QV	stereo correlation and visual odometry	22 million disparities per second

Sources: [1] for MER, [5] for the FPGA and desktop comparison, [4] for Perseverance. The FPGA figure in [5] is for a 1024x768 SAD1 system consuming 134 block RAMs and 49,924 slices on a Virtex-4 LX160 class part; a 512x384 SAD5 system takes 162 block RAMs and 38,581 slices. On the general purpose processors in that comparison, the bilateral subtraction filter rather than the correlation dominates, at 6947 ms of the 7024 ms total.

The per-step cost across missions tracks the pixel count rather than the clock alone [4]:

	Sojourner	Spirit and Opportunity	Curiosity	Perseverance
Stereo pixels processed per step	20	10,000 to 50,000	40,000 to 200,000	240,000 to 1,200,000
Image pairs per step	1	1 to 2	4	1
Autonomous navigation pause per step	not reported	about 120 s	about 120 s	typically 0 if not steering

Source: [4], Table 1. Perseverance processes about six times as many pixels per step as Curiosity in less time because the correlation runs in the Virtex-5 FPGA on the Vision Compute Element rather than on the RAD750.

The imaging software that feeds the correlator is itself a scheduling problem. MER Imaging Services holds requests in priority queues, two per priority level, and hands each image to a separate post-processing task as soon as the pixels are read so the camera control task can start the next exposure. Image buffers are a limited shared pool, and ground imaging commands queue when none is free [6].

Failure modes

Texture starvation. Opportunity could not get acceptable range data from its hazard cameras at Meridiani: the fine texture of the rock-free soil gave inadequate texture for correlation at hazard camera resolution and baseline. The workaround was to use navigation camera stereo instead, and because Meridiani is largely obstacle free, it was sufficient to check the traversability of the nominal path forward and stop the vehicle if a hazard was detected [1]. This is the same terrain property that starves visual odometry, and it is the dominant flight failure mode for both.

Range discontinuities. Correlation windows that straddle a depth edge produce a blended disparity. SAD5 exists specifically to improve range at object edges and range discontinuities, and the disparity gradient threshold exists to discard the results that remain wrong [5].

Calibration. Range depends on the camera model. The MER images are labeled with geometric camera models shortly after acquisition, mapping pixel space to the body-fixed frame shared by all subsystems, so a calibration change propagates into every consumer of the range data [6].

References

Maimone, M., Johnson, A., Cheng, Y., Willson, R. and Matthies, L. (2004). Autonomous Navigation Results from the Mars Exploration Rover (MER) Mission . International Symposium on Experimental Robotics (ISER). Source

BibTeX

@inproceedings{maimone2004autonomous,
  title = {Autonomous Navigation Results from the Mars Exploration Rover (MER) Mission},
  author = {Maimone, Mark and Johnson, Andrew and Cheng, Yang and Willson, Reg and Matthies, Larry},
  booktitle = {International Symposium on Experimental Robotics (ISER)},
  address = {Singapore},
  year = {2004},
  url = {https://dataverse.jpl.nasa.gov/dataset.xhtml?persistentId=hdl:2014/41077}
}

Maimone, M. W., Leger, P. C. and Biesiadecki, J. J. (2007). Overview of the Mars Exploration Rovers' Autonomous Mobility and Vision Capabilities . IEEE International Conference on Robotics and Automation, Space Robotics Workshop. Source

BibTeX

@inproceedings{maimone2007overview,
  title = {Overview of the Mars Exploration Rovers' Autonomous Mobility and Vision Capabilities},
  author = {Maimone, Mark W. and Leger, P. Chris and Biesiadecki, Jeffrey J.},
  booktitle = {IEEE International Conference on Robotics and Automation, Space Robotics Workshop},
  address = {Rome, Italy},
  year = {2007},
  url = {https://www-robotics.jpl.nasa.gov/media/documents/mer_autonomy_icra_2007.pdf}
}

Ansar, A., Castano, A. and Matthies, L. (2004). Enhanced Real-time Stereo Using Bilateral Filtering . Proceedings, International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT). Source

BibTeX

@inproceedings{ansar2004enhanced,
  title = {Enhanced Real-time Stereo Using Bilateral Filtering},
  author = {Ansar, Adnan and Castano, Andres and Matthies, Larry},
  booktitle = {Proceedings, International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT)},
  pages = {455--462},
  year = {2004},
  doi = {10.1109/tdpvt.2004.1335273},
  abstract = {In recent years, there have been significant strides in increasing quality of range from stereo using global techniques such as energy minimization. These methods cannot yet achieve real-time performance. However, the need to improve range quality for real-time applications persists. All real-time stereo implementations rely on a simple correlation step which employs some local similarity metric between the left and right image. Typically, the correlation takes place on an image pair modified in some way to compensate for photometric variations between the left and right cameras. Improvements and modifications to such algorithms tend to fall into one of two broad categories: those which address the correlation step itself (e.g., shiftable windows, adaptive windows) and those which address the preprocessing of input imagery (e.g. band-pass filtering, Rank, Census). Our efforts lie in the latter area. We present in this paper a modification of the standard band-pass filtering technique used by many SSD- and SAD-based correlation algorithms. By using the bilateral filter of Tomasi and Manduchi [(1998)], we minimize blurring at the filtering stage. We show that in conjunction with SAD correlation, our new method improves stereo quality at range discontinuities while maintaining real-time performance.}
}

Rankin, A., Del Sesto, T., Hwang, P., Justice, H., Maimone, M., Verma, V. and Graser, E. (2023). Perseverance Rapid Traverse Campaign . IEEE Aerospace Conference. Source

BibTeX

@inproceedings{rankin2023perseverance,
  title = {Perseverance Rapid Traverse Campaign},
  author = {Rankin, Arturo and Del Sesto, Tyler and Hwang, Pauline and Justice, Heather and Maimone, Mark and Verma, Vandi and Graser, Evan},
  booktitle = {IEEE Aerospace Conference},
  pages = {1-16},
  address = {Big Sky, Montana},
  year = {2023},
  doi = {10.1109/aero55745.2023.10115835},
  abstract = {Over the first 13 months of the Mars 2020 mission, the Perseverance rover traversed nearly 5 km along the Jezero Crater floor. Near the end of that period, the Science team was anxious to relocate to the ancient Delta region near the crater rim, over 5 km away. A Rapid Traverse Campaign was planned that would prioritize use of Perseverance's autonomous navigation software to drive at an unprecedented high pace and minimize science activities. The Rapid Traverse Campaign started in March 2022 and lasted 31 Martian days. During the campaign, Perseverance drove over 5 km in 24 drives, during which its autonomy software planned 94.8% of its overall driving, enabling it to set several new planetary rover driving records. Perseverance exceeded the longest daily drive distance record achieved by a previous planetary rover (219 meters) 11 times and set new records for the longest multi-sol drive distance in a single plan (528.7 meters) and the longest continuation drive (699.9 meters) by operating without human drive path input during 3 sols of driving. This paper details the planning and execution of the Rapid Traverse Campaign.}
}

Villalpando, C. Y., Morfopoulos, A., Matthies, L. and Goldberg, S. (2011). FPGA Implementation of Stereo Disparity with High Throughput for Mobility Applications . IEEE Aerospace Conference. Source

BibTeX

@inproceedings{villalpando2011fpga,
  title = {FPGA Implementation of Stereo Disparity with High Throughput for Mobility Applications},
  author = {Villalpando, Carlos Y. and Morfopoulos, Arin and Matthies, Larry and Goldberg, Steven},
  booktitle = {IEEE Aerospace Conference},
  pages = {1--11},
  address = {Big Sky, Montana},
  year = {2011},
  doi = {10.1109/aero.2011.5747269},
  abstract = {High speed stereo vision can allow unmanned robotic systems to navigate safely in unstructured terrain, but the computational cost can exceed the capacity of typical embedded CPUs. In this paper, we describe an end-to-end stereo computation co-processing system optimized for fast throughput that has been implemented on a single Virtex 4 LX160 FPGA. This system is capable of operating on images from a 1024 × 768 3CCD (true RGB) camera pair at 15 Hz. Data enters the FPGA directly from the cameras via Camera Link and is rectified, pre-filtered and converted into a disparity image all within the FPGA, incurring no CPU load. Once complete, a rectified image and the final disparity image are read out over the PCI bus, for a bandwidth cost of 68 MB/sec. Within the FPGA there are 4 distinct algorithms: Camera Link capture, Bilinear rectification, Bilateral subtraction pre-filtering and the Sum of Absolute Difference (SAD) disparity. Each module will be described in brief along with the data flow and control logic for the system. The system has been successfully fielded upon the Carnegie Mellon University's National Robotics Engineering Center (NREC) Crusher system during extensive field trials in 2007 and 2008 and is being implemented for other surface mobility systems at JPL.}
}

Litwin, T. E. and Maki, J. N. (2005). Imaging Services Flight Software on the Mars Exploration Rovers . IEEE International Conference on Systems, Man and Cybernetics. Source

BibTeX

@inproceedings{litwin2005imaging,
  title = {Imaging Services Flight Software on the Mars Exploration Rovers},
  author = {Litwin, Todd E. and Maki, Justin N.},
  booktitle = {IEEE International Conference on Systems, Man and Cybernetics},
  volume = {1},
  pages = {895--900},
  address = {Waikoloa, Hawaii},
  year = {2005},
  doi = {10.1109/icsmc.2005.1571260},
  abstract = {The imaging services module of the Mars Exploration Rovers' on-board flight software is responsible for providing image data to the rest of the system. It acquires images from a suite of cameras, performs on-board image processing, labels the results with metadata, and delivers the final products to a diverse set of consumers, both on board and on the ground. The demands for flexibility and speed led to a design involving multiple tasks and a large set of parameters controlling the acquisition of the images, the on-board processing, and the method of product delivery.}
}

Helmick, D. M., Angelova, A., Livianu, M. and Matthies, L. H. (2007). Terrain Adaptive Navigation for Mars Rovers . IEEE Aerospace Conference. Source

BibTeX

@inproceedings{helmick2007terrain,
  title = {Terrain Adaptive Navigation for Mars Rovers},
  author = {Helmick, Daniel M. and Angelova, Anelia and Livianu, Matthew and Matthies, Larry H.},
  booktitle = {IEEE Aerospace Conference},
  pages = {1-11},
  address = {Big Sky, Montana},
  year = {2007},
  doi = {10.1109/aero.2007.352684},
  abstract = {A navigation system for Mars rovers in very rough terrain has been designed, implemented, and tested on a research rover in Mars analog terrain. This navigation system consists of several technologies that are integrated to increase the capabilities compared to current rover navigation algorithms. These technologies include: goodness maps and terrain triage, terrain classification, remote slip prediction, path planning, high-fidelity traversability analysis (HFTA), and slip-compensated path following. The focus of this paper is not on the component technologies, but rather on the integration of these components. Results from the onboard integration of several of the key technologies described here are shown. Additionally, the results from independent demonstrations of several of these technologies are shown. Future work will include the demonstration of the entire integrated system.}
}