Compression and Prioritization

A surface rover produces far more data than its downlink can carry in a sol, on a link whose rate is itself set by antenna geometry, the deep space network schedule and Mars-Earth range rather than by anything onboard. Every camera on Spirit and Opportunity produced a 1024x1024 image at 12 bits per pixel, and the average returned rate was 1.13 bits per pixel [1]. That ratio, roughly ten to one, is what made a Mars imaging mission possible on a link that returns about 56 Mb/sol. Reducing the byte count is only half the problem; the other half is deciding, sol by sol and pass by pass, which of the products worth sending goes first when not everything will fit. What is compressed, how hard, and in what order it is sent are three separate decisions, made in the flight compressor’s design, in ground-set byte quotas, and in a priority table the operations team revises for every communication window.

The compressors

MER carries three: ICER for lossy and progressive compression, a modified LOCO for lossless, and LZO for engineering data [7]. Sojourner’s Mars Pathfinder used a Rice compressor and JPEG [2]. One Pathfinder color mosaic used a 24:1 ratio on the red filter image [9] and 48:1 on the green and blue images [9]. ICER’s wavelet-based progressive stream degrades gracefully under packet loss in a way a block-transform codec does not, and it gives an operations team a byte-quota knob tied directly to output size.

ICER is wavelet-based and progressive: the compressed stream is organized so that reconstructions of successively higher quality can be produced as more of it arrives [1]. It partitions the image into independent error-containment segments so that a lost packet damages only its segment, which mattered because a fair number of MER packets were lost. Its parameters are the wavelet filter, the number of decomposition stages, the number of error-containment segments, the byte quota and a minimum loss parameter.

LOCO exists alongside it because although ICER can compress losslessly, LOCO’s simple predictive approach is several times faster at similar effectiveness [1]. On six Mars Pathfinder test images at 12 bits per pixel, lossless rates averaged 8.78 bits/pixel for ICER, 8.84 for LOCO, 9.10 for JPEG 2000 and 9.36 for the CCSDS Rice compressor; the Rice implementation actually used by Pathfinder was worse still, ranging from 9.38 to 10.61 bits/pixel across the five single-frame images [2]. On the same images companded to 8 bits, ICER and LOCO both averaged 5.02 bits/pixel against 5.22 for JPEG 2000 and 5.52 for CCSDS Rice.

What was actually selected in flight

As of 7 February 2004 [1]:

Compressor	Image type	Images	Combined area, Mpixel	Volume, MB	Average rate, bits/pixel
ICER	regular	5132	2867.9	387.3	1.13
ICER	thumbnail	6075	24.9	3.8	1.27
LOCO	regular, lossless	489	124.5	92.7	6.24
LOCO	reference pixels	1114	36.5	9.2	2.12
none	regular	4	4.2	6.0	12.00

Source: [1], Table 1. Four images in the entire set were returned uncompressed. Lossless LOCO at 6.24 bits/pixel costs about five and a half times the volume of the ICER default, so it was reserved for cases where fidelity was the point [1].

Parameter selection varied by camera and by purpose [1]. Panoramic camera full frames on Spirit used 3 or 4 decomposition stages, wavelet filters A or E, 1 to 20 segments, and averaged 1.12 bits/pixel over 1014 images. Navigation camera full frames used 12 to 20 segments and averaged 0.97 bits/pixel over 235 images, the lowest rate of any full frame category, because navigation images are not science products [1]. Microscopic Imager frames took 24 to 32 segments and averaged 1.34 bits/pixel on Spirit and 2.36 on Opportunity. Hazard camera full frames used 16 to 32 segments, the highest segment counts, because they are the images a lost packet can least afford to damage [1].

Two mechanisms reduce volume before compression runs. Images are companded from 12 to 8 bits per pixel to improve fidelity in dark regions, which is worthwhile because the sensor noise is signal dependent [1]. Every acquired image also produces a 64x64 thumbnail by averaging 16x16 blocks, ICER-compressed to about 650 bytes, which is about 0.005 bits per pixel of the original; thumbnails preview full images that may not be sent for some time. Beyond that, MER sends subframes of interest at full quality with the whole frame at much lower quality, performs pixel averaging, and sends row sums, column sums or histograms alone.

What it cost on the flight processor

Timing for the MER implementation of ICER was measured under VxWorks on a 20 MHz RAD6000 identical to the flight units [2]. Three results hold regardless of the absolute scale: compression is faster when fewer compressed bytes are produced, the time at a bit rate of zero is essentially the cost of the wavelet decomposition alone, and the choice of wavelet filter and the number of decomposition stages have little effect on speed. The published figure gives compression time against output rate for a 1024x1024 test image with one error-containment segment, but its vertical axis is labeled in milliseconds per pixel, which cannot be literal at this image size, so the absolute value is not usable from the open document [2].

The byte quota does not give exact control over output size. ICER checks the quota only between compression of segments of subband bit planes, and does not account for the backlog still held in the segments’ interleaved entropy coders, so the output almost always exceeds the quota when the quality goal is not reached [2]. The overshoot depends on the segment count through two competing effects: more segments mean a larger total coder backlog, but also more frequent quota checks, so overshoot is minimized at a moderate segment count. MER controlled quality with the byte quota alone, setting the minimum loss parameter to zero, and transmitted the entire bitstream rather than truncating to the exact target, because truncation produces small quality variations between segments [1].

Standards

CCSDS publishes two relevant Blue Books: Image Data Compression, CCSDS 122.0-B-2, and Lossless Data Compression, CCSDS 121.0-B-3, the latter being the Rice-based algorithm that appears in the comparison above [5][6]. ICER-3D extends the ICER approach to hyperspectral imagery by compressing across the spectral dimension as well as the spatial ones [3]. No Mars surface vehicle has flown the CCSDS image compression standard; the flight compressors are mission-specific.

Later missions

MSL kept the mission-specific pattern but moved back toward a standard codec for its Mastcams and Descent Imager: most Mastcam and MARDI images are downlinked as lossy JPEG, compressed onboard when the scientific need for full fidelity does not outweigh the downlink volume it costs, with a lossless mode available separately for the cases that need it [10]. That is a narrower design than MER’s: one general-purpose codec with a selectable quality setting [10], rather than a wavelet compressor tuned per camera and per segment count.

Compression of non-image data is a separate, quieter problem that the imaging literature does not cover. On Mars 2020, GZIP compression of non-image telemetry removed more than two thirds of the volume actually generated in the first 216 sols, taking 88.8 Gibit of source data down to 27.5 Gibit on the link, against a daily total data return of 1.5 to 2 Gibit per sol [11]. That is compression applied to housekeeping and engineering products, not camera frames, and it exists because a rover generates far more non-image telemetry than any single downlink pass can carry either. The link that has to carry it is itself limited: the M2020 UHF return link tops out at 2048 kbit/s with LDPC encoding [11].

How downlink priority is set

Priority on MER is a property of the data product and it is set and revised from the ground. Each communication window command carries the durations for the real-time and recorded data priority tables along with start time, duration, antenna, rates and hardware configuration, and the window executes inside the communications behavior part of the flight software rather than through the sequence engine [4]. Data products carry a priority in their metadata file alongside collection time [8].

Onboard data organization places the highest priority playback data, fault and warning event reports and spacecraft health reports, earliest in a direct-to-Earth pass [4]. That ordering interacts with receiver lockup: lockup takes about 1 minute for high gain antenna rates but varies from pass to pass, so the project must plan a lockup allowance before the valuable playback starts. Too short and sent data is lost; too long and less data fits. The comm window lockup parameter was changed from 1 minute to 3 minutes for downlink rates of 3160 bit/s and higher, and 3 minutes consumes 10 percent of the return capacity of a 30 minute window; it was reduced to 2 minutes for some windows in the later extended missions [4].

Reprioritization is a routine operations activity. UHF reports were generated for every Odyssey pass at low priority in the extended mission by attaching a report sequence to the start of window preparation; to actually return one, data management issued a command raising its priority. Reports were reprioritized every seventh sol so telecom could spot-check relay performance, could be raised individually for interesting passes, and were marked for automatic deletion after 7 sols [4].

Priority also serves fault diagnosis. During the Spirit sol 18 flash anomaly a high priority communication window overrode the 10 bit/s fault default to give 40 or 300 bit/s on the low gain antenna, and the repeating event reports that came down at that rate are what first pointed at the flash memory [4].

What is not established

The MER compression numbers are drawn from a fixed operational snapshot: the vehicle-wide totals are as of a single date in February 2004, and the earlier per-image test that justified choosing ICER over LOCO and JPEG 2000 used six Pathfinder images, five of them the same scene from one camera, with the authors stating outright that the filter ranking may not hold for other image types [1][2]. No compression time figure from the flight processor survives with a usable absolute scale; the published curve is mislabeled and only the qualitative relationships it shows are usable [2]. The ICER-3D hyperspectral results are airborne AVIRIS data, not a flight compressor running on any space vehicle, and its own authors show it losing to a simpler compressor in the lossless case [3]. No source cited above gives a byte-quota overshoot statistic, a rate-distortion or image-quality number for any flown compressor, or a packet loss rate for MER’s downlink; the interaction between segment count and overshoot is described only qualitatively [1][2]. The priority and lockup mechanics are reconstructed from a single mission’s operations account, written by the team that ran it, so the generalization from MER’s tables and lockup allowances to other missions’ downlink scheduling is not demonstrated here [4].