Onboard Science Autonomy

Onboard science autonomy is the selection of what to measure, or what to keep, by software running on the vehicle rather than by an operator on the ground. It exists because the observation opportunity and the uplink cycle do not coincide: a rover that finishes a drive after the last uplink of the sol is standing in front of unmeasured rock until the next plan arrives, and an orbiter that images a target does not know whether the image is worth its downlink allocation [3][1]. The argument for moving recognition onto the spacecraft predates every flown system here: with smaller, more numerous missions the binding limit on science return is downlink, not instrument capability, and the first prototypes were ground runs on workstations with no flight-processor timing [19].

Five families of approach exist, and they differ in what the software decides and how much of it has flown. Event detection and acquisition triggering decides whether an observation is worth downlinking or repeating, and flew on Earth Observing 1 and on the Mars Exploration Rovers [1][12]. Target selection for a pointed instrument chooses where to aim, and flew on three rovers as AEGIS [4][5]. Opportunistic science couples image analysis to a planner that decides whether the rover can afford to act on a detection, and has been demonstrated on rover testbeds but not flown [9][10]. Adaptive sampling chooses when and where to take the next measurement, and has been demonstrated in field trials and in retrospective analysis only [13]. Data prioritization ranks products already collected for a constrained downlink, and has been shown in simulation and replay of flight data, not on flight hardware [15][16]. Scheduling the activities that carry these decisions is a sixth problem, treated at the end of the page.

Autonomous Sciencecraft Experiment, EO-1

ASE flew on Earth Observing 1, launched 21 November 2000 into a 705 km sun-synchronous orbit at 98.7 degrees inclination with 16 day repeat tracks [1]. Each scene from the Advanced Land Imager, Hyperion and Atmospheric Corrector produced over 20 Gbit onto the solid state recorder; Hyperion resolves 220 bands from 0.4 to 2.5 micrometers at 30 m ground sample over a 7.5 x 42 km footprint.

The autonomy software is three layers [1]:

Layer	Component	Time scale
Mission planning	CASPER, Continuous Activity Scheduling Planning Execution and Replanning	tens of minutes per planning cycle
Execution	SCL, Spacecraft Command Language, plus an independent fault protection layer	several seconds
Flight software	EO-1 flight software, low level control	below SCL

The closed loop is: CASPER plans periodic Hyperion imaging of a target list; onboard science algorithms analyze the image; a detection triggers downlink of that image, and a non-detection instead posts a goal to acquire the next highest priority target, which CASPER merges into the running plan [1]. Conflict repair is explicit: given a projected solid state recorder memory conflict, CASPER enumerates repair methods, selects one, and picks an activity to move or delete from those that subscribe to the contended resource [2]. The classifier set covers thermal anomaly detection from infrared spectral peaks, cloud detection from thresholds on six bands, flood scene classification from spectral ratios, change detection, trainable feature detection and Gabor filter anomaly detection [1]. Trigger-based acquisition reduces downlinked data volume. By July 2005 ASE had become the primary operations system for EO-1 (November 2004) and had acquired over 2400 images autonomously; the 100 times gain in science return quoted for it is given as events captured per megabyte downlinked, with no baseline or measurement period stated, so it cannot be checked from the source [18]. Its planner descends from the Remote Agent, which commanded Deep Space 1 in 1999 and was the first AI system to do so, built from a generative temporal planner, a plan executive and a model-based mode identification component [17].

What it cost on a Mongoose M5

EO-1 carries two Mongoose M5 processors, one for command and data handling and one inside the Wideband Advanced Recorder Processor. Each runs at 12 MHz for about 8 MIPS with 256 MB of RAM under VxWorks, and ASE runs on the WARP M5 so that it cannot disturb the main spacecraft processor [1]. Against a contemporary desktop at 2000 to 3000 MIPS, 5 to 20 MIPS was the typical spacecraft figure.

The allocation ASE actually received was narrower than the part [2]:

Resource	EO-1 total	Allocated to ASE
Processor	8 MIPS Mongoose V, MIPS R3000	4 MIPS, shared between SCL, CASPER and science processing
RAM	256 MB	40 MB for CASPER code and heap, of which a 32 MB heap; 32 MB was already the original flight software
Filesystem	none in the original design	8 MB ramdisk, 1 MB of it for CASPER log files
Uplink	2 kbit/s to 2 Mbit/s, 10 to 15 minute contacts	CASPER image uplinked compressed

Getting CASPER inside that envelope was an exercise in subtraction, against a planning cycle whose duration is on the order of tens of minutes [1]. The image began at 12 MB against 8 MB of code and heap space on the testbed; removing software components took about 3 MB off, moving to a reduced C++ standard library implementation took about another 3 MB, and addressing code duplication brought the final image to 5 MB [2]. For uplink, a 9 MB image was compressed to 1.5 MB and the existing flight software was patched to decompress it into memory, because contacts are short and infrequent.

The planning horizon was cut for the same reason. A single science observation expands to over 78 activities in the CASPER model, so a week of EO-1 operations, over 100 science observations and 50 ground contacts, would be over 7800 activities and exhaust the 32 MB heap; scheduling cost also grows with goal count through goal interaction [2]. The answer was a hierarchical horizon: abstract planning across the week enforcing constraints such as one observation per orbit, detailed planning only across the next 24 hours, with past observations dropped out of the window automatically.

ASE completed over 5000 data collection events against an original success criterion of 1000 [1].

The same SCL executive and CASPER planner were slated for Three Corner Sat, three 15 kg uncontrolled nanosatellites with a 40 MHz PowerPC 750 and 16 MB of non-volatile plus 16 MB of dynamic RAM shared with the operating system and imagery [20]. Because the spacecraft do not control attitude, most frames were expected to miss the Earth, so the planned onboard science check was image compressibility rather than a spectral detector [20]. That paper is a pre-launch plan, not a flight result.

AEGIS, MER, MSL and Mars 2020

AEGIS, Autonomous Exploration for Gathering Increased Science, selects instrument targets from an onboard image. It first flew on Opportunity in 2010 for Pancam target selection, then on Curiosity for ChemCam in two roles, autonomous target selection in NavCam images and autonomous pointing refinement in RMI images, then on Perseverance for SuperCam [4][5].

The pipeline is: acquire a NavCam stereo source image; run the Rockster segmentation algorithm, which is edge detection followed by flood fill and morphological operations to close contours; evaluate each candidate on properties including pixel count, a stereo 3D size estimate and the semi-major and semi-minor axes of a fitted ellipse; filter and rank under an operator-supplied scene profile; check the chosen pointing against a sun-safety exclusion zone and against an onboard model of rover articulation for collision; place a visual target tracking frame on the target; and command the spectrometer raster [4][5]. Two safety checks exist because ChemCam’s 15 cm primary mirror can focus sunlight into the instrument and damage it, and because a laser shot at the rover body would trip a collision fault; the collision check projects the current position of all rover hardware into the NavCam image and masks those pixels [3]. To the end of the period reported, AEGIS on Curiosity had never selected a target on the rover body.

AEGIS first flew on Opportunity: the software was uploaded in December 2009 and sixteen Mars runs are tabulated through sol 2550, each returning at most ten top targets for Pancam, with counts judged by the authors and no precision or recall reported [32]. The Mini-TES was nonfunctional, so only Pancam was used, and the onboard planner was dropped, leaving the ground to pre-allocate resources [32]. On Curiosity the operational cost is 3 to 4 minutes of plan time per run, and by the time of the SpaceOps report 138 AEGIS-selected post-drive targets had been measured, with the ChemCam rate rising from 256 to 320 LIBS shots per sol and AEGIS present in 63 percent of multi-sol plans with drives [27]. Those figures differ from the 255, 311 and 64 percent quoted below from other reports of the same system because the reporting windows differ; a 2019 slide deck and a 2023 slide deck carry later statistics through sol 2054 without error bars [25][23].

What it cost on the Rover Compute Element

Run duration on the Curiosity flight RCE scales with the number of targets found, because each target needs its own stereo computation and feature extraction; through sol 2830 AEGIS visually identified a target set in under 500 s in the usual case, on the spacecraft’s RAD750 [4]. Point stereo alone costs on the order of a second per target plus a few seconds of setup, and a source image can yield up to 255 targets, so stereo dominates the long runs. The same 133 MHz RAD750 takes 47 s for a visual odometry step [8].

Measured performance

Vehicle	Instrument	First operational use	Flight record
Opportunity	Pancam	2010	first flight demonstration of autonomous target selection [4]
Curiosity	ChemCam	sol 1343, planned 13 May 2016	54 runs selecting 59 targets between sols 1343 and 1663, of which 52 were post-drive NavCam runs and 2 were RMI pointing refinements [3]
Perseverance	SuperCam	sol 449, 24 May 2022, expanded capability from sol 698	170 targets selected by 25 September 2024 [5]

No AEGIS run on Curiosity failed to find targets or produced a poor quality observation: in every case the system detected feasible targets in the source image, ranked them, selected the commanded number, and obtained the ChemCam measurement [3]. After the sol 1400 parameter update, every LIBS point landed on outcrop for 44 of the 47 targets measured; modeled blind targeting of the same scenes would have managed that for only 10. The three residual cases are partial: two measurements fell on loose float rock rather than bedrock in fields of view holding little or no outcrop, and in one case three of nine LIBS points fell on sand covering the outcrop while the other six struck the outcrop itself. AEGIS was used on 53 percent of plans containing a drive, 36 percent of single-sol plans and 64 percent of multi-sol plans [3]. Target selection takes 150 to 450 seconds on the rover’s RAD750 flight computer, scaling nearly linearly with the number of targets found.

Pointing refinement recovers observations rather than creating them, and had been used only twice by sol 1663. On sols 1424 and 1463 the ground-selected pointing missed a bright vein; AEGIS recognized the offset and corrected it, saving the observation [3]. The mast points to about 2 mrad, the RMI field of view is 19 mrad across and the 3x3 LIBS grid is spaced at 1 mrad, so the residual error matters for features that small [3]. A second attempt in a later plan would otherwise have been required.

Throughput at instrument level changed at rollout. ChemCam laser shot rate rose from about 255 shots per sol before the sol 1343 AEGIS rollout to about 311 shots per sol after it [4]. Target class distribution over 281 AEGIS runs since rollout differs from the notional blind-target distribution over 215 cases, with the dominant class taking 75 to 77 percent of AEGIS runs against 46 to 47 percent notionally blind.

On Perseverance the measure is share of the science plan. Over the 83 sol drive campaign from sol 1092 to 1175, covering 2.44 km from Bunsen Peak to Bright Angel, the team prepared 48 uplink plans and used AEGIS in 45 of them; AEGIS supplied half or more of the LIBS-VISIR targets in 35 plans and all of them in 12; of 112 targets measured in the period, 56 were human-selected and 56 were AEGIS-selected [5].

Event detection on the Mars Exploration Rovers

WATCH, uploaded to both Mars Exploration Rovers in the R9.2 flight software load in summer 2006 [12], screens Navcam imagery on the rover and keeps only frames containing a detected event. Dust devils are found by differencing a test frame against an average of noise-reduced frames; clouds are found in a single frame by locating the horizon and looking for high-variance sky regions [12]. Across 26 sols of dust devil sequences an event was detected on 21 sols and 120 of 393 images held a detection, a 70 percent reduction in images returned. The detector was tuned toward false alarms because a missed event cannot be recovered, and of 533 flagged features 76 percent were nearby ground and 18 percent candidate dust devils on the horizon; no recall is reported, so the reduction counts what was flagged, not what was detected correctly [12].

Opportunistic science on rover testbeds

OASIS segments rocks from rover imagery, extracts albedo, texture and shape, and raises science alerts by matching a scientist-supplied target signature or by novelty detection against rocks already seen [9][10]. On the FIDO rover in the JPL Mars Yard a closed loop was demonstrated: image, rock detection, property analysis, alert, and a planner check of resource margin [10]. The planner treats an alert as a soft, optional goal inside a continuous planner, so the science-versus-resources decision belongs to the planner, and the schedule slack that makes room for it comes from deliberate slack, conservative duration estimates and time recovered by driving faster than predicted [10]. The planning paper reports no plan-quality scores, runtimes or success rates, and its four scenarios are scripted demonstrations with alerts fired at chosen times [10]. The earlier OASIS paper states its validation by expert pair-comparison rankings in the future tense and reports no detection rate [9]. The CLEaR framework adds an execution-time plan manager and resource manager to the planner, but its paper is a concept with one illustrated scenario and no timing or success figures [14]. A 2004 CLEaR status briefing reports replanning latency in seconds rather than minutes and a 50 m waypoint traverse on rover hardware, without trial counts; opportunistic science was still simulation-only at that date [11]. None of this has flown.

Adaptive sampling and survey

PIXL on Perseverance is the one flown case of a decision made from the composition of the sample itself. Onboard software re-bins each summed X-ray fluorescence spectrum into 22 pseudo-intensities and applies linear rules trained on Earth; a trigger extends a 10 s dwell with a 30 s long dwell [22]. Over more than 13,300 points on sols 847 to 921 the Cr2O3, TiO2 and P2O5 rules reached true positive rates of 98.4, 86.2 and 93.1 percent with false positive rates below 0.25 percent, and the carbonate rule reached 12.2 percent [22]. The truth labels come from the ground quantification pipeline the rules approximate, so the rates measure agreement with a ground model and not mineralogy.

Field work on the Zoe rover in the Atacama Desert combined spectrometer calibration, visual target detection and pointing, and an adaptive path planner driven by orbital data. The planner benefit is shown in simulation at Cuprite, Nevada, adaptive navigation ran in only two field instances, and a registration offset above 100 m between orbital data and rover GPS made the executed path differ from the plan [29]. A companion position paper reports a 35 percent reconstruction improvement from a Cuprite airborne-image simulation, with no rover traverse behind it [30].

Adaptive sensing of time series fits a Gaussian process to the measurements so far and picks the next sample time by information gain; refit and replan took 0.023 s on average on a laptop [13]. It was evaluated retrospectively, by subsampling visible near-infrared spectra collected at Amboy Crater under fixed 2.5 m spacing, so no closed-loop field result exists [13].

Detection for small bodies

An onboard plume detector for comet rendezvous applies Canny edges, a convex hull of the nucleus and an intensity threshold outside it. Run time is about 0.1 s at 256 pixels square and 7.0 s at 2506 pixels on unnamed ground hardware, and it was checked as a hit or miss table over roughly 25 archival images [31]. A simulated Rosetta campaign credits onboard detection with following up far more 16 day events than a 14 day ground cycle, which reaches about 25 percent of them [31]. Rosetta did not carry the system.

Data prioritization for the downlink

SCOTI trains a VGG-19 and LSTM captioning network on about 1,250 expert captions of Mars Science Laboratory images and ranks images for downlink against text queries; it is a proof of concept with no flight-processor run, no runtime figures and a validation split from the same dataset [15]. SYNOPSIS is a C library integrated into cFS and F-Prime flight software and exercised in the NOS3 simulator; it computes a science utility estimate and an eight-dimensional diversity descriptor per observation and ranks products by maximum marginal relevance. A replay of ChemCam LIBS and RMI data from 11 sample sites (sols 2828 to 2837) showed utility accumulating faster than a first-in first-out queue under a hypothetical degraded bandwidth [16]. Both remain ground or simulation demonstrations.

Failure modes

AEGIS fails by declining to act. If every candidate is filtered out by the scene profile, or falls outside the SuperCam laser range, or carries a poor stereo range estimate, no coordinate frame is placed and the observation is not attempted [5]. Curiosity has the same residual: runs that found no targets are limited to a mast fault that pointed the source image at the sky, an image centered on the rover’s own arm, and one entry into a new geological unit where nothing in the scene met the science suitability settings [4]. The documented worst case is geometric rather than algorithmic: an autonomous drive ended with the source image pointing west on a slope that tilted the camera upward enough to put the sun in the NavCam field of view, which left few targets found and all of them filtered for poor range confidence, and the fix was an operational constraint on source image azimuth for the rest of that campaign [5].

Scene profile choice moves the failure mode rather than removing it. The Perseverance default favors light-toned outcrop because in-place rock is usually lighter than the surrounding regolith, which means that in scenes with sparse, small, dark or shadowed float rocks it will select light-toned regolith that the Rockster edge finder has enclosed in a trace; rougher regolith texture makes this more likely because it yields more enclosable edges [5].

Rock detection is the dominant compute cost, and the processor decides how often it can run. Rockster took 10 to 15 minutes on the 20 MIPS RAD6000 of a Mars Exploration Rover under a full flight software load; the same algorithm over a 116 image regression set averaged 2.9 s on one core of a 750 MHz commercial 64 core board and 0.3 s on 32 cores [33]. Nearly 75 percent of the single-core time was software emulated floating point, and speedup saturated near 10 times at 32 cores. These are ground measurements on an evaluation board, not on the radiation-hardened derivative, and detection quality under parallelization was not demonstrated [33].

Pointing precision is the standing physical limit on the whole capability. Motor backlash, motor control error, resolver accuracy, thermal effects and rover settling or shifting give a targeting uncertainty of about 6 mrad, reducible to about 2 mrad by commanded compensation, against LIBS rasters whose points are spaced 1 to 2 mrad apart and which span 2 to 20 mrad [4]. Mars 2020 closes the loop with vision and resolver feedback, designed for 1 mrad, and models backlash disturbance to guide the actuator slew along a path where backlash is controlled.

For a planner, the failure mode is state divergence. On EO-1 the spacecraft consumed more solid state recorder memory blocks than planned when data collection ran several seconds long, so the telemetered free block count disagreed with CASPER’s model, which is the class of defect that onboard planner telemetry exists to expose: iteration counters, elapsed time per repair, stack margin, and pre-repair and post-repair conflict counts [2].

Scheduling as the adjacent problem

Planning research for self-reliant rovers and outer-planet missions has been tested on ground hardware. Pathogen orders activities by campaign intent and was evaluated on planning-time and battery sweeps and a 7-sol Mars Yard walkabout; it is a slide deck with no repeated trials [26]. MEXEC restores a plan after a flight software reset, resolving plan problems in under 0.1 s on a 200 MHz RAD750 testbed for the planned Europa mission, with other subsystems simulated [28]. MEXEC was later extended for the Endurance lunar rover concept, and in an October 2025 Mars Yard test a half scale rover drove a 200 waypoint, 4.1 km route in 5.2 hours of a 6.5 hour test with four unplanned navigation interventions [21]. None of these has flown.

For instrument analysis the compute question is whether flight processors can host it at all. Holographic and fluorescence microscopy summaries compress by factors near 1888 and 2247 and ran in about 108 s on a 2.1 GHz laptop-class CPU, but projections place a RAD750 or LEON4 short of the Europa Lander case by a large factor, and those projections are scaled rather than benchmarked [24].

The Mars 2020 onboard scheduler addresses the cost of the conservatism that ground scheduling requires. Activities are sequenced with conservative durations so that overruns almost never cancel a dependent activity, targeted at under one plan in 100; a study of Mars Science Laboratory campaigns found submaster sequences finishing on average more than 20 percent early, which leaves the rover awake and idle until the next fixed start time [6]. The onboard scheduler is priority-first and does not backtrack: for each activity it finds valid temporal intervals, accommodates required preheat and maintenance heating, and generates a wake and sleep schedule, and it will not modify an already scheduled activity. On the ground side, the Copilot scheduler has generated and scheduled Perseverance’s sleep and heating activities alongside the user-supplied ones since the February 2021 landing, sharing its core algorithms with the onboard planner, with the explainable scheduling tool Crosscheck over it so that operators can see how the schedule was constructed [7]. The power a heating activity draws depends on temperature and temperature varies through the sol, so the energy cost of an activity and its heaters is not constant across the plan horizon; state of charge rises only when the CPU is off and nothing else is drawing power [7].

What is not established

Flight evidence exists for event triggering (ASE, WATCH) and for target selection (AEGIS); everything else above is a testbed, field or replay result. The AEGIS comparison against blind targeting on Curiosity is modeled rather than executed, rests on 47 targets, and the two pointing refinement runs are too few to characterize that capability [3]. No quantitative comparison of AEGIS targets against human-selected targets on Perseverance has been published, and no onboard runtime for the Perseverance version is reported [5]. WATCH reports no recall [12]. Opportunistic science papers report no measured planner cost or success rate [10][9][14]. The Mars 2020 onboard scheduler overview gives no runtime, memory or priority budget, and its 20 percent early-completion figure is inherited from a separate MSL study [6].