Ground Operations Tooling

The software that runs a planetary rover is not the flight software. It is the set of ground tools a few dozen people use, once per sol, inside a shift whose length is fixed and whose inputs arrive on the relay schedule. What the tools cost is measured in sols of science lost, not in milliseconds.

RSVP, the sequencing tool

The Rover Sequencing and Visualization Program was MER’s command-level editor, built as independent applications plugged into an executive and coordinating by messages over a message-passing library derived from the Oak Ridge PVM implementation [1]. The architecture choice was deliberate: clean enforced interfaces between applications let components be written in different languages, developed independently, and distributed across machines for load balancing.

Two components carry the work. RoSE, the Rover Sequence Editor, is the textual command editor, data-driven from a command dictionary held as a configuration file so the editing interfaces are constructed at startup rather than compiled in; it assigns command identifiers, launches the other applications, and reads and writes sequences in an XML-based rover markup format [1]. HyperDrive is the three-dimensional editor: it displays the terrain, moves the rover through it, and is specialized for driving, arm motion and imaging commands, simulating the result and storing it for playback.

The validation is what makes it more than a text editor. RSVP simulates a sequence using the same flight software that runs on the rover, giving collision checks, trajectory generation error detection, and accurate duration estimates [1]. Duration estimates from HyperDrive feed back into the sequence timeline through the message bus. Perseverance’s tool keeps the same principle of validating against a model of the vehicle, and moves it into a browser [2].

COCPIT, the collaborative planning tool

Perseverance’s planning tool is a web application: JavaScript over a node.js server, with Backbone.js for the model and view layer, D3 and SVG for the timeline, a Redis database holding plan transactions as they are made, plan state synchronized between clients over web sockets, and deployment into Docker containers on Amazon ECS through Terraform, behind a single sign-on layer, exporting into an S3 and Elasticsearch data lake shared with the rest of the ground data system [2], a move driven by the same distributed-team problem that MER solved by relocating its persistence layer [6].

It replaced MSLICE, the MSL planning interface, and inherited the collaborative timeline, constraint and planning-unit structures of Playbook, the NASA Ames tool it was built on [2]. It continuously verifies constraints, models power and data, predicts when data will downlink to a passing orbiter, assigns instrument targets, records science intent, and translates the plan into the sequences bundled for uplink.

One interface covers five planning phases: parcel development, where reusable activities and components are defined and validated; strategic planning for activities never executed on Mars before; campaign implementation, producing a look-ahead plan across 3 to 5 sols; tactical uplink for the next planning cycle; and tactical downlink, showing the currently executing sol against a moving Mars time indicator [2]. Being genuinely multi-user and remote-first mattered because the team was distributed across North America and Europe from the start of operations.

The tactical timeline is the binding constraint

A Perseverance tactical planning shift is baselined at 9.5 hours and can start anywhere in the window from 6 am to midnight Pacific time, sliding within that window so that planning follows fresh data despite the offset between the Mars sol and the Earth day [7]. Most of the science team are not full-time on operations, which on previous missions meant only a small subset of the team could participate at all; splitting decisions between a predictive supratactical process at fixed Earth times and the reactive tactical process is the response to that.

MSL’s productivity study is the quantitative account of where the sols go. It combined structured interviews with operations personnel and analysis of three campaigns: Pahrump Hills over 19 sols, Artist’s Drive over 24 sols and 567 m, and Marias Pass over 24 sols and 130 m [3]. The identified drivers, in order of impact:

1. Ground in the loop for target selection and drive planning, the largest single factor, because activities that generate the data a decision needs must happen before the decisional relay pass. Restricted sols, where that constraint bites hardest, made up 41 percent of the mission [3].
2. The capacity of the tactical timeline to fill a multi-sol plan. Total activity across a multi-sol plan is lower than across the same number of single-sol plans, because developing and validating the command products takes time the shift does not have.
3. Ground in the loop to respond to an activity outcome, for instance re-doing an observation whose lighting or targeting turned out wrong, which costs extra sols of driving and repetition.
4. Inaccurate prediction of available vehicle resources, which restricts what gets planned; conservative activity estimates leave allocated margin unused and the vehicle idle.
5. Communication between science and engineering and within the team.
6. Science team engagement, degraded by part-time participation.

The resource cost of multi-sol planning is measured, and it is the cost the tactical shift imposes rather than one the vehicle imposes [3][7]:

Plan length	Flight computer duration used	Energy used
2-sol plans against 1-sol	12 percent less	7 percent less
3-sol plans against 1-sol	15 percent less	11 percent less

Source: [3]. Reducing the overhead was estimated to free 72 hours of additional campaign activity over the 19 sols at Pahrump Hills, 62 hours at Artist’s Drive and 69 hours at Marias Pass.

The study’s own conclusion is that the fixes are not tooling fixes: let the engineering team work effectively without knowing the exact vehicle state, let the vehicle detect a failed activity and respond onboard instead of waiting for the ground, plan multiple sols with less overhead, model resources more accurately, and improve the communication path between science and engineering [3]. The highest-satisfaction campaign, Pahrump Hills, produced fewer productive sols than the others; the walkabout strategy of a coarse survey followed by detailed revisits took longer and gave the team time to digest the data. The arm statistics for the same period show what that buys in placements [4].

What the tools cost the arm

Robotic arm activities were planned on about half of MSL’s first 200 sols, with contact science in more than 40 of them [4]. The output over those 200 sols was 6 samples, 5 by scoop and 1 by drill, and 19 portions delivered: 7 to the sample analysis instrument, 4 to the mineralogy instrument and 8 to the observation tray [4]. Placement accuracy after teach point calibration put the drill within 1 to 1.5 cm of target, and the sol 180 mini-drill hole center within 2 to 3 mm of the target selected in the microscopic imager. The drill preload test on sols 170 and 171 held preload on an overnight target for 24 hours and 40 minutes [4]. Microscopic imaging used three standoff regimes: 20 to 25 cm for context, 5 to 10 cm for stereo, and about 1 cm for high resolution [4].

Parallelism, and the hour per shift it returns

Perseverance was required to drive 50 percent farther than Curiosity, 15 km in a Mars year, and to collect more than twice as many samples, which made overlapping activities a requirement rather than an optimization [5]. The flight software supports it: over 100 tasks share the 133 MHz RAD750 under the VxWorks priority preemptive scheduler with rate monotonic priorities, and up to 16 sequences can execute in parallel.

The measured return, tracked by a tool built to identify executed parallel activities automatically, was 329 hours 52 minutes 58 seconds of cumulative parallelism by sol 570, about 30 to 35 minutes per sol returned to science on the vehicle, and about 1 hour per shift of tactical operations time across uplink and downlink combined [5]. A one hour saving per shift compounds across thousands of planning cycles. Parallel activities in operational use include neutron spectrometer sensing while driving, instrument cooling overlapped with other work, and imaging during communication window preparation. Each was a separate certification against the flight rules the planning tool enforces [2].

Parallelism has to be certified rather than assumed. A surface electromagnetic interference test campaign produced a source and victim matrix of interactions, rated by operations priority and available test data, and a working group reviews new parallelism needs monthly, tests candidate scenarios on testbeds, and validates them in flight [5].

On sol 516, sample processing ran in parallel with a communication pass; the image compression background task handling a CacheCam image lost CPU to higher-priority communication tasks, the automatic camera power off triggered before the readout completed, and the image data was lost [5].

Where the tools moved to reduce latency

MER’s planning tool history is a sequence of attempts to serve operators who were no longer in the building. The Science Activity Planner was centralized on JPL workstations with file-based persistence over NFS, fast because it sat next to its storage and with no integrated search [6]. Desktop sharing over a VPN was tried and performed badly for cross-country and international users [6]. Maestro, on the Eclipse Rich Client Platform, moved to a client and server split, but its Hibernate persistence layer decomposed a plan into hundreds of interconnected rows, and at 25 ms of network latency a 300-row save took 8 seconds; a REST server reduced the round trips but added its own latency.

The resolution was to stop treating a plan as a relational structure. Plans are stored as blobs in S3, named by the hash of their contents, with a metadata row in SimpleDB holding that hash; the blob is written before the metadata transaction commits, so a crash between the two leaves an orphan rather than a corrupt database, and a daemon deletes orphans older than a day [6]. Plan searches then returned in tens of milliseconds including network latency, the largest plans retrieved in seconds, and save time grew roughly linearly with activity count with a much smaller constant than the relational implementation.

The infrastructure cost fell with it. The legacy configuration ran two REST servers at $1340 per month plus about $2500 per month of storage and backup for 2.5 TB, roughly $40,000 per year; the cloud configuration was $412.50 per month of storage with backup, $18 per month for the load balancer, $191 per month for one large machine running continuously and $340 per month for ten large machines running 21 hours a week [6]. The usage pattern is what made the old arrangement wasteful: heavy use for about 8 hours every two days, on hardware provisioned for the peak [6], against a shift that is now baselined at 9.5 hours [7].

Comparison

	RSVP, MER	COCPIT, Mars 2020
Form	desktop applications over a message bus [1]	web application, node.js and Redis, on AWS [2]
Users	rover planners at JPL [1]	distributed international team, remote-first [2]
Validation	simulation using the flight software itself, collision and trajectory checks [1]	continuous constraint verification, power and data modeling [2]
Sequence output	rover markup language files [1]	plans translated into bundled uplink sequences [2]
Planning phases covered	tactical sequencing [1]	parcel development, strategic, campaign implementation, tactical uplink, tactical downlink [2]