Manipulation Planning

Manipulation planning on a planetary vehicle produces a joint-space or Cartesian trajectory that reaches a target without touching anything else. The split between ground and flight is sharp: the trajectory is planned on Earth and the flight software’s job is to check and execute it. Nothing on Mars has yet planned its own arm motion from a science goal.

The two collision problems

An arm collides with two different things and the checks cost differently. Self-collision, arm against arm and arm against vehicle, is a fixed geometry problem answered from the kinematic model. Terrain collision is answered against a mesh built from stereo imagery taken from the vehicle that sol, and is far more expensive [1].

Measured inside the Perseverance ground planner:

Function	Average duration
Forward kinematics	0.7 ms
Self-collision check	1.3 ms
Terrain clearance computation (Proximeter)	11.8 ms

Source: [1], Table I. Terrain clearance costs nine times a self-collision check and 17 times a forward kinematics evaluation, which is why the planner’s structure is organized around avoiding it. Flight software on Perseverance does not run this query at all; the clearance is settled on the ground before the sequence is uplinked [2].

Flight software uses a coarser model. The InSight Instrument Deployment Arm flight software, inherited from the Phoenix robotic arm software which inherited it from the MER Instrument Deployment Device, composes each collision link from cylinders and boxes and checks each generated via point against a table of link pairs [3]. Checking can be enabled and disabled per pair, and static link positions are parameters the ground can update. On InSight that mechanism carried the science instruments: with SEIS, the wind and thermal shield, or HP3 on the deck, that instrument’s static link was checked and its dangling link was not; once grappled, the dangling link that hangs from the grapple attachment point in the gravity direction was checked and the static link was not; once set on the terrain, the static link’s position parameters were rewritten to the new Mars location and checking switched back.

Planning on the ground: RRT-Connect and viapoint reduction

Perseverance arm activities are planned in ArmSketch. Inputs are the rover’s initial joint angles and location, terrain meshes from rover stereo imagery, and a set of science activities; the tool computes motion plans between every pair of activities, simulates each in the SSim flight software simulator to get its duration, and searches combinatorially for the activity ordering with the shortest total execution time. Rover planners are expected to produce a rough plan in about 10 minutes [1], inside the same tactical timeline that constrains the sampling assessments Proximeter automates [2].

Motion planning uses RRT-Connect [1]. The operational constraint is unusual: each viapoint in the path becomes a separate arm motion command that a rover planner must review before uplink, so the number of viapoints, not the path length, is the quantity to minimize, and in the scenarios studied a single viapoint suffices to clear any obstacle. Standard RRT-Connect does not minimize viapoints, and its raw output depends strongly on the step-size parameter:

RRT-Connect step size	Mean viapoints	Mean path length
0.25	15.73	4.05
0.5	7.83	4.16
1.0	3.86	4.36
2.0	2.16	5.32
5.0	1.37	8.56

Source: [1], Table III, averaged over 50 random seeds.

A rapidly-exploring random tree is probabilistically complete and not optimal, and the rate of that convergence is stated by its author to be an open problem, so an RRT planner carries no bound on planning time and no bound on solution quality [10]. Its path length in bounded convex two-dimensional free space with a holonomic model and a Euclidean metric runs 1.3 to 2.0 times the optimal path to the tree root, the most favorable case measured, with no figure given for cluttered or nonholonomic problems. The metric that would be ideal for a kinodynamic problem is the cost of the optimal trajectory between two states, which is as expensive as the planning problem, so every implementation substitutes a metric whose effect on convergence is unquantified [10]. Planning on the ground absorbs that: the 10 minute budget is a rover planner’s, not a flight timeline’s.

Path simplification recovers both. At step size 1.0, binary shortcutting takes the path from a mean of 3.86 viapoints and length 4.36 to 1.08 viapoints and length 3.71, at a cost of 234 collision checks [1]. Greedy graph shortcutting at step size 1.0 gives 1.40 viapoints and 197 collision checks. The evaluation weighted viapoints at 10/15, collision checks at 3/15 and path length at 2/15 [1]. The weighting follows from the review cost: each viapoint is a command a rover planner must check [1][5].

The collision checking itself was restructured around a safety bubble. The baseline splits an edge into uniform steps, orders them by binary subdivision so a collision is found early, and checks self-collision and terrain clearance at each; if the edge is clear it has still paid for every step [1]. The adaptive version calls the terrain clearance query once, takes the returned clearance distance as the radius of a bubble in Cartesian space around each joint, and then uses forward kinematics alone, at 0.7 ms, to confirm that neighboring steps have not moved any joint outside its bubble, calling the expensive query again only when one has. Over a benchmark of 6 scenarios, 334 RRT plans and 802 edges, 528 of them collision-free, the adaptive algorithm is 3.3 times faster on average [1]. On unsafe edges it is slightly slower, because it makes more forward kinematics calls, and both algorithms agreed on every edge’s safety across the benchmark plus 23 added edge cases including long motions needing two or more viapoints, self collisions, paths hugging the terrain, wide and narrow obstacles, and turret rotation near the ground.

Proximeter: what the clearance query does

Proximeter computes the worst-case minimum distance between triangular meshes of the robot hardware and of the local terrain, over the volume of the placement’s position and orientation uncertainty, which is resolved anisotropically into normal, lateral, clocking and angular components because normal uncertainty is generally the smaller [2]. Distance checks use the Proximity Query Package, and the search over the uncertainty volume is a grid search accelerated by Lipschitz optimization, which uses the bound on the function’s rate of change to skip evaluations that cannot beat the current minimum. Queries complete in under 50 ms [2].

Before Proximeter, operators selected points by hand on the rover collision model and on the terrain, measured the distance between them, and repeated for further point pairs, a process that was slow, subjective and conservative enough that valid arm placements were rejected [2]. TargetWizard, which drives Proximeter through a gRPC server using protocol buffers, performs sampling and caching evaluations that take human operators 30 to 40 minutes in 15 seconds, removing the need to pause operations for a sol to assess a complex science activity. Applied by command line to Curiosity’s Feed-Extended Sample Dropoff procedure, developed after the loss of the drill feed actuator and requiring collision checks to be disabled so the flight software cannot assess it, Proximeter reproduced the manual assessment and saved 10 to 20 minutes of operator time [2].

Terrain meshes have defects from poor stereo range data or lighting, so Proximeter can miss a surface feature absent from the mesh or alarm on one that does not exist, and in those cases operators are responsible for detecting the defect because the clearance cannot be trusted [2]. Mesh hole and surface roughness checks were being added before any use in a fully autonomous mode. Running Proximeter onboard would remove the ground loop from contact science target evaluation entirely [2][5].

Force as the terminal condition

Placement and preload are not position problems. Perseverance’s arm is about 2 m long with 5 degrees of freedom, carries a 45 kg turret, applies several hundred newtons of preload to surface targets and over 700 N to its dock, and senses through a 6 degree of freedom force torque sensor between the arm and the turret with redundant A and B strain gauge sets of which the software can use only one at a time [4].

Preload uses Iterative Force Control rather than the continuous force control common in terrestrial robotics. The arm holds still, reads the force sensor, computes the error to the preload goal, uses a compliance model of the arm at the current pose to compute the small Cartesian motion that would produce the missing force, converts that to joint commands, executes, and reads again [4]. It terminates on the goal within a parameterized deadband, or on a parameterized iteration limit, in which case failure is declared. The compliance matrices are computed analytically per link and tool and tuned by kinematic calibration with laser metrology.

Docking, the one autonomous arm operation

The corer must mate with the bit carousel to exchange drill bits and sample tubes. Four posts on the corer face enter four concave cones on a rotating ring; each cone has a contact switch triggering within 0.5 mm of its bottom, and the ring’s hardstops sit at +3.13 and -10.08 degrees from center [4]. ARM_DOCK runs the whole sequence autonomously from free space with no operator monitoring: move to standoff, move to a teach point on a slower straight-line Cartesian path, move to contact until the force sensor exceeds its noise threshold, apply an open-loop initial preload, then lateral alignment driving the forward preload to its goal while minimizing lateral loads until at least one contact switch triggers, then angular alignment raising the preload while also nulling moments until at least three switches trigger, then a final preload for margin during bit exchange. It is the only arm operation on Perseverance that closes a loop onboard without a ground-planned trajectory [1][4].

Fault protection runs against the same signals: motor monitoring at 64 Hz for current, stalls, shorts, temperature, position discrepancy and joint limits; a fault if any contact switch is pressed before contact is expected; the measured preload constrained to the target plus a margin, for example a 700 N limit against a 600 N target; lateral forces and moments held inside deadbands; dock rotation checked against softstops more conservative than the hardstops; and a per-phase iteration limit [4].

Ground testing was extensive because the operation is irreversible. The Controls and Autonomy for Sample Acquisition and Handling testbed, whose software mimicked the flight architecture with a faster development cycle, performed more than 1200 dockings, testing cone and post materials and surface finishes and characterizing force sensor inaccuracy and arm positioning uncertainty by design of experiments [4]. Qualification model dirty testing added more than 400 dockings and reached three times life, with per-docking maximum forces flat over time. Assembly testing found under 1 mm of clearance between a drill stabilizer and the bit carousel housing, mitigated by removing material and adding a flight rule, and thermal testing found a limit-cycling condition fixed by an algorithm change [4], the class of defect a ground mesh check cannot reach [2]. The experimental preload-to-dock-rotation slope, -0.138 mrad/N, matched the analytical -0.131 mrad/N [4]. Perseverance docked 16 times to the adaptive caching assembly in the first 210 sols [5].

Docking was first performed on Mars on sol 87 and repeated successfully eight more times through sol 211, on sols 150, 164, 189, 195, 196 and 211 [4]. Docking itself takes about 2 minutes of a roughly 20 minute deposit-bit sequence or 25 minute retrieve-bit sequence. Sampling sequences including docking are simulated before uplink in SSim, which runs the ARM, DRILL, MOT and ACA flight software modules and models the dock as a rigid flat plane whose reaction forces are written into simulated strain gauge channels, a model that misses the post and cone contact physics but catches sequencing errors. The same simulator supplies the activity durations ArmSketch orders its plans by [1].

What the ground actually spends

Over the first 210 sols Perseverance issued 20,764 arm and sampling commands [5]. The arm was unstowed 27 times, the contact sensor touched the surface 17 times, the drill was placed on the surface 9 times, and the instruments were placed 152 times for WATSON, 10 for SHERLOC and 8 for PIXL, the ordering following how tight each instrument’s standoff requirement is. A typical sampling operation including docking, bit exchange, acquisition and processing uses over a hundred command products; the sol 196 core used 27 scripts, 73 strategic sequences and 11 tactical sequences [5].

The Three Forks sample depot is the largest single manipulation campaign flown. Ten returnable sample tube assemblies were dropped between sols 652 and 693, 42 sols, over 207.93 m of driving, documented by about 4000 images [6]. The drop is from roughly 90 cm above the ground and the requirement is that each tube land within a 70 or 95 cm radius drop zone, with tubes at least 5.5 m apart so each has a hazard-free circle for the retrieval helicopter. Achieved radial offsets were 38.4, 29.1, 48.8, 40.0, 9.6, 24.2, 14.7, 48.6, 19.7 and 12.3 cm [6]. Every tube landed horizontal, so the tube topple contingency, which would have approached at 25 degrees above the ground plane in 1 cm increments over 20 cm with a 30 cm alternating lateral sweep of the stabilizer shaft, was never used [6].

Two faults interrupted it. On sol 672 a transient spike on the shoulder azimuth resolver at motion start pushed the encoder-to-resolver mismatch past its persistence limit; recovery took sols 673 and 674, closing the WATSON cover and wiggling the joint with a wider persistence setting [6]. On sol 687 the sample handling arm’s internal force exceeded its 23 N positive limit in the home position, traced to a sequencing error in which forces were not tared across the nine previous drops, so false internal force accumulated; the drop succeeded on sol 690 after the sequencing was corrected. The ground test did not find either, because the testbed cleared the persistent volatile state on every power cycle. Post-drop imaging before driving away was made mandatory, adding a sol to each drop, because a tube near a wheel is unrecoverable.

Autonomy that was demonstrated and not flown

Single cycle instrument placement was demonstrated in the field. MER practice required three to four sols per target: approach, refine the rover position, then deploy the arm [7][8]. The demonstrated alternative takes a target designated once in initial imagery, then autonomously drives to it, tracks it through the traverse, hands the track off from the mast cameras to the body cameras, refines the base placement, and deploys the arm, all from one command.

Tracking is the hard part. The mast is pointed by an exact inverse kinematics solution to the target’s 3D position, and the target is matched by a combination of global normalized cross-correlation and a local affine transform over multi-size templates at several pyramid levels, holding under a pixel between frames and a few pixels over a 10 m traverse [8]. The camera handoff, projecting the target from mast stereo into the body camera view across a large viewpoint change, is the largest single error contributor. Two dimensional tracking was validated to 4 cm at 3 sigma, and image-based hazard-camera instrument placement reduces the final error from several centimeters to under 1 cm [8].

Over 43 runs on the Athena rover in the JPL Mars Yard, with targets 5 to 15 m away, targets from 7 cm to 0.5 m in radius, 28 small-target runs and 15 runs at 10 m each involving at least one major obstacle, 93 percent of placements landed within 2 to 3 cm of the designated target [7]. Terrain was mapped by quarter-resolution dense stereo into a 20 cm grid, with 20 cm the traversable step height, 80 percent of wheel diameter. Three consecutive tracking failures aborted the run.

Real-time onboard collision avoidance was also built and not flown to a planetary surface. A perturbation approach for the two 7 degree of freedom arms of the Ranger telerobotic flight experiment adds virtual repulsive forces when any arm segment enters an obstacle’s safety zone, perturbing the Cartesian command into the existing position loop rather than replacing the controller [9]. On a MIPS R4600 at 100 MHz, line-segment to face obstacle detection took about 1.23 ms, the perturbation computation under 0.2 ms, and the full cycle including forward and inverse kinematics about 2.5 ms, against a 400 Hz servo interface to the arm control unit. Maximum measured intrusion into the safety zone was about 0.28 cm at the arm angle and under 0.2 cm at the end effector, against 3.15 cm with avoidance disabled, at stand-off distances of 10 to 20 cm [9].

Families of approach

The ground-planned RRT-Connect pipeline above is one of five families the literature covers. They differ in what they measure, what they cost, and how far each has traveled.

Resolved-rate and null-space methods plan nothing global. Resolved motion rate control computes joint rates from a commanded end-effector velocity through the Jacobian [12]. For a kinematically redundant arm, the pseudoinverse solution can carry a homogeneous term that drives the point nearest an obstacle directly away from it, which keeps the commanded end-effector trajectory exact [13]. Configuration control instead augments the forward kinematics with user-defined functions to form a full-rank set of task variables, and tracks them with an adaptive PD controller without a pseudoinverse or a dynamic model [14]. The null-space paper is a graphics simulation with the obstacle geometry read from an exact world model [13], and the configuration-control demonstration used a PUMA 560 with 3 of 6 joints active [14]. Both are laboratory results.

Reactive and potential-field methods put collision avoidance inside the servo loop. Khatib reformulated avoidance as a repulsive field evaluated at the servo rate, paced by the controller rather than a planner that runs orders of magnitude slower, and states that local minima can trap the robot in a cluttered field [11]. The Ranger perturbation scheme [9] and a constraint-based damped-least-squares scheme tested on a 7 degree of freedom arm reaching into a truss mock-up at JPL [16] are descendants. A behavior-based architecture on a redundant JPL arm merged trajectory tracking, force tracking, singularity, joint-limit and obstacle avoidance in one impedance-control motion space, shown in a single contact run [15]. None of these has operated on a planetary surface.

Model and sensor collision checking is what has flown. The FIDO collision detector composed rover and arm geometry from oriented bounding boxes and a bounding prism for cylindrical links, checked it against an elevation map from the hazard cameras, and was ported into the MER flight software; its published timings are from a desktop processor and the MER figures are an extrapolation [17]. The coarse link-pair table on InSight [3] is the same lineage.

Sampling-based and optimizing planners are used on the ground. RRT [10] and RRT-Connect [1] are probabilistically complete and not optimal. PRM* and RRT* add asymptotic optimality at the same O(n log n) complexity as the classic algorithms, shown on abstract Euclidean spaces [29]. CHOMP refines a straight-line or RRT initial path into a smooth collision-free trajectory by gradient descent on a signed-distance cost [28]. Neither [28] nor [29] is a flight or planetary-terrain result. For free-floating space manipulators, base motion couples to the joints, so the dynamic equations differ in structure from the fixed-base arm [31]. Learned planning for dual-arm free-floating manipulators uses DDPG [32] and a task-space constrained bidirectional RRT for tightly coupled arms [33]; both are kinematic or dynamic simulations without hardware.

Learning-based and trajectory-optimization planners for orbital arms are the most active recent family, and all of it is simulation or ground-rig work. A constrained reinforcement learning planner with a PointNet pose estimator and Extended Kalman Filter tracks a spinning non-cooperative target with a free-float dual-arm manipulator in simulation [38]. A soft actor-critic planner handles teleoperated manipulators under communication delay, comparing mapping, prediction and state augmentation across four MuJoCo environments with fixed and free-floating bases [40]. An adaptive compliance method outputs the dual-arm object trajectory and the impedance parameters together, and was tested in simulation and on a ground two-arm UR rig [37]. A hybrid of trajectory optimization and a PPO tracking policy for a free-flying four-arm, 27 degree of freedom servicer runs only in a rigid-body simulation [42]. The terrestrial dual-arm research platform AILA, with 7 degree of freedom arms and 32 degrees of freedom overall, is the mobile manipulation reference these planners are developed against, and reports no planning success rates [36]. Reviews of the field are in [39], [41] and the Springer Handbook chapter [43].

Vision-closed placement and mobile manipulation cover the gap between a planned pose and where the tool ends up. Reachability analysis places a mobile base so a mounted arm reaches a point, line, surface or volume, demonstrated on one arm in one laboratory [35]. Rocky 7 carried a sampling arm and an instrument mast [18] and autonomously grasped rocks with stereo, odometry and tactile sensing, with 11 of 25 grasp trials fully successful [19]. End-effector position error compensation compares the kinematic prediction with a fiducial triangulated by fixed stereo cameras and adds the difference to the command, with three flight attempts, one of which failed, and no ground truth [22]. The DARPA ARM-S system at JPL touched the table and door handle deliberately to reduce state uncertainty before grasping, scored in five runs per task [30]. DLR’s Lightweight Rover Unit combined workspace analysis and onboard manipulation planning in the SpaceBotCamp 2015 field trial [34].

What the flown arms measured

Force sensing precedes any contact planning. A six-axis wrist force torque sensor beat four other sensing schemes on every axis on a 5 degree of freedom arm kinematically equivalent to the MER device, within 1 percent of full range [23].

Arm	Reported placement or operations result	Source
MER IDD	about 1 mm and 1 degree repeatability, stated without an error budget; requirement 10 mm	[20][21]
Phoenix	53 dig sequences, 10 trenching areas, 18 samples delivered in over 150 sols; no placement figures quantified	[24]
Phoenix	152 sols, 31 samples to TEGA, MECA and WCL; strength inferred from motor current	[25]
MSL	plus or minus 20 mm position, 10 mm repeatability requirements, verified on Earth and by model	[27]
MSL	drill placement within about 2 to 3 mm of target on one hole, sol 180	[26]

The MSL ground result is a single measurement on one hole [26], and the error budget numbers are Earth test and model prediction [27].

Comparison

	InSight IDA	Perseverance arm
Degrees of freedom	4	5
Collision primitives	cylinders and boxes	triangular meshes on the ground [1][2], flight model not published
Where the trajectory is planned	ground, expanded onboard to actuator commands	ground, in ArmSketch [1]
What flight software checks	every via point against the enabled link pairs [3]	via points, plus force, moment, contact switch and dock rotation limits during docking [4]
Force sensing	current limits and torque checks on guarded moves [3]	6 degree of freedom force torque sensor, redundant gauges [4]
Autonomous closed loop operations	none; grapple state machine only [3]	docking, from free space to preload, no operator monitoring [4]
Fault response	stop all motors and heaters, announce, mark safed, accept only recovery commands [3]	per-phase iteration and force limits, then fault [4]

What is not established

No flown arm has planned its own motion from a science goal; the flown record is ground planning plus onboard checking [1][3][4]. The only closed loop onboard is docking [4]. The RRT-Connect step-size and viapoint figures come from Perseverance scenarios on the ground [1], and the convergence rate of RRT-class planners remains an open problem [10]. The reactive, null-space and optimizing methods [11][13][16][28][29] are demonstrated in simulation or laboratory arms; the learned and dual-arm planners [32][33][38][40][42] have no hardware validation, and the compliance method [37] was tested only on a ground rig. The accuracy figures for the flown arms are operations summaries or model predictions, and the MER repeatability figure has no stated measurement campaign [20][27]. Proximeter and its onboard use are not yet qualified for autonomous operation because terrain mesh defects remain the operator’s responsibility [2]. Free-floating, orbital and bimanual planners are covered only to show the contrast with surface arms, and the on-orbit servicing literature is not surveyed here.