Time Delayed Teleoperation

A bilateral teleoperation loop closes through the communication link, so the link delay enters the control loop and can destabilize it. The engineering problem is to keep a human-in-the-loop system stable and useful when the loop delay is a substantial fraction of, or larger than, the time constant of the contact dynamics being controlled [1][4]. The problem predates space robotics: Sheridan and Verplank’s 1978 review of undersea teleoperator control already treats supervisory control, where the operator specifies goals and the machine closes the fast loop itself, as the response to a transmission delay too long for direct force reflection [7]. Space links inherit the same tradeoff at delays that range from tens of milliseconds on a direct line of sight to tens of minutes to a planetary surface.

Families of approach and what has flown

Five families cover the work in this area. Direct bilateral coupling with passivity-based stabilization has flown from the ISS and on ETS-VII (sections below). Predictive display and task-level programming on a simulated workspace flew on ROTEX and GETEX [8][2][17]. Supervised autonomy, in which the operator issues goals and the remote side closes the fast loop, was formulated in the 1960s [12], developed for manipulators in the 1990s [16], and flew from the ISS as METERON SUPVIS Justin [21][22]. Low-latency telepresence from orbit around the target body is proposed for the Moon and flown only as an ISS-to-ground analog [24][25]. Ground-in-the-loop command cycles with onboard hazard handling are what the Lunokhod, Yutu and Mars rovers actually used operationally [15][29][27]. Learned delay compensation is simulation only [11].

Early delayed driving and supervisory control

Ferrell and Sheridan’s 1967 formulation has a local computer execute short-range goals while the operator intermittently sets subgoals and adjusts parameters, with a local model supplying quasi-feedback under transmission delay [12]. Its completion-time-versus-delay relationship is described qualitatively, without task statistics [12]. An MIT simulation the same year put a human operator in rate control of a manipulator under long delay and compared it with supervisory control [13]. The 1978 undersea review is the later statement of the same trade [7].

Field tests followed. MSFC drove a teleoperated vehicle from a mountaintop station 6 km away: a 3.7 km seven-target sortie took 5 hours at 0.68 km/h average, about 15 percent of the time spent driving and the rest studying the monitor and correcting headings, with 10 to 40 m dead reckoning error [14]. At 4, 6 and 8 s of artificial delay, stop-and-go driving beat self-paced driving, with self-pacing producing about 25 percent more course departures at 8 s [14]. Lunokhod-1 flew the same pattern: five go commands and one stop, narrow-band television at one frame per 3 to 20 s, and a five-person ground crew [15].

The delay each system actually had

Experiment	Year	Link	Measured round trip delay	Other measured link properties
ETS-VII ground to space	1999	NASDA ground station to satellite	6 to 7 s	commands issued every 250 ms [1]
Avatar-EXPLORE	2009	ISS to CSA rover	not reported as a delay figure; command file based	3 hours of session time, six command sequences [5]
Surface Telerobotics, K10	2012 to 2014	ISS to Ames Roverscape	up to 750 ms	three sessions of 3.5 h each [5]
METERON Haptics	2014 onward	ISS to ground robot	about 800 ms	single degree of freedom joystick [5]
KONTUR-2, ISS S-band	2015	direct line of sight S-band	20 to 30 ms	negligible mean data loss; experiment window limited to 8 minutes by line of sight [4][5]
KONTUR-2, training link	2015	internet, Star City to Oberpfaffenhofen	65 ms mean	5 to 15 percent packet loss over UDP [4]
KONTUR-2, RTC sessions	2015	ISS S-band plus internet	about 85 ms	256 kbit/s upload for video [4]
METERON SUPVIS Justin	2017 to 2018	ISS to Col-CC to JSC to TDRSS Ku-band	latency not stated as a figure; treated as low	supervised autonomy chosen to be insensitive to it [5]

A round trip delay of 20 to 30 ms is not the planetary case [5]. It is the number that direct line of sight S-band buys, and the price is an 8 minute experiment window per pass [5]. Round trip light time to a planetary surface is at least 6 minutes and up to 42 minutes in non-optimal geometry, which is why the recurring architecture is a human in orbit around the body commanding a robot on its surface rather than a human on Earth.

Direct bilateral coupling, ETS-VII

The ETS-VII experiment of 22 November 1999 closed a direct bilateral loop through a 6 to 7 s round trip [1]. Commands from the master arm passed through a NASDA external gateway and the NASDA operations facility to the satellite every 250 ms. The master was a two degree of freedom Immersion Impulse Engine 2000 force feedback joystick, and the controller was a PD-type bilateral coupling rather than a scattering-based wave variable formulation.

The same arm was driven from Germany in April 1999 under the GETEX experiments by predictive graphics rather than direct coupling, so the two approaches were exercised on one flight article within a year of each other [2].

Tasks were slope tracing and peg in hole. Performance was compared between bilateral mode and a unilateral mode in which force telemetry was displayed visually on a screen. All tasks were completed under bilateral control with no visual information at all, and kinesthetic force feedback improved task performance even at that delay [1].

That result is the counter-evidence to the one second rule. The authors argue against drawing a hard line at 1 s of delay and instead treat delay as a continuum, citing the ROTEX experience where the loop delay varied between 5 and 7 s and predictive display by computer graphics was the accepted answer [1].

Predictive display, GETEX

Predictive graphics is the other flown answer to a multi-second loop, and it flew before GETEX did. DLR’s ROTEX experiment on the Spacelab D-2 mission in April 1993 introduced tele-sensor-programming: the operator specifies a task on a simulated, sensor-updated model of the workspace rather than issuing joint commands directly, so that large signal delay is absorbed by the simulation rather than fought in the control loop [8]. GETEX carried the same concept to a second flight article six years later. The DLR ground station for the NASDA-DLR GETEX experiments in April 1999 drove the ETS-VII arm through a task directed programming scheme in which the operator specifies what is to be done rather than issuing joint commands, and the operator interacts with a simulated world instead of the delayed one [2]. The position of the predicted state is drawn as a wire frame overlay on the live image. The requirement that makes it work is the ability to update the simulated world from the real video images, so the model does not drift away from the vehicle it is predicting. Model-based online collision detection supervises the commanded motion, and each handled object carries a relative approach definition so that positions are derived from the selected task rather than typed in [2].

Four-channel bilateral control with passivity, KONTUR-2

The DLR RJo joystick flew to the Russian segment of the ISS in August 2015 and operated to December 2016 [3]. It is two degrees of freedom, generates up to 15 N, and allows plus or minus 20 degrees of travel on both axes.

RJo parameter	Value
Local control sampling frequency	1000 Hz
Intrinsic lag, torque command to effect	1 ms
Maximum stable stiffness	1.57 Nm/rad
Maximum force at the handle	15 N
Travel	+/- 20 degrees on both axes
Transmission frequency over the S-band link	500 Hz

Sources: [3] for the joystick figures, [4] for the transmission frequency.

The 1000 Hz local loop is the standard figure for rendering contact with a hard surface, and both the joystick and the robot run at it [3][4]; the S-band link carries only 500 Hz, so the transport rate rather than the controller sets the achievable loop bandwidth [4]. Keeping the local loops faster than the transport still pays, because it permits higher virtual damping values and more accurate passivity observers [4].

The controller is a four-channel architecture: position and force from the joystick to the robot, computed and measured force back [4]. Both ends are impedance controlled, taking force commands and returning positions, with a proportional-integral controller at the robot and a local damper at the joystick, and scaling factors matching the two dynamics [3]. Stability under delay, jitter and packet loss is enforced by Time Domain Passivity Control over a Time Delay Power Network representation of the link [3][4]. The same controller parameters were used unchanged on both the 20 to 30 ms S-band link and the 65 ms lossy internet link, which is the property the passivity formulation is there to provide [4].

The ROKVISS arm on the ISS Russian Service Module, operating since February 2005, carried the telepresence and automatic-mode experiments that preceded KONTUR-2 [18]. The KONTUR-2 link was inherited from ROKVISS and is asymmetric, 256 kbit/s up and 4 Mbit/s down. Under standard CCSDS framing only 128 kbit/s of the uplink carried application data [20]. Trimming the protocol overhead raised it to 192 kbit/s and left 96 kbit/s for video [20]. Contact was about 10 minutes per 90 minute orbit [22].

Flight record

The joystick was installed in the Russian segment in August 2015 for operation to December 2016, and the first experiments teleoperating ground robots at DLR and RTC ran that same month [3].

Twenty-three space sessions were held between August and December 2015: 9 teleoperating the DLR ROKVISS arm, 4 the RTC Surikat manipulator, 5 the RTC Yula mobile robot, and 5 cooperative DLR and RTC sessions on the humanoid Rollin’ Justin, one arm commanded from the ISS and the other from RTC on the ground [4]. Tactile feedback and system stability were verified with control loop delays up to 85 ms.

The Surikat case is instructive because that robot has no torque sensors. Force feedback was generated from commanded against achieved position, so the handle resisted the operator moving faster than the robot had actually reached, which conveys the robot’s inertia and the delay without any force measurement, and measurably improved both speed and accuracy [4]. Yula needed position-speed control rather than position control, and its force feedback was computed from modeled distance to obstacles in a 2D representation, with the operator driving a labyrinth.

Supervised autonomy as the alternative

Where the delay cannot be made small, the flown alternative is to raise the command level rather than compensate the loop. METERON SUPVIS-E and SUPVIS-M had astronauts select predefined task-level commands and monitor telemetry delivered over a Delay Tolerant Network, which permits high bandwidth transfer under variable delay [5]. SUPVIS Justin extended this to a humanoid: the astronaut commands object-centred actions on a tablet and the robot plans and executes them, with the interface presenting only the commands relevant to the current world state.

Sessions ran on 25 August 2017 with ESA astronaut Paolo Nespoli and on 2 March 2018 with NASA astronaut Scott Tingle, with a third scheduled for August 2018 [5]. Both ran over the Ku-band path from the ISS through the Columbus Control Centre and NASA JSC to the Tracking and Data Relay Satellite System. The first session established that astronauts could command the robot in scenarios not trained before flight, because the interface guides selection from the robot’s own model of the situation; the second required manual parameterization of commands, for example specifying a target position, and still succeeded on maintenance and adjustment tasks. The design claim being tested is that commanding becomes a low workload side task that an astronaut can return to between other work, and the astronauts reported quick switching to and from the interface.

The ground-side basis for this was set out for time-delayed manipulators in a 1993 JPL chapter: the local site builds parameterized commands that are simulated and modified before uplink, and the remote site executes them closed loop with its own monitoring, so round trip delay never sits inside a control loop; the demonstration was a seven-command door sequence on a laboratory manipulator in 1 g, with no flight or space-environment evidence [16]. JPL’s SURROGATE later applied the same intent-level approach to a two-armed torso robot for button, valve and bimanual tasks, as demonstrations without trial statistics [32]. In the satellite servicing setting, a plan-then-execute system for multi-second telemetry delay was compared with a VR interface in a nine-operator study: all operators succeeded with both, and workload fell from 3.21 to 2.39 on the VR interface (p=0.070), short of conventional significance [31].

The METERON concept was announced as a tablet interface needing 2 Mbit/s at 2 to 10 s round trip [23]. Before flight, over 150 ground runs were completed, some from a Columbus mockup 35 km from the Solex site with 800 ms injected delay, against a flight allocation of 1 Mbit/s with 800 to 850 ms round trip via TDRS [21]. All three SUPVIS Justin sessions completed all six task protocols with five crew members, who rated the system easy to use and low workload [22]. The ESA program carried the underlying store-and-forward transport on a Delay Tolerant Network; the DINET flight validation ran Bundle Protocol on Deep Impact at 24 million km for 27 days, and two of its four metrics missed their pre-declared criteria as first measured [19].

Earlier ISS-to-ground experiments set the baseline this was measured against: Avatar-EXPLORE in ISS Expedition 20/21 had one astronaut complete six command sequences to a CSA rover in three hours of session time under a non-interactive command-file process, and Surface Telerobotics ran three 3.5 hour sessions with different astronauts deploying a simulated radio telescope with K10 at up to 750 ms latency [5].

The measured trade is workload against capability. Task-level command lowers cognitive and physical workload and makes better use of a limited link, but it can only do what the robot’s action templates cover; direct teleoperation raises workload but lets the operator handle situations the autonomy cannot [5].

Telepresence from orbit around the target body

The case for crew in lunar orbit teleoperating surface robots, between boots on the ground and Earth-based teleprogramming, is argued in concept form with no measured latency, cost or science-return figures for the approach itself [25]. The nearest flown analog is the 2013 Surface Telerobotics experiment: three ISS astronauts drove the K10 rover over 11 hours in three sessions, with rover utilization of 65 to 80 percent and Bedford workload ratings of 1 to 3 [24]. With three subjects, the authors state no statistical result can be drawn [24]. In KONTUR-2 and SUPVIS Justin comparison, practical operator endurance before performance degraded was 20 to 30 minutes [22].

Earth-based teleoperation of an Artemis Lunar Terrain Vehicle was studied in two simulator experiments at zero, 4, 6 and 8 s round trip delay [26]. Average speed including stops fell from 3.24 kph at 0 s to 2.56, 2.03 and 1.76 kph [26]. When stops are excluded the delay penalty largely vanishes, so the cost is in the stop-and-wait cycle rather than in driving. The studies are simulations with no vehicle or regolith, so the speeds depend on the modeled vehicle and terrain as much as on the operators.

Operational practice on flown rovers

Flown planetary rovers resolved delay by command cycles rather than by closed loops. Yutu-2 on the lunar farside was run by the Beijing Aerospace Control Center team using onboard dead reckoning and ground visual positioning that agreed at the centimeter to decimeter level between waypoints up to 10 m apart, and diverged by more than 1 m under heavy slip, where the visual solution corrected the onboard one [29]. For Chang’e-3 Yutu, cross-site visual localization held error to 1 to 4 percent of traverse, 3.3 percent mean, against 15 percent for dead reckoning alone [30]. On Mars, a study of three MSL campaigns found only 25 to 42 percent of sols made significant direct progress on campaign objectives, with multi-sol plan constraints and deferrals consuming the rest [27]. Relay capacity sets the cadence as well: LDPC coding on the Mars Relay Network returned 70 percent more data than convolutional coding predicts in four MAVEN demonstration overflights, and one Perseverance overflight returned 2340.2 Mbits [28].

Failure modes

Delay itself is not the failure. A closed loop containing non-negligible delay destabilises when the coupling is tight, meaning when high frequency control action is required to capture the dynamics of the controlled system, which is exactly the hard contact case at 1000 Hz [4]. Scattering-theory formulations guarantee stability under delay of any magnitude, but stability and maneuverability are separate properties and the second is not thereby secured [1].

Jitter and packet loss are separately damaging and behave differently on different links. The ISS S-band link had negligible mean loss but is subject to shadowing, which produces signal attenuation, higher loss ratios or outright blackouts; the internet training link had a stable mean delay but 5 to 15 percent loss oscillating over the session under UDP [4].

Bandwidth constrains the feedback channel that is not the control loop. On KONTUR-2, video uplink was limited to 256 kbit/s, and the compression needed to fit it added its own appreciable delay at both ends, so a 3D model of the robot animated from prioritized telemetry was used alongside video because its compactness allowed a much higher refresh rate [4].

Line of sight limits the session, not the task. The 8 minute KONTUR-2 window is a direct consequence of choosing the sub-30 ms S-band path [5].

Analog-1

METERON Analog-1 is the current end point of the force-feedback line: an operator on the ISS teleoperating the Interact rover, a KUKA LWR arm on a mobile base, to select and collect geological samples, with full haptic feedback and a delay-compensation controller [6]. The ISS session was flown in November 2019 by Luca Parmitano, with a follow-on campaign on Etna in 2022. It sits inside a series of 13 ESA telerobotics experiments beginning in 2012 [6], of which the 2015 Interact experiment had already demonstrated a sub-millimeter peg-in-hole task teleoperated from the ISS with force feedback [5].

A 2016 survey of the discipline groups these approaches together as bilateral control with time-delay compensation, predictive display, and supervised autonomy, and treats them as the standard repertoire rather than as competing candidates: a program picks among them by the delay and bandwidth it has, not by a judgment that one is generally superior [10]. A separate review of space manipulators for debris removal reaches the same three-way grouping from the servicing side, flagging the time-delay problem as a precondition that any on-orbit manipulation architecture has to clear before dynamics and control questions become tractable [9].

What is not established

None of the flown systems compensate delay by learning a model of it online. A 2024 simulation study trains a deep reinforcement learning controller, using state augmentation to give the policy access to delayed state history, to plan trajectories for a free-floating space manipulator under constant and randomly varying round trip delay, and reports better accuracy than mapping or prediction baselines in simulation [11]. Whether a learned delay-compensation policy can be certified for a flight controller, and how it would be validated against the passivity guarantees that Time Domain Passivity Control provides analytically, is not addressed by that work or by any flown mission in this set. Telepresence from lunar orbit is not flown; the only evidence is ISS-to-ground analog work [24][25]. The record above is also silent on what happens when link characteristics degrade during a session rather than being fixed in advance: KONTUR-2 characterized loss and jitter on each link before flying it, and no experiment here reports a controller adapting online to a link whose statistics are changing.