Time Delayed Teleoperation

A bilateral teleoperation loop closes through the communication link, so the link delay enters the control loop and can destabilize it. The engineering problem is to keep a human-in-the-loop system stable and useful when the loop delay is a substantial fraction of, or larger than, the time constant of the contact dynamics being controlled [1][4].

The delay each system actually had

Experiment	Year	Link	Measured round trip delay	Other measured link properties
ETS-VII ground to space	1999	NASDA ground station to satellite	6 to 7 s	commands issued every 250 ms [1]
Avatar-EXPLORE	2009	ISS to CSA rover	not reported as a delay figure; command file based	3 hours of session time, six command sequences [5]
Surface Telerobotics, K10	2012 to 2014	ISS to Ames Roverscape	up to 750 ms	three sessions of 3.5 h each [5]
METERON Haptics	2014 onward	ISS to ground robot	about 800 ms	single degree of freedom joystick [5]
KONTUR-2, ISS S-band	2015	direct line of sight S-band	20 to 30 ms	negligible mean data loss; experiment window limited to 8 minutes by line of sight [4][5]
KONTUR-2, training link	2015	internet, Star City to Oberpfaffenhofen	65 ms mean	5 to 15 percent packet loss over UDP [4]
KONTUR-2, RTC sessions	2015	ISS S-band plus internet	about 85 ms	256 kbit/s upload for video [4]
METERON SUPVIS Justin	2017 to 2018	ISS to Col-CC to JSC to TDRSS Ku-band	latency not stated as a figure; treated as low	supervised autonomy chosen to be insensitive to it [5]

A round trip delay of 20 to 30 ms is not the planetary case [5]. It is the number that direct line of sight S-band buys, and the price is an 8 minute experiment window per pass [5]. Round trip light time to a planetary surface is at least 6 minutes and up to 42 minutes in non-optimal geometry, which is why the recurring architecture is a human in orbit around the body commanding a robot on its surface rather than a human on Earth.

Direct bilateral coupling, ETS-VII

The ETS-VII experiment of 22 November 1999 closed a direct bilateral loop through a 6 to 7 s round trip [1]. Commands from the master arm passed through a NASDA external gateway and the NASDA operations facility to the satellite every 250 ms. The master was a two degree of freedom Immersion Impulse Engine 2000 force feedback joystick, and the controller was a PD-type bilateral coupling rather than a scattering-based wave variable formulation.

The same arm was driven from Germany in April 1999 under the GETEX experiments by predictive graphics rather than direct coupling, so the two approaches were exercised on one flight article within a year of each other [2].

Tasks were slope tracing and peg in hole. Performance was compared between bilateral mode and a unilateral mode in which force telemetry was displayed visually on a screen. All tasks were completed under bilateral control with no visual information at all, and kinesthetic force feedback improved task performance even at that delay [1].

That result is the counter-evidence to the one second rule. The authors argue against drawing a hard line at 1 s of delay and instead treat delay as a continuum, citing the ROTEX experience where the loop delay varied between 5 and 7 s and predictive display by computer graphics was the accepted answer [1].

Predictive display, GETEX

Predictive graphics is the other flown answer to a multi-second loop. The DLR ground station for the NASDA-DLR GETEX experiments in April 1999 drove the ETS-VII arm through a task directed programming scheme in which the operator specifies what is to be done rather than issuing joint commands, and the operator interacts with a simulated world instead of the delayed one [2]. The position of the predicted state is drawn as a wire frame overlay on the live image. The requirement that makes it work is the ability to update the simulated world from the real video images, so the model does not drift away from the vehicle it is predicting. Model-based online collision detection supervises the commanded motion, and each handled object carries a relative approach definition so that positions are derived from the selected task rather than typed in [2].

Four-channel bilateral control with passivity, KONTUR-2

The DLR RJo joystick flew to the Russian segment of the ISS in August 2015 and operated to December 2016 [3]. It is two degrees of freedom, generates up to 15 N, and allows plus or minus 20 degrees of travel on both axes.

RJo parameter	Value
Local control sampling frequency	1000 Hz
Intrinsic lag, torque command to effect	1 ms
Maximum stable stiffness	1.57 Nm/rad
Maximum force at the handle	15 N
Travel	+/- 20 degrees on both axes
Transmission frequency over the S-band link	500 Hz

Sources: [3] for the joystick figures, [4] for the transmission frequency.

The 1000 Hz local loop is the standard figure for rendering contact with a hard surface, and both the joystick and the robot run at it [3][4]; the S-band link carries only 500 Hz, so the transport rate rather than the controller sets the achievable loop bandwidth [4]. Keeping the local loops faster than the transport still pays, because it permits higher virtual damping values and more accurate passivity observers [4].

The controller is a four-channel architecture: position and force from the joystick to the robot, computed and measured force back [4]. Both ends are impedance controlled, taking force commands and returning positions, with a proportional-integral controller at the robot and a local damper at the joystick, and scaling factors matching the two dynamics [3]. Stability under delay, jitter and packet loss is enforced by Time Domain Passivity Control over a Time Delay Power Network representation of the link [3][4]. The same controller parameters were used unchanged on both the 20 to 30 ms S-band link and the 65 ms lossy internet link, which is the property the passivity formulation is there to provide [4].

Flight record

The joystick was installed in the Russian segment in August 2015 for operation to December 2016, and the first experiments teleoperating ground robots at DLR and RTC ran that same month [3].

Twenty-three space sessions were held between August and December 2015: 9 teleoperating the DLR ROKVISS arm, 4 the RTC Surikat manipulator, 5 the RTC Yula mobile robot, and 5 cooperative DLR and RTC sessions on the humanoid Rollin’ Justin, one arm commanded from the ISS and the other from RTC on the ground [4]. Tactile feedback and system stability were verified with control loop delays up to 85 ms.

The Surikat case is instructive because that robot has no torque sensors. Force feedback was generated from commanded against achieved position, so the handle resisted the operator moving faster than the robot had actually reached, which conveys the robot’s inertia and the delay without any force measurement, and measurably improved both speed and accuracy [4]. Yula needed position-speed control rather than position control, and its force feedback was computed from modeled distance to obstacles in a 2D representation, with the operator driving a labyrinth.

Supervised autonomy as the alternative

Where the delay cannot be made small, the flown alternative is to raise the command level rather than compensate the loop. METERON SUPVIS-E and SUPVIS-M had astronauts select predefined task-level commands and monitor telemetry delivered over a Delay Tolerant Network, which permits high bandwidth transfer under variable delay [5]. SUPVIS Justin extended this to a humanoid: the astronaut commands object-centred actions on a tablet and the robot plans and executes them, with the interface presenting only the commands relevant to the current world state.

Sessions ran on 25 August 2017 with ESA astronaut Paolo Nespoli and on 2 March 2018 with NASA astronaut Scott Tingle, with a third scheduled for August 2018 [5]. Both ran over the Ku-band path from the ISS through the Columbus Control Centre and NASA JSC to the Tracking and Data Relay Satellite System. The first session established that astronauts could command the robot in scenarios not trained before flight, because the interface guides selection from the robot’s own model of the situation; the second required manual parameterization of commands, for example specifying a target position, and still succeeded on maintenance and adjustment tasks. The design claim being tested is that commanding becomes a low workload side task that an astronaut can return to between other work, and the astronauts reported quick switching to and from the interface.

Earlier ISS-to-ground experiments set the baseline this was measured against: Avatar-EXPLORE in ISS Expedition 20/21 had one astronaut complete six command sequences to a CSA rover in three hours of session time under a non-interactive command-file process, and Surface Telerobotics ran three 3.5 hour sessions with different astronauts deploying a simulated radio telescope with K10 at up to 750 ms latency [5].

The measured trade is workload against capability. Task-level command lowers cognitive and physical workload and makes better use of a limited link, but it can only do what the robot’s action templates cover; direct teleoperation raises workload but lets the operator handle situations the autonomy cannot [5].

Failure modes

Delay itself is not the failure. A closed loop containing non-negligible delay destabilises when the coupling is tight, meaning when high frequency control action is required to capture the dynamics of the controlled system, which is exactly the hard contact case at 1000 Hz [4]. Scattering-theory formulations guarantee stability under delay of any magnitude, but stability and maneuverability are separate properties and the second is not thereby secured [1].

Jitter and packet loss are separately damaging and behave differently on different links. The ISS S-band link had negligible mean loss but is subject to shadowing, which produces signal attenuation, higher loss ratios or outright blackouts; the internet training link had a stable mean delay but 5 to 15 percent loss oscillating over the session under UDP [4].

Bandwidth constrains the feedback channel that is not the control loop. On KONTUR-2, video uplink was limited to 256 kbit/s, and the compression needed to fit it added its own appreciable delay at both ends, so a 3D model of the robot animated from prioritized telemetry was used alongside video because its compactness allowed a much higher refresh rate [4].

Line of sight limits the session, not the task. The 8 minute KONTUR-2 window is a direct consequence of choosing the sub-30 ms S-band path [5].

Analog-1

METERON Analog-1 is the current end point of the force-feedback line: an operator on the ISS teleoperating the Interact rover, a KUKA LWR arm on a mobile base, to select and collect geological samples, with full haptic feedback and a delay-compensation controller [6]. The ISS session was flown in November 2019 by Luca Parmitano, with a follow-on campaign on Etna in 2022. It sits inside a series of 13 ESA telerobotics experiments beginning in 2012 [6], of which the 2015 Interact experiment had already demonstrated a sub-millimeter peg-in-hole task teleoperated from the ISS with force feedback [5].