Transforming Teleoperations: A Synchronized Extended Reality-Enabled Platform for Remote Control and Monitoring of Industrial Robotic Systems

Teleoperations in industrial robotics applications reduce risk and enhance task performance. Extended Reality (XR) technologies, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), provide fully immersive interfaces for teleoperation. The XR interfaces enhance perception and spatial awareness, addressing the limitations of traditional 2D interfaces. Studies reveal that operators are 66% faster using XR interfaces compared to 2D interfaces. Despite their advantages, XR interfaces (environments) face challenges in achieving accurate environmental representation and minimizing latency (below 50 milliseconds (ms)). High latency (exceeding 100 ms) and poor fidelity (below 3K resolution) affect real-time control, reducing the effectiveness of human operators in the XR environment. The reported challenges of latency and image fidelity motivated the development of an XR-enabled teleoperation platform with three objectives: (1) select the optimal XR environment for teleoperation, (2) develop a synchronized XR-enabled platform for controlling a six-degree-of-freedom (6-DOF) robotic arm, and (3) design and integrate advanced control protocols to minimize latency (below 50 ms) and maintain synchronization. A comparative study of XR environments was conducted, and MR was selected as the optimal environment. The Niryo Ned2 robotic arm was selected as the physical system. A comprehensive methodology was used to develop and test the platform, integrating Unity, Robot Operating System (ROS), Mixed Reality Toolkit (MRTK), PyNiryo, and ROS-Transmission Control Protocol (ROS-TCP). Three controllers were designed and implemented: event-driven, open-loop adaptive smoothing, and closed-loop predictive control. The final platform enabled real-time feedback, achieving an average latency below 10 ms, meeting optimal thresholds for responsive, synchronized teleoperation.