Purdue University Graduate School
Natalia_Thesis_Document_NS6_accepted_Final.pdf (2.91 MB)

Learning Multi-step Dual-arm Tasks From Demonstrations

Download (2.91 MB)
posted on 2020-07-29, 13:36 authored by Natalia S Sanchez TamayoNatalia S Sanchez Tamayo
Surgeon expertise can be difficult to capture through direct robot programming. Deep imitation learning (DIL) is a popular method for teaching robots to autonomously execute tasks through learning from demonstrations. DIL approaches have been previously applied to surgical automation. However, previous approaches do not consider the full range of robot dexterous motion required in general surgical task, by leaving out tooltip rotation changes or modeling one robotic arm only. Hence, they are not directly applicable for tasks that require rotation and dual-arm collaboration such as debridement. We propose to address this limitation by formulating a DIL approach for the execution of dual-arm surgical tasks including changes in tooltip orientation, position and gripper actions.

In this thesis, a framework for multi-step surgical task automation is designed and implemented by leveraging deep imitation learning. The framework optimizes Recurrent Neural Networks (RNNs) for the execution of the whole surgical tasks while considering tooltip translations, rotations as well as gripper actions. The network architecture proposed implicitly optimizes for the interaction between two robotic arms as opposed to modeling each arm independently. The networks were trained directly from the human demonstrations and do not require to create task specific hand-crafted models or to manually segment the demonstrations.

The proposed framework was implemented and evaluated in simulation for two relevant surgical tasks, the peg transfer task and the surgical debridement. The tasks were tested under random initial conditions to challenge the robustness of the networks to generalize to variable settings. The performance of the framework was assessed using task and subtask success as well as a set of quantitative metrics. Experimental evaluation showed favorable results for automating surgical tasks under variable conditions for the surgical debridement, which obtained a task success rate comparable to the human task success. For the peg transfer task, the framework displayed moderate overall task success. Quantitative metrics indicate that the robot generated trajectories possess similar or better motion economy that the human demonstrations.


A Fundamental Theory for Dexterous Surgical Skills Transfer to Medical Robots

Congressionally Directed Medical Research Programs

Find out more...

See-What-I-Do: Increasing Mentor and Trainee Sense of Co-Presence in Trauma Surgeries with the STAR Platform

Congressionally Directed Medical Research Programs

Find out more...


Degree Type

  • Master of Science in Industrial Engineering


  • Industrial Engineering

Campus location

  • West Lafayette

Advisor/Supervisor/Committee Chair

Juan P. Wachs

Additional Committee Member 2

Ramses Martinez

Additional Committee Member 3

Yexiang Xue