Augmented Reality for Haptic Display in
Robot-Assisted Surgical Systems
THE PROBLEM TESTING
• Tele-manipulated MIS robots do not include
• We will validate the system using ten subjects (surgeons
significant haptic (tactile or force) feedback
and non surgeons with varying levels of experienced on
• This lack of feedback is detrimental to
the da Vinci) performing an instrument tie with and without
performance of some surgical tasks
the augmented reality system
• We will compare performance with and without the
augmented reality system.
• All experiments performed at the Johns Hopkins Medical
• Comprehensive experiments with tasks that rely more
Da Vinci Surgical System View from surgeon’s console heavily on force
(Intuitive Surgical, Inc.) • Try different sensory substitution methods
• Experiments were conducted with visual and
• Showed that sensory substitution of haptic
information improves repeatability of applied
suture forces and minimizes suture breakage
• Develop an augmented reality system for visual
Design of overlay that changes colors as forces are being applied and tracks
sensory substitution of haptic information where instrument tips using kinematics tool tracker.
forces are displayed to surgeons in an intuitive PEOPLE INVOLVED
and accessible way.
• Graduate Students: Takintope Akinbiyi, Carol Reiley, Sunipa Saha
• Determine the effectiveness of forces displayed
• Research Faculty: Darius Burschka
visually to the user using a tele-manipulation
system. • Principle Investigator: Allison M. Okamura
1) Inexpensive strain gauges added to lower shaft of • Clinicians: David Yuh
micro-forceps tool of da Vinci Surgical System to
• Industrial Collaboration: Christopher Hasser, Intuitive Surgical Inc.
measure bending forces. Using specialized
cannulas to allow for strain gage wire.
2) Orientation and position of tool end effector and
endoscope are tracked using robot kinematics
and used in computer vision for tool tracking. REFERENCES
3) Superimpose a spherical overlay on each tool,
• T. Akinbiyi, A. M. Okamura, D. D. Yuh, “Dynamic
whose colors change in response to the forces
Augmented Reality for Haptic Display in Robot-Assisted
measured at tool tip. Overlays follow tool in real
Surgical Systems,” Medicine Meets Virtual Reality 13, J.D.
Westwood, et al., IOS Press, pg. 33, 2005.
tool shaft • A. M. Okamura, “Methods for Haptic Feedback in
Strain Special “clamshell”
cannulas Teleoperated Robot Assisted Surgery,” Industrial Robot,
Vol. 31, No. 6., pp. 499-508, 2004.
• NSF EEC-9731478
• Whitaker Foundation RG-02-911
• NIH R01-EB002004
Engineering Research Center for Computer Integrated Surgical Systems and Technology
Two pairs of strain gauges in full bridge configuration to maximize sensitivity and
measure bending forces are mounted on a da Vinci instrument