Get documents about "A Distributed, Collaborative and Haptic-Enabled Eye Cataract
Document Sample
HAVE'2006 - IEEE International Workshop on
Haptic Audio Visual Environments and their Applications
Ottawa, Canada 4-5 November 2006
A Distributed, Collaborative and Haptic-Enabled Eye Cataract Surgery Application
with a User Interface on Desktop, Stereo Desktop and Immersive Displays
Abdelwahab Hamam, Saeid Nourian, Naim R. El-Far, Francois Malric, Xiaojun Shen, and Nicolas D.
Georganas
Distributed & Collaborative Virtual Environments Research Laboratory
School of Information Technology and Engineering
University of Ottawa, Ottawa, ON, Canada
{ahamam, saeid, naim, frank, shen, georganas}@discover.uottawa.ca
Abstract – In this paper, we discuss the technologies and on this project include CANARIE1, the University of Ottawa
approaches utilized in developing a cataract eye surgery simulation Eye Institute and the University of Alberta.
that will be used for training novice surgeons. The three different
techniques described in this paper, are all hapto-visual techniques Step by step, our work plan started with enhancing an
that resulted in three different implementations of the application:
existing eye model and adapting it to our proposed
2D simulation, 3D immersive simulation, and completely immersive
simulation.
application’s needs. Following that, we modeled from scratch
the instruments and tools required for the surgery after
The paper begins with an introduction and a general overview of the carefully examining these tools through surgery DVDs and
medical procedure of the cataract surgery. Then an overview of the visits to the Ottawa Hospital. After we had the eye and all
eye and surgical tools modeling is given. Following that, the necessary tools modeled, we decided to implement our
architecture and technology of each of the three design techniques is simulation using three different approaches: 2D Tele-Haptic
given. Finally the paper concludes with future work to improve the Simulation using OpenSceneGraph [3] and the Omni haptic
application. device [4] ; 3D immersive simulation using Reachin Display
[5] and the Phantom haptic device [6] ; and finally, a visually
Keywords – Immersive Haptic Application, Surgical Simulation, immersive simulation on multiple screens using our custom-
Cataract Surgery, Virtual Reality, Physics Engine, Software built DIVINE system [7] .
Engineering, Immersive Projection Technology (IPT), Graphical
User Interface (GUI), Tele-Surgery, DIVINE System, Cave
Automatic Virtual Environment (CAVE), VRJuggler, CAVElib,
II. OVERVIEW OF CATARACT SURGERY
OpenSceneGraph, Reachin
Cataract, the formation of cloudy material in the lens of
I. INTRODUCTION the eye, is a condition that causes blurred vision progressively
worsening with age. Cataract surgery is a procedure in which
Cataract surgery is a complicated procedure despite the the “clouded” lens is replaced with an artificial one as shown
fact that it only takes around seven minutes for a skilled in Figure 1 below. In order to take out the old lens, a
surgeon to perform one and the success rate for such surgery precision surgery tool is used to cut through the thin layer of
nowadays is around 99%; Medical residents need months to cornea. Another specialized tool is then inserted inside the
acquire the skills necessary to become proficient at this cornea to break down the old lens and suck its remnants in.
procedure avoiding the several minor and major Finally, the new lens is inserted in place of the old one.
complications that are typical of it [1] .
Generally, the surgery procedures are as follows:
Common training approaches that medical residents Sedation
undertake include training on dead animals’ eyes or on elastic o A local anesthetic is injected below the eye of the
rubber eyes. However, with the advancements in science and patient.
technology, these approaches are becoming more and more o The eye is stretched using hooks so that the
obsolete; a more sophisticated and cheaper approach has maximum area of the eye is available to the surgeon.
become feasible and is preferred [2] o The surgery then begins by two small incisions into
the cornea using a keratome at the top and the
3D Virtual Reality and haptics have been two useful bottom of the eye parallel to the iris parameter.
technologies that can simulate real life situations with real- o A third larger incision is made to the right of the
time user interaction; these technologies are a main research eye.
interest of ours at the DISCOVER Lab at the University of Capsulorhexis
Ottawa where we are working on a project to simulate
cataract eye surgery using various software frameworks,
display technologies and haptic devices. Main collaborators 1
CANARIE is Canada's advanced Internet development
organization. http://www.canarie.ca
o A medical forceps is inserted in the large incision to Eye surgery tools were also modeled in 3D Studio Max,
rupture the lens capsule by grabbing the surface of but their modeling was from scratch based on measurements
the lens and going through circular motions. of actual tools provided by the University of Ottawa Eye
o A hydro-dissection tub is inserted into the same Institute.
incision to apply fluid to the lens.
Phacoemulsification All the graphics modeled were converted to VRML and/or
o The phacoemuslification tube is inserted into the 3DS formats in order to load them into our simulations at a
large incision while an aiding hook is inserted into later stage. Although 3D Studio Max is a high-end modeling
the lower incision. software, the format conversion resulted in some loss of
o The tube then breaks down the lens into smaller and quality. The end result, however, was satisfactory after some
smaller pieces until the cloudy lens disappears; it is elaborate texture-mapping.
referred to as divide and conquer process.
o Any remains of the previous process are cleared by Having used 3D Studio Max allows the tool models to be
inserting cortical cleaning tubes into the incisions converted into different formats in the future if necessary or
one at a time. so desired. One of the challenges faced during the modeling
Replacement of Artificial Lens of the tools was balancing model quality and the model’s file
o An artificial lens placed on a tube is inserted size. We chose to create moderate quality models with
through the large incision to the middle of the eye. optimized file sizes paying special attention to details at
o A hook is the inserted into one of the incisions to points where the tool model would interact with the other
spread the new lens gently so that it does not objects in the scene (e.g. the eyelids, the lens, etc), which
contain any folding in it. allows for more realistic graphic interaction at points of
o Fluid is applied to the top of the cornea near the contact. Figure 3 shows some tools that we have modeled.
incision to clean any remains of blood.
Figure 1- A blurry cataract lens (left); A cataract
surgery procedure (right) where a new lens is being
implanted [9]
III. EYE AND TOOLS MODELING
The eye model we used was bought commercially[10] but
was later modified in order to fit our application’s needs.
Modification, done in 3D Studio Max, included appropriately
changing the size as well as completing the eye envelope
since the model purchased was of “half” an eye as shown in
Figure 2.
Figure 3- Some Surgical tools modeled;
Forceps and Keratome
IV. 2D SIMULATION
The 2D surgical simulation was built on a framework for
developing haptic-enabled surgery applications. This
framework is being developed at the DISCOVER Lab and
Figure 2- Eye Model Purchased (left), Eye Model
contains the several layers shown in Figure 4. The
Modified (right)
geometrical models of the organs and surgical tools are at the
bottom of the hierarchy. They are used as the basis of the
virtual environment that is created and managed by the Scene
Graph Layer. The dynamics of the scene are controlled by the haptic devices. It continuously receives inputs from
Physics Layer which assures that the interaction of the whichever haptic device assigned to it; it will then translate
objects in the scene conforms to the laws of physics [20] . those inputs into high-level commands that can be used for
The inputs to the Physics Layer that ultimately result in the variety of tasks such as controlling the position of a surgical
scene interactions are generated by the Control Layer. These tool. Specifically for this application the haptic control uses
interactions are usually triggered by the inputs received from either OMNI or MPB haptic devices. The choice of which
the input devices or higher layers. The Application Layer is haptic device to use can be simply set through the
responsible for the overall flow of the simulation. It is here configuration component. The remote client component uses
that the implementation of the components is dependant on a TCP connection for graphics data (to assure consistency)
the type of surgery that is being simulated. Finally, the and UDP for haptic data (for higher data rate).
Configuration Layer is strategically located on top of all
others in order to have a clear view of the overall operation. It The simulation component is the brain of the simulation as
is responsible for general tasks such as recording a simulation it utilizes all other components in order to simulate a
session or reconfiguring the general parameters of the particular surgery procedure. It is therefore highly
application. customizable and extensible by the application developers.
The user interface and training components are also highly
customizable but they additionally offer a rich set of features
Configuration Layer that the application developers can choose from instead
High-Level implementing their own.
Application Layer
Control Layer
Physics Layer Session
Low-Level Recorder/Player Configuration
Scene Graph Layer
Model Layer
Figure 4-Layered Architecture of proposed framework User Training
Interface Simulation Aid/Score
Each of the design layers presented above consist of one
or more key components each with their own set of
functionalities. Figure 5 shows these components along with
the general inter-component data flow. The 3D models
needed for the simulations (such as organs and surgery tools)
are usually stored in some ASCII or binary format such as Remote Client Haptic Control Tools Manager
VRML or 3DS. The model loader component loads these
models, converts them to scene-graph and sends them to the
graphics rendering component for display.
In addition to rendering, the graphics rendering component
makes these models available to the Physics and Collision
Engines. Instead of manipulating the graphics directly, the
higher layer components issue high-level commands to the Collision Engine Physics Engine
physics engine for moving or manipulating the models. The
physics engine, in coordination with the collision engine, Graphics
generates physically realistic results that correspond to the Rendering
incoming high-level commands.
The components at the control layer are in charge of Model
controlling the virtual scene by issuing commands to the Loader
lower layers based on the requested tasks. The remote client
for example is in charge of synchronizing the scene so that
the remote and client users perceive a same simulation in a Figure 5- Major components of the proposed
framework
distributed application. The tools manager is specific to
surgical applications and it provides a convenient means of Most surgical simulations and training applications
managing the collection of tools to be used in the surgery require that the sessions be recorded for later analysis. This is
procedure. The haptic control is a generic component for all particularly useful in training applications when the trainer
might want to go over the simulation again in order to
pinpoint the trainee’s mistakes. The session recorder
component is in charge of the task of recording and replaying
simulations. The configuration component reads script files
that outline general configuration settings that are needed by
the various components in the framework.
V. 3D IMMERSIVE SIMULATION
Reachin Technologies is a company located in Stockholm,
Sweden. It specializes in human-computer interaction via
haptics; the sense of touch. Reachin manufactures hardware
related products including the Reachin Display, which is a
stereo display hardware that allows the user to intuitively
manipulate the Phantom haptic device coherently while
looking at the stereo display. Reachin also has software
products carrying its name including the Reachin API, which
provides libraries to allow haptic and graphic synchronization.
The Reachin Display and the six degrees of freedom Phantom
Haptic device are shown in Figure 6. Figure 7 – Reachin Architecture: graphics rendering is
independent of haptic rendering.
Figure 6- Reachin Display (left) [5] . Phantom Haptic
Device (right) [6]
The architecture of the Reachin API, allow the separation
of rendering the graphics and rendering the haptic devices.
The frequency of rendering differs for both, and each runs on
a separate thread. However this eases the loading and
customization of the graphics and haptics separately as shown
in Figure 7. For our 3D immersive application, we utilized
the Reachin API to load the eye model and tools from VRML
files. Then we customized the collision detection and force
feedback to meet our goals using methods in the API.
We created a simple user interface that enables the user to
change the tools by clicking on the button using the haptic
device thus eliminating the need for a mouse or keyboard.
When the user selects a certain tool the haptic stylus changes
to the selected tool, allowing the user to perform the required
operation. Each tool has the proper algorithm activated upon
selection. For example, the cutting algorithm is activated
when the user selects the surgical knife[12] ; the keratome.
Figure 8 shows screenshots of the application. Figure 8- 3D Immersive simulation screenshot (2D and
inverted)
VI. DIVINE IMMERSIVE SIMULATION Placement for the cameras that would provide some good
viewing angles of the user without interfering with the
Immersion can be defined as the state of deep and natural immersive projection was another constraint. Furthermore,
involvement a user experiences when interacting within some only a limited space was available for such a system.
space. A computer graphic display is said to be immersive
when it is rendering on its display mediums, specifically for
some user’s view, in a way to make him or her perceive some
virtual space as part of the real physical space. In other words,
if an immersive display were to show a coffee cup on a
tabletop display medium, then that coffee cup should remain
in the same place on that tabletop no matter the viewer’s
position and angle of sight, exactly as it would, had there
been a real coffee cup on that tabletop. To achieve this
coherency, the display platform has to account for the user’s
movement (head and body, position and rotation) and display
the graphics in such a way that the displayed objects look as
if they were physically there and viewed from a given angle
and position. The most common Immersive Projection
Technology (IPT) setup is the CAVE (Cave Automatic
Virtual Environment) system. Typically, a CAVE system
constitutes of several large display screens to fill as much of a
user’s field of view as possible. With video projectors, Figure 9- The DIVINE system at the University of
computer systems that can synchronously display in stereo on Ottawa DISCOVER Lab
these multiple screens, a 3D position and orientation sensor,
and stereo glasses complete a classic CAVE setup. By
entirely surrounding the user with viewpoint specific stereo
rendering, “full” immersion can be achieved whereas all the Considering all these requirements, DIVINE is composed of
perceived physical space is virtual space. This full immersion two corner visualization volumes and one central projection
is very difficult to achieve by reason of its requirements. The screen. Figure 9 shows the hardware setup. This central
integration of a multitude of technologies, all with their own screen can combine the two corners into a long, almost
practical limits, imposes severe restrictions in the end product, seamless virtual window. Each screen projection in the
with a price tag exponentially proportional to the size and system is driven by one workstation and two LCD projectors.
number of screens. Circular polarization filters in front of each projector and
non-depolarizing screen materials and first-surface mirrors
In contrast, our DIVINE system allows the display of were used to provide the stereo projection capability.
immersive computer graphics on a desktop-like workspace
more appropriate for close interaction, and alleviates some of The workstations are dual-Xeon based, with nVIDIA Quadro
the problems related to a fully immersive CAVE. A major FX 3000 video cards, capable of 3D stereo on independent
advantage making it suitable for finer use is the greater pixel left and right outputs. Projectors are SANYO PLC-XT16
density offered by concentrating the video projection on with the wide-angle lens option and network controller. They
smaller and closer screens. Also of significance is its better are stacked in left and right pairs using an extruded aluminum
affordability and its much smaller space requirements. structure. The circular polarizing filters (3M HNCP37%
Worthy of noting here is that DIVINE was custom designed x0.30”) left and right were purchased in large sheets and cut
from the ground up by DISCOVER lab members and was to 7” squares. They are placed ~5” from the lens in front of
built by a local company. It was designed to be versatile in its each projector using simple metal brackets and paper clips.
use. Its main purpose is to provide an immersive visualization This distance is required to prevent the filter’s plastic from
experience for two users (one per corner unit) and allow them melting because of the considerable heat generated by the
to directly interact within the projected space. For maximum projectors. The screen material used is STEWART filmscreen
immersive effect, the video projection had to be done in 150 on 1/4” plastic backing. Each screen is 4’x3’ (for a
stereo, and a position/orientation tracker integrated in the 1024x768 resolution’s 4/3 aspect ratio). The mirrors are
system. For the users to be able to extend their arms and DaLite PFS 90% reflective mylar mirrors.
move their head in the projected space (for natural
manipulation and visualization), three orthogonal rear- Each corner unit can be used independently. Having two
projection screens were used: a table at waist height, and two usable IPT in this way allows us to experiment with
walls to form a corner. Another consideration in the design collaborative virtual reality applications in a relatively small
was the need for a well lit environment for video-capture of space. Collaboration can be directly observed by third parties
the user (not used in the context of this application). by having the two systems side-by-side in the same room.
For rendering 3D graphics on an IPT, each of its constituent VIII. CONCLUSION
screens must display the proper view projection of the
graphics relative to the eyes of the user (approximated by In this paper we have presented our current work in
using a 3D tracker attached to his/her head). Several software developing a hapto-visual eye cataract surgery training
packages propose to do this IPT rendering.. In this work, we application on three different platforms. Each platform has its
explore the use and limitations of some of these offerings in own advantages. Based on the review of the eye doctor, the
the context of our eye cataract surgery application, having 3D Reachin application is the more realistic, while the 2D
“mixed and matched” software and hardware combinations in application is more practical since it does not need its own
order to come up with a generic framework that we could use display and can be portable easily. Overall the application is
to run our 3D applications in an immersive environment, prosperous and with continuous testing and modification it
using different display platforms (e.g. desktop, HMD, CAVE, will reach its goal of replacing the more expensive training on
DIVINE, etc) with no need to recompile our code. Some of animal eyes.
the software packages that we have studied include: For
networking and cluster management, VRCO CAVElib [8]
and VRJuggler [13] ; For graphics rendering, SGI OpenGL IX. REFERENCES
Performer [14] , OpenSceneGraph [3] , Virtools Dev VR
authoring software [15] with Physics pack and VR pack, and [1] S. El-Defrawy, Basic and Clinical Science of Cataract, University of
basic OpenGL [16] ; And for tracking using our Ascension Ottawa Eye Institute, Basic Science Course Presentation, 2001.
[2] G. Welch, A. State, A. Ilie, K. Low, A. Lastra, B. Cairns, H. Towles, H.
Flock of Birds tracker, VRCO trackd [17] , and the Fuchs, Immersive Electronic Books for Surgical Training, IEEE
University of North Carolina’s VRPN [18] . Multimedia, July-September 2005, 22-35.
[3] OpenSceneGraph, http://www.openscenegraph.net/. Accessed on
After several trial-and-error runs, we built a programming 1/9/2006.
[4] Phantom Omni Haptic Device,
framework that allows any 3D application to run on DIVINE http://www.sensable.com/products/phantom_ghost/phantom-omni.asp.
provided that the graphics are rendered by OpenSceneGraph, Accessed on 1/9/2006.
the physical tracking be done by VRCO trackd, and the [5] Reachin Display,
networking/cluster-management be done by VRCO CAVElib. http://www.reachin.se/products/Reachindisplay/index.asp. Accessed on
1/9/2006
[6] Phantom Desktop Device,
The above combination was arrived at in stages: First of http://www.sensable.com/products/phantom_ghost/phantom-
all, we favored VRJuggler to manage our computer cluster desktop.asp. Accessed on 1/9/2006.
mainly because it was open-source and supported [7] DIVINE System, http://www.discover.uottawa.ca/divine/. Accessed on
1/9/2006.
OpenSceneGraph, the library we were using for our graphics [8] CAVELib, http://www.vrco.com/CAVELib/OverviewCAVELib.html.
programming. However, we could not translate DIVINE’s Accessed on 1/9/2006.
physical coordinate system to that of VRJuggler due to a bug [9] SJudith Lee, Gretchyn Bailey, “Cataracts and Cataracts Surgery”,
in VRJuggler so we switched to CAVElib. VRCO’s trackd Journal of the American Medical Association, 2004
works well with CAVElib so we chose that as our tracking [10] Human Eye Model [Search Result], Exchange3D. Accessed on
8/9/2005. www.exchange3d.com
software. We had an issue getting OpenSceneGraph to call [11] Reachin Technologies, http://www.reachin.se, accessed on 17/8/2006.
CAVElib functions, but we were able to solve this problem. [12] Y. Xu, “Making Incisions on an Eye Model in Reachin Display”.
2/9/2005. Internal report. DISCOVER Lab at the University of Ottawa,
VII. REMAINING WORK Ottawa, ON, Canada.
[13] VRJuggler, http://www.vrjuggler.org/, accessed on 1/9/2006.
[14] SGI – OpenGL Performer,
The initial implementation of the application is completed. http://www.sgi.com/products/software/performer/, accessed on
The 2D and 3D Reahin implementation include haptic 1/9/2006.
[15] Virtools,
sensing and force feedback, while the DIVINE immersive http://www.virtools.com/solutions/products/virtools_dev_new3_5.asp,
simulation is not haptic-enabled yet and research is still going accessed on 1/9/2006.
on how to integrate haptic hardware with the proposed [16] OpenGL, http://www.opengl.org/, accessed on 1/9/2006.
framework for running immersive applications. [17] Tackd, http://www.vrco.com/trackd/Overviewtrackd.html, accessed on
1/9/2006.
[18] VRPN, http://www.cs.unc.edu/Research/vrpn/, accessed on 1/9/2006.
Testing has started to evaluate the implementation. An eye [19] N.R. El-Far, S. Nourian, J. Zhou, A. Hamam, X. Shen, N.D Georganas,
surgeon did a comprehensive review on all three A Cataract Tele-Surgery Training Application in a Hapo-Visual
implementations, and provided our team with valuable Collaborative Environment Running over the CANARIE Photonic
Network, HAVE 2005.
comments and suggestion for improvement. Our next step is
[20] S.Nourian, “xPheve: Extensible Physics Engine for Virtual
to apply those changes on the current design while Environments” , University of Ottawa, 2004
maintaining the testing from different categories of users.
This way we can receive as much feedback as possible while
applying them gradually to the application.
Related docs
