SIGCHI Conference Paper Format - Get as DOC

Document Sample
SIGCHI Conference Paper Format - Get as DOC Powered By Docstoc
					                  Tactile Diagrams at a Distance: A Multimodal E-
                               learning Prototype
                 Declan McMullen                                                         Donal Fitzpatrick
                School Of Computing                                                    School Of Computing
                Dublin City University                                                Dublin City University

ABSTRACT                                                               deliver graphical material at a distance. A system was
To date, learning material that relies heavily on graphically          designed, implemented and evaluated that allows for talking
intensive material has been predominantly inaccessible to              tactile diagrams to be delivered at a distance without the
visually impaired learners. Various solutions have evolved             need for teacher support. Finally, user interaction with
but have relied on the presence of a teacher to aid the                tactile diagrams will be evaluated in order to assess whether
learner’s interaction. This paper outlines research that aims          enhancements can be made to improve a users learning
to provide independent learning of graphical material to               experience. The target group for this work will be second
visually impaired learners, through the use of a multimodal            and third level learners.
distance-learning environment.
                                                                       TACTILE IMAGE HISTORY
INTRODUCTION                                                           Methods of tactile image creation and delivery have
Historically, the conveying of graphical materials to                  evolved over the years. Each method brings its own
visually impaired students has been difficult to achieve.              advantages and disadvantages. The following are some of
Learning material that relies heavily on the use of                    the most commonly used methods.
diagrammatical data or other visually intensive material
remains predominantly inaccessible for visually impaired               Collage and Tooling
learners. This is especially true in STEM (Science,                    The collage method involves the layering of material and
Technology, Engineering and Mathematics) subjects. A                   objects on to a surface in order to produce a tactile
variety of methods have evolved that aim to solve the                  representation of a graphic. A wide variety of items can be
problem, ranging from those based on technology to the                 used to create the collage. Materials such as threads, wire,
construction of models which the learner, relying on tactile           cloths of varying thickness and textures, corks, foam, wood
representations, can employ                                            and so on, can be combined to produce the affect desired by
                                                                       the tactile creator. An advantage of the collage method is
The various methods used to depict graphically intensive
                                                                       that the layering of objects can produce multiple tactile
data assume the presence of a teacher, or the inclusion of
                                                                       levels and provide the student with a sense of depth.
Braille labels. If a learner is operating in a distance-learning
                                                                       Disadvantages of the method are that it can be quite time
environment, or cannot read Braille there is little to no
                                                                       consuming to produce a diagram using the collage method.
access to graphical material. This is a barrier preventing
those with a visual impairment from the autonomy of                    Also, in order to identify the material the student is
learning that is becoming commonplace in the modern E-                 touching, Braille labels are required to be placed on the
learning climate. The aim of this ongoing research is to               tactile. There is limited space for the labels and thus limited
investigate techniques that will allow for the independent             information can be placed on the tactile diagram.
learning of visually intensive material by visually impaired           Additionally if the student does not read Braille they will
learners. As multimodal learning has been shown to                     gain no benefit from the additional information provided by
enhance the experience of learners [8], our aim is to                  the labels.
produce a multimodal distance-learning environment for
visually impaired learners.                                            Tooling is a similar process to the collage method involving
                                                                       the manual creation of tactile diagrams. The tooling method
The research has been split into three steps. Firstly,                 involves the carving out of a diagram onto a surface such as
historical methods used to deliver graphical material to               Braille paper, aluminium diagramming foil, or plastic
visually impaired learners have been investigated. Each                sheets. The graphic is carved into the surface on the rear
technique has been compared and contrasted in order to                 side pressing the front side out and thus producing a tactile
inform our own design decisions. Secondly, current                     diagram on the front of the sheet.
methods for delivering graphical material in a classroom
                                                                       Collage is often used as the first step in the tooling process,
setting were evaluated and the process adapted in order to
                                                                       where the surface being used for the diagram is pressed

onto the completed collage in order to produce the tactile       are the T3 developed by Touch Graphics [6] and IVEO
diagram. The advantage of tooling is that the completed          developed by ViewPlus [5].
diagram can be quite clear and easy to navigate. It has
                                                                 Both systems operate best when used in a classroom
similar disadvantages to the collage method in terms of
                                                                 scenario with the presence of a teacher who can assist the
time required to create the tactile and the limited amount of
                                                                 visually impaired learner with navigating the graphical
additional information that can be provided.
                                                                 material. A different approach would need to be taken in
                                                                 order to provide visually impaired learners with
Mass Production
Vacuum form, often referred to as thermoform, is a process       independent access to graphical material. The E.U
used for making copies of master tactile diagrams. The           sponsored AHVITED project [1] aims to produce a system
masters can be made from any of the methods described            which can deliver talking tactiles to visually impaired
above. These master copies are kept safe and the                 learners at a distance [4]. This is discussed in the following
thermoform copies are distributed to the students. The           section.
advantage of thermoform copies is that they produce very
                                                                 TACTILE DIAGRAMS AT A DISTANCE
clear tactile diagrams and are quite sturdy which aids in
                                                                 It was decided that the first step to solving the aims of the
transportation and repeated use. A disadvantage is the
                                                                 research was to take an existing technique of delivering
equipment required is large and expensive.
                                                                 tactile graphics and modify it to operate in a distance-
Swell Paper is a type of plastic sheet that is coated with       learning scenario. All learners require some form of
thermally foamed microcapsules which respond to intense          guidance from time to time. Courses being delivered in an
heat. Diagrams can be printed directly on to the Swell Paper     E-learning environment are no exception. It would be
using an inkjet printer. A piece of equipment, called a fuser,   necessary to use software to provide guidance to a learner
is used to heat the sheet. When the diagram is passed            when they required it. As talking tactiles already required
through the fuser all black ink areas swell up producing a       software to operate, it was decided to produce a system that
tactile image. The advantage of such a method is that            could deliver talking tactile diagrams to learners at a
production of the diagrams can be extremely rapid. The           distance.
disadvantage lies in the fact that the tactile diagrams
                                                                 From evaluating visually impaired learners and assessing
produced are not as clear as other methods as the ability to
                                                                 the operation of current talking tactile systems a set of
convey depth is minimal.
                                                                 requirements were produced that a talking tactile system
Lately, advances in Braille embossing technology have            should adhere to in order to make it viable in a distance-
created another method of producing diagrams. Modern             learning scenario. The requirements were:
embossers allow for the dots to be placed in closer
proximity to each other and at various heights, thereby                 An intuitive overlay authoring procedure
facilitating the production of high quality graphical material          The ability to identify an overlay without the use
with varying levels of depth. The ability to place dots in               of additional equipment
closer proximity provides the latest Braille embossers with            A straight forward overlay loading procedure
a higher dots-per-inch resolution. This allows for a variety           Simple localization of the software and the overlay
of textures to be achieved using a combination of varying                content
dot heights and proximities.                                     The solutions to these requirements are discussed in the
                                                                 following sections.
Tactile Information
Historically Braille labels were the only means a tactile        Overlay Authoring
producer had of conveying additional information. As there       It was important that authors be given the ability to create
was limited space for the label not much information could       overlays as quickly and as intuitively as possible in order
be conveyed. Talking tactiles have endeavored to solve this      for the content to be viable in a distance learning scenario.
restriction.                                                     To this end the authoring process was arranged into four
                                                                 simpler steps. This procedure is referred to as image
The most common talking tactile overlays are created with        annotation.
ink on swell paper or with Braille dots. The overlay with
the image is placed on top of a touch screen device.             Firstly, an author executes the image annotation program. A
Important regions on the image can be marked using an            template will appear immediately on screen inside the
authoring program. Layers of audio description can then be       programs main panel. Secondly, an author chooses an
attached to those regions. When a learner presses on the         image to import; this can be in JPG, BMP, GIF or PNG
regions of the image the layers of audio can be played back.     format. Once they have chosen their image it will be
This solves the space restriction that Braille labels suffered   imported into the program and placed inside the relevant
from and enables larger amounts of supplementary                 workspace. The next stage allows authors to mark regions
information to be conveyed. Two examples of this method          of interest in the image on screen using the mouse. An
                                                                 author can click around the border of the relevant regions in
order to mark them for annotation. Layers of information              overlay they will be given instructions to move their finger
can now be attached to annotated regions. The information             up, down, left or right until they reach the desired element.
layers are what the author wishes the learner to here when
                                                                      Previous systems suffered from a complex counter intuitive
they press on that portion of the graphic. Layers can either
                                                                      series of tactile controls that could confuse the learner. The
be text, delivered using text-to-speech, or audio files
                                                                      tactile controls available in the AHVITED system provide
delivered using a multimedia player. Finally, the overlay
                                                                      intuitive control of the system and the tactile overlay. An
can be printed to swell paper directly from the annotation
                                                                      example of the tactile template can be seen in Figure 1. The
program and passed through a fuser.
                                                                      image displays all of the tactile buttons, the tactile pin
The ability to import and annotate an image on screen                 system and a sample tactile graphic.
simplifies the interaction strategy used in previous systems
and thus provides an intuitive and relatively speedy                  Overlay Identification
authoring environment. Versions of the system have been               For distance learning it was imperative the learner be
created for Windows XP, Windows Vista, Ubuntu Linux                   capable of identifying an overlay without the need to use
and Mac OSX. This allows for the authoring and delivery               any additional devices. During the delivery of lesson
of overlay content on any of those platforms. An overlay              content the system will ask the learner to place a specific
authored in one environment can be delivered in another.              overlay on the device and they must be able to locate it. In
This provides authors and learners with a freedom of choice           order to achieve this each overlay is given a unique four-
not available in previous systems.                                    digit pin number. This number is represented on the overlay
                                                                      using a system called a tactile pin.
Overlay Template
                                                                      On the right side of the overlay there is a series of tactile
As the learner would be using the tactile overlay to interact
                                                                      circles, four columns consisting of nine circles per column.
with the system, it was important that a simple overlay
                                                                      Each circle in a column is connected to the next using a
template be designed that would provide intuitive access to
                                                                      tactile line and the top row of circles are connected with a
all necessary functions. The template contains three tactile
                                                                      similar line. The tactile line is there to provide guidance
buttons that run along the top of the template. Two are
                                                                      whilst a learner moves their finger from one circle to the
placed beside each other justified to the left and the final
                                                                      next. Each circle is initially empty. The circle that relates to
button is justified to the far right. A work area takes up the
                                                                      a part of the pin number is filled in.
majority of the space underneath the buttons spanning from
left to right. The graphic the learner will be interacting with       When asked to identify an overlay the learner begins by
is placed inside the work area. The tactile pin system is             placing their finger in the top circle of the first column.
located to the right of the work area. This is discussed in the       They move their finger down the column counting the
next section.                                                         circles as they go. If they encounter a filled circle this
                                                                      represents part of the pin number. If they do not encounter a
Every element on the template was spatially placed to ease
                                                                      filled circle then this column represents the number zero.
learner interaction with the graphic and provide intuitive
                                                                      They then move their finger back to the top circle, follow
control of the system. For example the triangular load
                                                                      the line across to the top circle of the next column and
button on the right hand side of the template is centered
                                                                      perform the process again. This occurs for all four columns
directly above the tactile pin inputs. This allows the button
                                                                      in the tactile pin. Once completed the learner should now
to be found easily when the learner wishes to load an
                                                                      know the pin of the overlay they were checking.
overlay after entering the appropriate pin. Additionally the
square pause button and circular index buttons are placed to          Capitalizing on the tactile nature of the overlay enables this
the top left of the work area. This allows a learner to use the       process to function correctly. The tactile pin can be seen in
tactile border around the work area to quickly locate either          Figure 2.
of those buttons when they wish to perform certain
The pause button can be used to temporarily stop the
playback of audio layers attached to the diagram. This
allows a learner to put a layer on hold if they are interrupted
and resume it later by pressing the pause button once again.
The index button allows a learner to cycle through a list of
all annotated regions in the graphic. Each press of the
button will select the next item in the list. When the learner
finds an element they wish to navigate to they hold their
finger on the index button and the element is selected. The
learner can now be provided with audio guidance in order
to locate the selected region. As the learner presses the
                                                                                       Figure 1. Overlay Template

Overlay Loading                                                    is used to translate the overlay content into additional
Previous systems required the user to calibrate their touch        languages.
screen at various times. Performing this operation each time
was time consuming and provided a possibility for                  When a user wishes to translate text into another language
inaccurate calibration thus introducing errors into overlay        they first select which language they are translating to. A
identification and loading. In a distance learning scenario        window will appear listing all English text strings in a
constant calibration could become a barrier for the learner.       column on the left hand side of the screen. There is a
In order to simplify the learners’ interaction the system          corresponding column on the right hand side of the screen
requires this process to occur only once. This is referred to      into which the user can type the relevant translations. This
as synchronization and should only need to be performed            mimics the look and feel of Microsoft Excel. Once
again if the touchscreen loses synchronization with the on         completed the user can save their translation.
screen graphic. With the synchronization already in place          The software generates and exports a catalog file in the
overlay loading is a straight forward procedure.                   required language. The catalog file can then be placed into
The learner follows the same procedure to load the overlay         the locale directory of the software and the new language is
as they did to identify an overlay except when they locate a       immediately available. Providing a quick and intuitive
filled in circle in the tactile pin they press it. This sets the   method of translation ensures that the maximum number of
part of the pin number that the column they are checking           distance learners can be accommodated
represents. If no filled in circle is found nothing is pressed
and the system assumes a zero for that column. Once they
                                                                   The prototype is now entering its initial evaluation phase.
are satisfied that the correct pin has been entered they press
                                                                   An iterative approach will be taken to evaluation by
the load button and annotation information for their overlay
                                                                   refining and reevaluating the system based on user trials.
becomes available. They can now press regions of the
                                                                   The system will first be assessed by each of the European
graphic in order to receive audio feedback.
                                                                   partners involved in the AHVITED project. The evaluators
                                                                   have expertise in tactile production and delivery and will
                                                                   provide valuable feedback as to the effectiveness of the
                                                                   system. This information will be used to refine the system
                                                                   before it begins a second phase of evaluation involving
                                                                   visually impaired learners from across Europe.
                                                                   Areas to be assessed include; ease of overlay authoring, the
                                                                   design of the overlay template, the usability of the tactile
                                                                   pin system, the capability of a learner to use the system
                                                                   without teacher support, the cognitive aspects of using a
                                                                   talking tactile at a distance in comparison to an in class
                                                                   scenario and the overall intuitiveness of the approach.

                                                                   FUTURE WORK
                                                                   With a prototype system in place and entering its evaluation
                                                                   phase the final stage of the research can be investigated.
                                                                   The aim is to investigate the next phase of evolution in
                                                                   delivering graphical material to visually impaired learners.
                                                                   Through user evaluation a picture of what techniques
                     Figure 2. Tactile Pin                         learners are using to interact with tactile diagrams will
                                                                   emerge. This will allow the construction of a model of
Localization                                                       various interaction strategies that a system should support.
It was important for the system to be viable in a distance-        Users will be asked for suggestions as to how interacting
learning scenario that the software and the overlay                with images could be made more intuitive or interactive and
annotation information be capable of intuitive, quick              thus enhance their learning experience. The evaluation
localization. For this reason catalog files are used to            should also expose the flaws in current tactile interaction
represent all textual information. Translation capabilities        techniques and systems.
have been created to allow for rapid localization.
                                                                   One obvious barrier preventing full distance learning is the
The translation software comes in two forms. The first is a        static nature of the tactile diagrams. Unless the learner has
separate program that can be used to translate all text            the necessary equipment their tactile diagrams will need to
strings related to the system itself such as menu items and        be produced elsewhere and provided to them either in
help information. The second uses functionality from the           person or through the mail. This hinders any dynamic
first but it is embedded into the authoring environment and        distribution model of images. There are limitations as to
how interactive the experience can become when the                   provide further education           opportunities     and    foster
diagram is not capable of altering in any way. There are             independent learning.
also barriers preventing us from taking advantage of
adaptive hypermedia techniques such as user profiling in             ACKNOWLEDGMENTS
order to personalize the learning experience.                        This work is supported by the European Union Socrates
                                                                     Minerva Program.
Ideally if the printed image was not required and graphical
material could be delivered using software alone the above           REFERENCES
problems could be overcome. In order for this to work                1. AHVITED. Ahvited Project. (Accessed
another mode of interaction must be made available so the               12/08/08)
learner can navigate the graphic. We believe that haptic
interaction could be used to interact with graphical material        2. Baillie, S., Rendle, D.: A Virtual Reality Simulator for Training
that exists purely in software.                                         Veterinary Students to Perform Rectal Palpation of Equine
                                                                        Colic Cases. International Meeting for Simulation in
Research, such as the haptic cow [2] and haptic back [10],              Healthcare , 2008
have shown that haptic interaction can be used successfully
to enhance the learning experience. If haptic interaction            3. Chen, H., Sun, J., Jin, X.: “Interactive Haptic Deformation of
proved to be a viable image interaction technique for                   Dynamic Soft Objects”, VRCIA‟ 06, PP. 255-261
visually impaired learners it could expand the capabilities
of the delivery environment. Textures could be replicated            4. Fitzpatrick, D., McMullen, D.: Distance learning of graphically
                                                                        intensive material for visually impaired students. ICCHP2008
haptically in order to provide more realistic sensations to
the learner. These textures could be dynamically altered             5. Gardner, J., Bulatov, V.: Scientific Diagrams Made Easy with
based on the preferences a user has set in their user profile.          IVEOTM. ICCHP 2006, PP. 1243-1250
Learners could be provided with an overall view of an
image and then allowed to zoom in on specific section in             6. Landau, S., Wells, L.: Merging of tactile sensory input and
order to gain supplementary information. Learners could                 audio data by means of the Talking Tactile Tablet. Eurohaptics
also be allowed to manipulate an image and move elements                2003
around in order to explore it further similar to that seen in
[3]. The capabilities of two haptic devices, the Sensable            7. Novint Falcon
Phantom Omni [9] and the Novint Falcon [7], will be
explored in order to assess if they can be used to deliver
graphical material in the manner required.                           8. Pietrzak, T., Martin, B., Pecci, I., Saarinen, R., Raisamo, R.,
There are numerous possibilities however it is important                Järvi, J.: "The MICOLE Architecture: Multimodal Support for
                                                                        Inclusion of Visually Impaired Children", ICMI' 07
that any solution meet the needs of visually impaired
learners. Any solution that hinders the learning experience          9. Sensable Phantom.
or causes cognitive overload for the learner is not a valid             omni.htm (Accessed 12/08/08)
outcome. For this reason the results of the user assessments
will be extremely valuable in driving forward this portion           10. Williams, R.L., Srivastava, M., Howell, J.N., Conatser, R.R.,
of the work.                                                            Eland, D.C., Burns, J.M., Chila, A.G.: The Virtual Haptic Back
                                                                        for Palpatory Training, „ICMI‟ 04
This paper has outlined research that aims to provide
visually impaired learners with independent access to
graphically intensive material. The paper reviewed
historical methods of tactile production and discussed their
advantages and disadvantages. The requirements for a
talking tactile system capable of delivering graphical
material at a distance were described. A system was
discussed, as part of the AHVITED project, which aims to
provide solutions to those requirements. The paper was
concluded with a description of future work.
This work will provide new methods for visually impaired
learners to interact with graphical material. It will also
make technical subjects, that rely heavily on graphical
material, available to visually impaired learners in an
autonomous fashion. This will enhance accessibility,