Learning Center
Plans & pricing Sign in
Sign Out

Evaluating an interactive voice response system for persons - CSIR


									                    Evaluating an interactive voice response system for persons with disabilities
                                                          M. Greeff*, L. Coetzee, R. Alberts and H. Easton
                                                       *Meraka Institute, CSIR, PO Box 395, Pretoria, 0001

                                                           The rest of the paper's layout is as follows:
Access to information empowers people when                 Section 2 highlights related research while
making decisions in their every day life. In South         Section 3 discusses the NAP Initiative. Section 4
Africa, one of the most prominent problems that            highlights the usability evaluation methodology
persons with disabilities face, is access to               that was followed and Section 5 presents the
disability-related information. The South African          participants' response to the questionnaires.
National Accessibility Portal was built to address         Lessons learned from the evaluation through
this need. However, many people do not have                observation and the participants' comments are
access to Internet facilities. Therefore, an               discussed in Section 6 and conclusions drawn
Interactive Voice Response engine was                      from the evaluation are presented in Section 7.
connected to the NAP portal, enabling people to
access information on the portal via a telephone.          2. Background
This paper highlights the usability evaluation, as
well as lessons learned through the usability              IVR systems have been used in a wide variety of
evaluation by users.                                       application areas, such as health care services
                                                           (Lee 2003), disease management (LaVigne
1. Introduction                                            1998) and call centers (Suhm 2002). Some
                                                           research about an IVR system connected to a
Information empowers people in making                      portal, often called a “voice portal” (Yamazaki
decisions with regards to everyday life. One of            2004), has been used for E-service delivery
the most prominent problems that persons with              (Gant 2002, Karim 2003) or to create platforms
disabilities (or caretakers or family members of           that enable other people to develop a voice portal
persons with disabilities) face in South Africa, is        (Radha 2005). However, not much work has
access to disability related information. To               been done where the community contributes
address this need the South African National               information to a portal that contains disability
Accessibility Portal (NAP portal) was developed            related information. In addition, not much is
which enables persons with disabilities to easily          known with regards to connecting such a portal to
get hold of information. However, most people in           an IVR system. When the community contributes
South Africa do not have access to Internet                information to a portal, simplicity of language or
facilities. To overcome this obstacle, an                  terms that are used in the portal cannot be
Interactive Voice Response (IVR) engine was                controlled and guaranteed. Furthermore, when a
connected to the NAP portal. The user can                  portal consists of a vast amount of information, it
navigate to specific information on the portal, with       leads to all kinds of difficulties with regards to
the requested information retrieved and voiced             structuring the information. This directly
out to the user. The user can then continue                complicates the menu structure (and navigation)
navigating through the information, retrieve other         of the portal (many possible categories and
information elements or end the call. In order to          subcategories) and therefore the menu structure
assess the experience of users while using the             has to be adapted for the IVR system.
system, a usability evaluation was performed to
find out how the users experienced the system.             Barnum presents the process of conducting a
These evaluations are important to ensure that             usability evaluation and doing research within this
the issues that were raised by the users, or those         field (Barnum 2001). Once an IVR system has
highlighted through observation, are addressed in          been developed, it is important to ensure that it is
a new design of the IVR system. A system will              usable and accessible to maximise the impact of
only become widely used if people's needs are              the system.
addressed and their levels of satisfaction after
using the system are high. Therefore, lessons              The next section discusses the South African
learned will be used in the design of the next             NAP initiative, as well as the NAP portal (a
version of the NAP IVR system, to ensure an                component of the NAP initiative) and the IVR
improvement with regards to the usability aspects          system that was connected to the NAP portal to
of the system.                                             enable access to the information via a telephone.

                                                           Due to the extremely high cost of imported
3. National Accessibility Portal Initiative                assistive technologies, the NAP initiative aims to
                                                           develop client applications and devices for
The South African NAP initiative is a research,            persons with disabilities that are more affordable
development & innovation project. It aims to               and localised for South Africa’s unique multi-
address the marginalisation of persons with                cultural and multi-lingual environment.
disabilities from the mainstream economy and
society. The project was conceptualised and                An important outcome of the NAP initiative is a
developed by the CSIR, in partnership with the             methodology that describes the process required
Office on the Status of Disabled Persons (OSDP)            to establish an initiative like NAP in a developing
in the Presidency and a representative group of            country. The methodology will be developed
Disabled Persons’ Organisations (DPOs)1.                   based on experiences and lessons learned
                                                           throughout the course of the project.
The NAP initiative (illustrated in Figure 1)
consists of a number of components, that as a              3.1 NAP portal
whole ensures that people with different
disabilities have access to information and                The NAP portal is an accessible, Internet-based
services.                                                  collaborative, content management platform,
                                                           aimed at enhancing inclusivity in the South
The NAP portal is an Internet based collaborative          African disability sector by enabling access to
content management platform and is discussed               disability-related information (Coetzee 2007). The
in more detail in Section 3.1.                             NAP portal allows community-based contribution
                                                           of information that can then be accessed by all
The establishment of NAP centres addresses the             users, once the information has been verified and
need to get relevant services and information to           published.
all people, including those who do not necessarily
have access to computers and the Internet at               An accessible web interface enables interaction
home.                                                      with, and the presentation of information to users
                                                           utilising assistive technologies (Coetzee 2008).
All activities in the NAP initiative are underpinned       The usability and accessibility of the NAP portal
with research and development activities. The              was evaluated through usability evaluations by
research activities include domain research as             participants that represent typical users (refer to
well as         technology research. Technology            (Greeff 2007) for results of the first usability
research includes: basic research that addresses           evaluation) and the results of these evaluations
new theory and exploratory work; directed                  were used as input for a new and improved
research that transforms theory into technology;           interface design of the NAP portal.
and the development of technology building
blocks. Domain research focusses on the                    The main components of the NAP portal are:
disability domain and includes research into the              ● An Enterprise Application Server that
face of disability in South Africa, needs                         comprises of a web container and an
identification, usability and accessibility. All                  Enterprise JavaBean (EJB) container.
domain research is supported and guided by                    ● The NAP portal application consisting of
national and international guidelines and                         the following EJB 3.0 (Sun Microsystems
standards, from both the disability domain and                    2008b) compliant components: Stateless
the usability and accessibility domain. Applicable                Session Beans (SLSB); Stateful Session
results from the research and development                         Beans (SFSB); Message-Driven Beans;
component feed into all the other components of                   and EJB 3.0 entities that utilise a
the initiative.                                                   presentation technology, known as
                                                                  JavaServer Faces (Sun Microsystems
                                                                  2008), combined with a templating
                                                                  framework (Facelets).
1The Independent Living Centre (ILC), The SA                  ● PostgreSQL        database   (PostgreSQL
National Council for the Blind (SANCB), The Deaf                  2008).
Federation of South Africa (DeafSA), The National
Council for Persons with Physical Disabilities in SA       3.2 NAP Interactive Voice Response System
(NCPPDSA), The QuadPara Association (QASA) and
The South African Federation for Mental Health
                                                           The telephony interface to the NAP portal was
                                                           built using an Asterisk (Asterisk 2008) telephony

platform that hosts an instance of MobilED                that retrieves information from the NAP portal.
(MobilED 2008), an IVR engine that runs various           The retrieved information is then voiced out
scripts to guide the user's actions. The NAP              through the Asterisk platform, along with
portal publishes a web services interface,                additional command IVR prompts. These
allowing other applications to browse through the         commands allow the user to continue navigating,
various information categories to retrieve specific       retrieve other information or end the call.
information. MobilED utilises this functionality to
retrieve and represent information as voiced text.        The next section discusses a usability evaluation
The IVR scripts lead the user, retrieving                 of the NAP IVR system that was performed by
appropriate inputs from the user (through DTMF            participants, as representative as possible, of
key presses), resulting in a web services call            future system users.

Figure 1: NAP initiative interaction diagram

Usability Evaluation Methodology
                                                          well as the intended use of the results. After the
This section highlights a usability evaluation that       explanation the user had the option whether or
was performed by participants representative of           not to sign the appropriate form of consent. Only
future users. It discusses the usability evaluation       once the user gave his/her consent did the
process, as well as the results that were obtained        evaluation of the system take place.
from the evaluation.
                                                          It is extremely important that a process is
3.3 Usability Evaluation Process                          followed to ensure ethical practice. Usability
                                                          testing must adhere to an established moral and
Before the user evaluated the system, a briefing          professional guideline because of its involvement
or discussion took place. During this discussion          of people. We followed guidelines provided by
the whole process was explained to the user, as           the Human Sciences Research Council (HSRC).

The evaluation of the NAP IVR system was done              latest events, a talking thermometer, the Orlando
at the Meraka Institute. The user’s interactions           stadium and signing to your child.
with the system were observed and video-
recorded for later analysis. The participants were         3.7 Questionnaires
also asked questions with regards to their
experience while using the system.                         The participants were asked to perform specific
                                                           tasks as highlighted in Section 4.4. After they
3.4 Recruitment of Participants                            completed the tasks, they were asked the
                                                           questions of the aesthetic and post-evaluation
A request for volunteers was sent out via E-mail           questionnaires.
to staff of the Meraka Insitute and then the
people who volunteered to assist with the                  Aesthetics Questionnaire
evaluation were contacted. When someone was                The Aesthetics Questionnaire was used to
identified as a possible volunteer, they were              determine the participant's overall experience,
contacted either via phone or E-mail and asked             emotions and feelings after using the NAP IVR
specific demographical questions. According to             system. Some of these questions also re-appear
these questions the participants were selected for         in the post-evaluation questionnaire to ensure a
the evaluation.                                            true reflection of the participants' evaluation of
                                                           the usability of the system.
3.5 Demographic Profile of Participants
                                                           Post-evaluation Questionnaire
Table 1 contains the demographic profile of the            The post-evaluation questionnaire focusses more
participants. Any sensitive information was                on the various tasks that the participants had to
removed to adhere to the privacy, confidentiality          perform and how they experienced the NAP IVR
and anonymity ethics.                                      system while performing these tasks. It does
                                                           contain certain questions of the Aesthetics
3.6 Tasks                                                  Questionnaire as cross-reference, to ensure a
                                                           true reflection of the participants' experience with
The tasks that the participants had to complete            regards to the usability of the NAP IVR system.
were finding information about the following:

Table 1: Demographic Profile of Participants
              Participant A   Participant B   Participant C     Participant D   Participant E   Participant F

Gender       Female           Female          Female            Male            Male            Male
Age          27               29                                52              41              36
First        Tshivenda        Sepedi          Setswana          Isixhosa        English/        English
Language                                                                        Afrikaans
Disability   None             Mobility        None              Visual        Visual          None
                              impairment –                      impairment –  impairment –
                              in wheelchair                     wears glasses Blind; Mobility
                                                                              impairment –
                                                                              limited use of

4. Usability Evaluation Questionnaires                     Question 1: How did you feel after using this
                                                           service of the NAP portal?
This section highlights the participants' replies to       Possible answers: Very satisfied, Satisfied,
selected questions of the questionnaires.                  Neither satisfied nor frustrated, Somewhat
                                                           frustrated, Very frustrated
4.1 Aesthetics Questionnaire                               Replies: 3 participants selected very frustrated
                                                           and 3 participants selected somewhat frustrated.
Satisfaction and emotional experience                      Reasons: 5 participants highlighted the voice as
Below, questions are highlighted that were used            the reason for their frustration – they indicated
to measure the participants' satisfaction and              that the voice “doesn't get tired of talking”,
emotional experience while using the service.              doesn't pause after a comma or full stop, is

unclear and unnatural. 4 participants indicated               Replies: 1 participant selected very unpleasant, 2
that the menu structure and information creates               participants    selected    unpleasant    and     2
frustration, especially     the length of the                 participants selected neither pleasant nor
sentences, menus are too long with too many                   unpleasant.
options to remember, standard menu options                    Reasons: 3 participants highlighted the voice as
vary (e.g. going back to main menu) and difficulty            causing the service to be rated as unpleasant, 1
in navigating between menus.                                  participant said that the system is just unfriendly
Additional Information: The participants' replies to          and 1 participant found the system unpleasant
this question can be found in Table 2.                        because s/he could not find the information that
                                                              s/he was looking for.
Question 2: How pleasant/unpleasant is this                   Additional Information: The participants' replies
service of the NAP portal to use?                             to this question can be found in Table 3.
Possible answers: Very pleasant, Pleasant,
Neither pleasant nor unpleasant, Unpleasant,
Very unpleasant

Table 2: Participant's response to Question 1
             Participant A Participant B Participant C Participant D              Participant         Participant
                                                                                       E                   F
Selection    Very            Very frustrated Very               Somewhat        Somewhat             Somewhat
             frustrated                      frustrated         frustrated      frustrated           frustrated
Reason       The voice, it   The voice is      Couldn't use     “The voice is   Struggling to        Participant:
             doesn't get     not clear,        own tricks to    unnatural,      understand the       Could not hear
             tired of        sentences are     use the          menus are too   speech, very         clearly what the
             talking; it     too long – lose   system           long, have to   slow, standard       information is.
             doesn't         concentration                      remember a      options vary         Cannot
             pause for a                                        lot of stuff,   (e.g. to return to   navigate when
             comma or full                                      very            main menu),          he wants to.
             stop. “Record                                      autonomous.”    cannot navigate      Cannot always
             my voice”.                                                         the text that you    remember what
                                                                                are listening to     to press,
                                                                                (must say            message is too
                                                                                (global option)      long – cannot
                                                                                press * to skip to   remember the
                                                                                next item and        number.
                                                                                something to
                                                                                skip to previous
                                                                                item), cannot
                                                                                search for
                                                                                speaks too
                                                                                much ( instead
                                                                                of saying: ...
                                                                                “press 2# to
                                                                                hear about ...”;
                                                                                you can only
                                                                                say “2# for ...”

Table 3: Participant's response to Question 2
            Participant A        Participant B   Participant C       Participant D     Participant        Participant
                                                                                            E                  F
Selection   Difficult           Very difficult   Difficult          Difficult        Between             Neither easy
                                                                                     Neither easy        nor difficult
                                                                                     nor difficult
                                                                                     and Difficult
Reason      There are so Sentences too           “I can't see       “The voice is    It would have       In general: it is
            many options long                    them (portal       unnatural,       been even           difficult to
            -forget what                         and IVR            menus are too    more difficult if   visualise the
            options are                          system) as the     long, have to    I didn't know       menu structure
            available. It                        same thing. If     remember a       the NAP             – must remind
            will be better                       the voice has      lot of stuff,    portal. It would    himself a few
            to break it                          past, the          very             be easier if        times to
            down into                            moment is          autonomous.”     you could say       remember
            smaller units.                       gone and I lost                     to the system       where
                                                 it”.                                what                information is
                                                                                     you are
                                                                                     looking for.

Accomplishing tasks                                              structure and therefore to know where to look for
Below, questions are highlighted that were used                  specific information.
to determine how easy/difficult the service makes                Additional Information: The participants' replies
it for participants to accomplish the required                   to this question can be found in Table 4.
                                                                 Question 4:How easy/difficult could you recover
Question 3: How easy/difficult does this service                 from mistakes?
of the NAP portal make it for you to accomplish                  Possible answers: Very easy, Easy, Neither easy
what you wanted to do?                                           nor difficult, Difficult, Very difficult
Possible answers: Very easy, Easy, Neither easy                  Replies: 2 participants selected very difficult and
nor difficult, Difficult, Very difficult                         4 participants selected difficult
Replies: 1 participant selected very difficult, 3                Reasons: The following reasons were given by
participants selected difficult and 2 participants               the participants for their selections: Sentences
selected neither easy nor difficult.                             are too long and too much information is given;
Reasons: The following reasons were presented:                   the list of options are too long; there are no global
There are too many options per menu - forget                     options, e.g. to go back to the main menu, which
what options are available when end of list is                   makes navigating really difficult.
reached; the sentences are too long; voice is                    Additional Information: The participants' replies
unnatural; and it is difficult to visualise the menu             to this question can be found in Table 5.

Table 4: Participant's response to Question 3
             Participant A         Participant      Participant        Participant D    Participant E       Participant
                                        B                C                                                       F
Selection   Difficult           Very difficult    Very difficult       Difficult       Difficult          Difficult
Reason      There is a          Sentences too “I didn't!”              To go back       Participant       When you pick
            long list –         long, too much                         you have to      feels that        up the phone
            shorter list will   information                            listen until the since he is an    the system
            be easier and       (need to                               end and there IT person,           should inform
            then you don't      summarise the                          is no global     other people      the user of a
            have to wait        information).                          option – the     without this      global option
            so long             What will                              number to go knowledge or          to return to the
                                happen if there                        back differs     background        main menu.
                                are lots of stuff                      from menu to will struggle
                                on the system,                         menu.            even more.
                                will you have to
                                press 20# or

Table 5: Participant's response to Question 4
            Participant A         Participant           Participant        Participant D    Participant E      Participant
                                       B                     C                                                      F
Selection   Neither easy        Difficult            Difficult             Difficult       Neither easy       Easy
            nor difficult                                                                  nor difficult
Reason      If the system       Couldn't hear        The system is         All of the      It you use the     Couldn't
            has a clearer       what they said.      probably not          reasons         system many        remember
            voice, not so       Voice speaks         set up how I          mentioned       times, you will    what the
            many options        without a break      learn.                above           be able to use     options were.
            and change          between words        Therefore it is                       it quicker. The
            the pace of the     – difficult to       difficult.                            system is
            voice, it will be   pick up what                                               usable.
            easier              has been said

4.2 Post-evaluation Questionnaire                                    Possible answer: Open ended
                                                                     Replies: Listening to the voice; and that you
Question 5: What did you like the most about                         cannot navigate the menus (move easily between
accessing information over the telephone?                            the various options)
Possible answer: Open ended                                          Additional Information: The participants' replies
Replies: Accessing information from home where                       to this question can be found in Table 7.
no Internet access is available; and the amount
of information that is available                                     The next section presents the lessons that were
Additional Information: The participants' replies                    learned through the participants' response and
to this question can be found in Table 6.                            observations that were made during the usability
Question 6: What did you like the least about
accessing information over the telephone?

Table 6: Participant's response to Question 5
   Participant        Participant             Participant            Participant        Participant          Participant
        A                  B                       C                      D                  E                    F
I can access       Nothing at all           Hearing that         Technology –      The availability of That there is
NAP at home                                 there is a lot of    the fact that you the service         much information
when I have no                              information –        can access                            and it is available
access to                                   but I cannot         information
Internet                                    hear what            using a
                                            information          telephone

Table 7: Participant's response to Question 6
  Participant A      Participant B           Participant C           Participant D      Participant E        Participant F
Voice              Listening to the     Voice                    Synthetic voice That you cannot           Quality of the
                   voice – it is really                          – cannot          navigate through        automated
                   a horrible voice!                             understand it, it the menus               speaker. Would
                                                                 is inhuman, gets                          have felt more at
                                                                 to be boring                              ease if he could
                                                                                                           hear or

5. Usability Evaluation Results                           6. Summary of Results

This section discusses general observations that          The participants identified three aspects of the
were made and the participants' comments with             system that should be improved, namely the
regards to navigating the information and locating        voice, menu navigation and provided information.
specific information via a telephone. In this paper
only subjective data, such as questionnaires and          Voice
user comments, are reported.                              All participants felt that the voice was
                                                          unsympathetic, inhuman and unclear. The speed
5.1 General Observations                                  of the voice is also not consistent – at times the
                                                          voice speaks very slow, e.g. when reading a
The following general observations were made              telephone number, and at other times the voice
while the participants used the system:                   speaks too fast, with no break between commas
    ● Participants looked under the sections              and end of sentences. An important lesson
        events or news for the 2010 World cup –           learned is that many participants were so irritated
        the term recreation is not commonly               by the voice that the other issues weren't even
        understood and most participants didn't           mentioned or seemed much less important to
        connect the 2010 World cup with                   them.
    ● Recovering from errors were extremely               Menu Navigation
        difficult – there is no standard number for       The participants identified the following problems
        global options, e.g. to go back to either         with regards to menu navigation:
        the main menu or previous menu or to                  ● The main menu doesn't provide an option
        listen again to the available options of a                to listen again. This forced participants to
        specific menu.                                            hang up and redial in order to listen to
    ● Many times, e.g. when reading a list of                     the main menu again.
        events, the system doesn't just give the              ● Many participants had to listen a few
        name of the events, but also reads out a                  times to understand that one option of
        summary. This is very irritating and                      the main menu is to “Hang up”. “Recent
        frustrating when the list contains a few                  events” is also not very clear when read
        elements and will be even worse when                      by the voice.
        the list is long. The system should only              ● When a list of options are read, only the
        read the event name and provide the                       title should be given and not the title and
        information after selection. Otherwise it                 a summary. This takes a lot of time and
        takes extremely long before the user can                  leads to high levels of frustration.
        even select to go back to the main menu               ● The list of options on many menus are
        or even to list through the list to be able               too long. This is time consuming and by
        to make a selection. Another solution will                the time that the user reaches the end of
        be to allow the user to interrupt the                     the list, s/he forgot the options that are
        system, and to move on to the next                        available. Many participants then also
        option on the menu at any point in time.                  wanted to go back to the main menu, but
    ● The pace of the voice is not consistent                     by accident selected the wrong number
        and not effective. At times it is very slow               (due to a lack of memory) and then
        and at other times it is too fast. It also                ended up selecting the repeat option.
        doesn't pause after commas or a full stop                 Then they had to listen to the whole list
        (end of sentence).                                        again, unless they could remember the
    ● The        pronunciation of words are                       number to go back to the main menu.
        sometimes so bad that the participants                ● One participant mentioned that it would
        couldn't understand what was being said.                  have been helpful if it was possible to
        Even in the main menu, the participants                   navigate through the items in the list, for
        had to listen a few times to understand                   example if there were global keys that
        that one of the available options is to                   allowed a user to skip to the next item in
        “hang up”.                                                the list or to skip to a previous item in the
                                                              ● There are no global options to return to
                                                                  the main menu or a previous menu. This
                                                                  forced participants to listen to the whole

        list of options before they could go back          voice, as well as a paper-based evaluation where
        to another menu. This led to high levels           the participants can concentrate on other issues,
        of frustration.                                    without being put off or influenced by the quality
    ●   Too much information was given when                of the voice.
        reading a list of options . When the
        system said “Press 2# to listen to                 Results obtained from the participants' replies to
        information on ...”, by the time that the          the questionnaires, as well as observations made
        sentence was finished, the participant             during the evaluation, will be used to improve the
        couldn't remember the number that s/he             IVR system.
        had to select any more. It also irritated
        many participants to keep hearing “Press           8. Acknowledgements
        x# to listen to information on...”. They
        would have preferred that the system               The authors wish to express their gratitude to the
        rather said: “Select the number according          NAP technical team for developing the portal and
        to the information that you want to listen         IVR system and the NAP usability and
        to. Press 1# for assistive devices, 2# for         accessibility team for conducting the usability
        recent events..”                                   evaluations.
                                                           9. References
The system provides too much information when
providing the possible options in a menu. When
e.g. a list of events are given, only the event            ASTERISK., 2008. Asterisk: The Open Source
should be given and not the name of the event              PBX & Telephony Platform.
and a summary.                                   
                                                           (Accessed: 25 August, 2008).
Many participants also didn't understand the term
recreation – this can occur frequently in future           BARNUM, C.M., 2001. Editor: DRAGGA, S.
with users whose first language is not English.            Usability Testing and Research, Longman Pub
Since the community contributes information to             Group.
the portal, simplicity of language or terms that are
used in the portal, cannot be controlled or                COETZEE, L., GOVENDER, N., VIVIERS, I.,
guaranteed.                                                2007. In Proceedings of the 2007 international
                                                           cross-disciplinary   conference      on    Web
The next section highlights conclusions that were          accessibility (W4A); The National Accessibility
made from the project and the usability                    Portal: An accessible information sharing portal
evaluation of the IVR system.                              for the South African disability sector. Volume
                                                           225 of ACM International Conference Proceeding
7. Conclusions                                             Series, pages 44–53, Banff, Canada.

In this paper the NAP IVR system is presented.             COETZEE, L., VIVIERS, I., 2008. In Proceedings
An IVR engine is connected to a Internet-based             Cunningham and M. Cunningham, editors, IST-
content management platform, namely the NAP                Africa     2008       Conference    Proceedings;
portal. A usability evaluation was conducted to            Accessibility Observations of Visually Impaired
test the usability of the NAP IVR system. Results          Users using the South African National
that were obtained from the usability evaluation,          Accessibility Portal, Windhoek, Namibia.
as well as lessons learned, were presented.
                                                           GANT, J., GANT, D., 2002. In Proceedings of the
The most important issues highlighted by the               35th Annual Hawaii International Conference on
participants were the quality of the voice, menu           System Sciences; Web portal functionality and
navigation and the amount of information                   state government e-service. Pages 1627–1636.
presented to the user.
                                                           HSRC., 2008.
An important lesson learned is that many
participants were so irritated by the voice, that          LAVIGNE, M. and TAPPER, K.A., 1998. Disease
the other issues weren't even mentioned or                 Management & Health Outcomes, Interactive
seemed much less important to them. Therefore,             Voice Response in Disease Management:
it will be useful in future to do both a usability         Providing Patient Outreach and Improving
evaluation where the participants can hear the             Outcomes, 4(1):1-16.

LEE, H. et al., 2003. Nursing Outlook, Interactive        SUHM, B. et al., 2002. In Proceedings of the
Voice Response System (IVRS) in health care               SIGCHI Conference on Human Factors in
services, 51(6):277-283.                                  Computing Systems: Changing Our Wolrd,
                                                          Changing Ourselves, A Comparative Study of
KARIM, M., 2003. International Review of                  Speech in the Call Center: Natural Language
Administrative Sciences, Technology and                   Call Routing vs. Touch-Tone Menus, pages
improved service delivery: Learning points from           283-290, Mineapolis, Minnesota, USA.
the malaysian experience. 69(2):191–204.
                                                          SUN MICROSYSTEMS., 2008. Java Server
MOBILED., 2008. MobilED.         Faces Technology.
( Accessed: 25 August 2008).                    
                                                          (Accessed: 25 August, 2008).
POSTGRESQL., 2008. PostgreSQL.                               SUN MICROSYSTEMS., 2008b. JSR-000220
(Accessed: 25 August 2008).                               Enterprise Javabeans 3.0.
RADHA, G., RAO, K., SIEW, E., 2005. In                    final/jsr220/index.html.
Proceedings of the 7th International Conference           (Accessed: 25 August 2008).
on Advanced Communication Technology
Design, development and implementation of a               YAMAZAKI, Y., IWAMIDA, H., WATANABE, K.,
voice portal builder using next generation                2004. Fujitsu scientific and technical journal,
networks technology – a case study. Pages 201–            Technologies for voice portal platform. 40(1):
204.                                                      179–186.


To top