Docstoc

PURPOSE

Document Sample
PURPOSE Powered By Docstoc
					 Becky Jones (bjones@depaul.edu)
 Joanne Keung (joanne.keung@comcast.net)
 Anna Mikrut (annamikrut@hotmail.com)
 HCI 460
 Assignment 3
 October 30, 2006

                   Usability Testing and Final Report


A. EXECUTIVE SUMMARY

 The usability test conducted for Metrarail.com was intended to help discover
 areas in need of improvement that would provide an opportunity to enhance
 the quality and attractiveness of the Metra website. The test was designed to
 address the specific goals of the company, which are to attract more clients
 and reduce the customer service call volume. The specific test objectives,
 nature of the research and data collection methods, results and
 recommendations are addressed in the sections to follow.

 Based on the test results, several problem areas were identified. The first
 major area is the overall website navigation, which provided a significant
 challenge and fair amount of confusion to the users. Creating a more uniform,
 consistent and clear navigation would be recommended. The second major
 problem area is the train schedule and route information functionality. One of
 the key factors contributing to the difficulty in locating train routes or
 schedules is a hard-to-locate and poorly marked train system map. Although
 interactive, the map lacks clear indicators (e.g. hover effect) that it is
 clickable. Also, the train route abbreviations are meaningless to the user and
 are not easily associated with the full route names. Improving the system map
 by providing clear route names, mouse-over indicators, clickable train
 stations leading directly to schedule and fare information would increase
 immensely the effectiveness of the schedule and route locator functionality.
 The final major problem area is the online ticket purchase. The first-time
 buyers are required to complete and mail an application along with a copy of
 a cancelled check, which has caused a tremendous level of frustration. An
 electronic registration form should be created and allowed to process credit
 card information.

 The major findings, as outlined above, identify the significant weaknesses of
 the Metrarail.com website and need to be addressed to alleviate potential
 problems and increase the effectiveness of the website. Further details are
 provided in the Findings and Recommendations section.




                                          -1-
B. METHOD
 The following Test Plan describes the nature, setup and methodology used
 during the usability testing of the Metrarail.com website.

 PURPOSE
 Metra Rail has decided to redesign their website. They realize their current
 website is out of date and would like to provide a better transit informational
 site to help Metra riders and potential riders to learn more about the
 convenience of public transportation. Based on the data they gather from their
 website tracker, there are several keys reasons they would like to improve
 their website.

    1. Would like to promote Metra Rail as the number 1 public transportation
       for Chicagoland area and surrounding suburbs.
    2. Would like to increase the usage of users to buy fares online.
    3. Would like to cut down the number of calls to the help desk for
       information; instead would like to increase the number of users to use
       the website to find the information they need (e.g. train routes).

 Metra Rail has decided to perform a usability test to understand the areas of
 their website that need improvement to accomplish their business goals.


 PROBLEM STATEMENT AND TEST OBJECTIVES
 The following are the specific questions that need to be answered during the
 usability test. We would also like to record the time it takes to complete these
 specific tasks.

    1. Are the end users able to find a specific train line?
    2. Will the end users be able to find fare for their specific train route?
    3. Will users be able to purchase train tickets online?
    4. Will users know where the train stations are located?
    5. Will users be able to find information on other transportation options to
       help them get to places of interest after getting off a Metra train?
    6. Will users buy train fare using this website again?
    7. Do users find this website helpful and convenient?


 USER PROFILE
 Our business goal is to increase the number of Metra riders.Therefore, our
 target users are the potential riders that haven‟t and/or do not often ride
 Metra. Potential riders range from 16-65 years old. For our test plan, we will
 recruit students, co-workers, teachers and family members from age 19-45.
 Our test subjects should have basic knowledge of the use of computer and the
 internet. They should have some experience with the general public


                                           -2-
transportation but not necessarily with Metra. For the pilot test, one
participant will be evaluated based on availability.


TEST DESIGN
Using a script to ensure the uniformity of observation steps and procedures,
the usability test will be structured in the following manner:

   1. Participant greeting and informed consent documentation
      presentation.
      The test monitor will greet each participant ensuring that they feel
      welcome and relaxed. The monitor will express the gratitude to the
      participants for offering their time to take part in this study. The
      informed consent form will then be disclosed to each participant to
      obtain their fully informed and voluntary agreement to participate in
      the test. The monitor will let participants know that they will be
      observed and that notes related to the test will be taken. The user will
      be informed that they can refuse to participate at any point in the
      duration of the test and have the right to withdraw the data gathered
      from their participation in the test.

   2. Orientation.
      The test monitor will provide information on the nature and purpose of
      the test. The monitor will explain to each participant the structure of the
      test and what course of action they will be expected to take, which will
      include introducing the tasks to be performed by participants. The
      participants will be asked to behave in a way that is natural to them in a
      given situation. It will be emphasized that the subject of evaluation is
      the application and not the user in order to help the participants feel
      more at ease.

   3. Usability Test.
      The usability test consists of five tasks to be performed sequentially.
      The test monitor will administer tasks one by one paying attention to
      the participant‟s cues as to each task‟s completion. The test monitor will
      make mental or occasional written notes on the problem areas in the
      participant‟s performance. The timer and data logger will record
      elapsed time elapsed, errors and other detailed information on the
      user‟s performance pertinent to measuring the application‟s usability.

   4. Satisfaction Questionnaire and Participant Debriefing.
      Upon completion of the usability tasks, the participants will be asked to
      complete a questionnaire intended to obtain information on the user‟s
      satisfaction with the MetraRail website. After completing the
      questionnaire, the participants will be debriefed by the test group
      (monitor, timer, data logger) in order to answer any questions that
      participants might have or discuss any unclear issues that were noticed

                                          -3-
      during the test performance. The debriefing session is more informal
      and allows the participants to express their concerns more freely.

      After the debriefing session, the participants will be thanked for their
      time and willingness to take part in the test and released.


TASK LIST
Below is a list of tasks that will be administered to each test participant. The
system requirements for each task are an uninterrupted Internet connection
and a functional browser.

   1. Scenario: Assuming a hypothetical situation where the user lives in the
      Schaumburg area and plans to go downtown Chicago on the upcoming
      Saturday. To avoid traffic and the inconvenience of finding and paying
      for parking, the user intends to take a Metra train, but needs to find the
      schedule first.
      Task: Find a schedule for train going from Schaumburg to
      downtown Chicago on a Saturday that arrives downtown before
      1PM.

      Successful Completion Criteria:
      The user navigates to the page with Saturday Inbound time table for
      Milwaukee District West Line and locates the schedule. The user can
      either read the schedule or point it out on the screen.

      Benchmark: Finds schedule within 3 minutes.

   2. Scenario: Same as in number 1 above. Having located the schedule, the
      user wants to know how much the train ride will cost.
      Task: Find the fare for the train going from Schaumburg to
      downtown Chicago.

      Successful Completion Criteria:
      The user navigates to the page with Fares for the Schaumburg –
      Downtown Chicago train route.

      Benchmark: Finds the fare within 3 minutes.

   3. Scenario: Builds upon number 1 and 2 above. The user knows the
      schedule and fares. Now, he or she would like to purchase a train ticket
      online for convenience.
      Task: Purchase a train ticket by Internet.
      Note: It will be explained to the test participants that no actual purchase
      will be taking place, but only a simulation of purchase.

      Successful Completion Criteria:

                                           -4-
      First time users – Users complete the “Application Form” and print the
      form (if printer is available).

      Registered Users with PIN #: The user will fill out the “Ticket-by-
      Internet” Order Form and will indicate that the “Order Now” button
      needs to be clicked.

   4. Find the address and telephone number for the Schaumburg
      station.

      Successful Completion Criteria:
      The user navigates to the Schaumburg station information page.

      Benchmark: Finds the station information within 2 minutes.

   5. Scenario: The purpose of the user‟s trip downtown is to go to the Field
      Museum to see a new exhibit. It has been established that the user will
      take a Metra train and will arrive downtown around 1 PM on Saturday.
      At this point, the user wants to find out how to get around after getting
      off the Metra train. Are there other transportation options available that
      will take them to their desired destination (e.g. Field Museum)? The
      user would like to find out whether the Metra website provides that
      additional information.
      Task: Find information on other forms of public transportation that
      will help you find your way from the Union Station to the Field
      Museum.

      Successful Completion Criteria:
      The user selects the “RTA Planner” from the Transit Links navigation
      menu.

      Benchmark: Finds the link in 1 minute.


TEST ENVIRONMENT
Our test environment will be setup similar to the Simple Single Room Set up
described in the Rubin text. We will have all of our people in one room and
there will be a computer workstation at a desk. We will have our Test Monitor
sit off to the side and a few feet behind the user at the workstation. This way
the Test Monitor can see facial cues and have a good seat to observe the user.
We will also have our timer and data recorder off to the side in the observer
section. Our room set up will differ slightly from Rubin‟s because we will not
have a camcorder recording video of the session. In our test environment we
also will not have the monitor that displays the same screens as the user as
they follow along the tasks. The visual of the room will look something like
this (taken from Rubin text):


                                         -5-
TEST MONITOR ROLE
The test monitor will greet each test participant, provide an overview of the
test purpose and procedures, present the informed consent documentation
and obtain the participant‟s agreement to take part in the test. The test
monitor will initiate the tasks but will refrain from providing guidance through
obstacles encountered by test participants. The test monitor will make efforts
to minimize any distractions toward the participants and ensure that they feel
comfortable with the test and the test environment. The test monitor will make
mental notes of actions and problem areas encountered by test participants.

After the tasks test, the monitor will ask the participant to complete the
satisfaction questionnaire. Following, the monitor will debrief the participant
trying to clarify any issues or performance problems encountered during the
test. At the end, the test monitor will express the gratitude and will thank the
participant for taking part in the test.

In addition to the test monitor, a data logger and timer will be present during
the usability test.


EVALUATION MEASURES
Our test objectives should obtain both qualitative as well as quantitative data.
We will record our observation of the user performing the tasks on a log
sheet. This log sheet will collect data that details whether or not the user
completed the task, how much time they took to complete the task wherever
relevant based on our task objectives, and gives us some space to comment
or make notes about the observations.

Another way we will collect data for evaluation will be the post test
questionnaire. The questionnaire will address two sections the Navigation
and the Content of the site and will get information from users that we weren‟t
                                          -6-
able to collect just by observing their interaction with the Metrarail.com site.
We will use a Likert type scale that user‟s can rank their response to our
statements.


REPORT CONTENTS
We anticipate that our final report will find that users had trouble navigating
their way through the Metrarail.com website to complete the tasks.
Specifically, they will struggle in determining the specific train line they are
looking for in order to get from Schaumburg to Chicago, the cost of the fare
and purchase the ticket online, and find travel directions to their next
destination after leaving the train station in Chicago. We also anticipate that
users will not want to purchase tickets online and will not find the site
convenient and easy to use. Our evaluation method of recording the tasks on
the log sheet will address our first 5 test objectives and our post test
questionnaire will address the last 2 test objectives. Our users will likely
provide feedback that they did not enjoy the experience while completing the
tasks we have asked them to perform.




                                          -7-
C. RESULTS
 User Profile Notes:
 User 1 – Traveled on Metra trains numerous times (used two lines other than
 Milwaukee District West). Visited Metrarail.com website occasionally. Non-
 native to the Chicagoland area.
 User 2 – Traveled on Metra several times (did not take the Milwaukee District
 West line). Visited Metrarail.com website occasionally.
 User 3 – Did not take a Metra train. Did not visit Metrarail.com website. Non-
 native to the Chicagoland area.
 User 4 – Frequent Metra train traveler (one line only). Visited the
 Metrarail.com website frequently to check schedules.
 User 5 – Traveled on Metra few times (did not take the Milwaukee District
 West line). Visited Metrarail.com website occasionally.


 Test Objective 1:

 Are the end users able to find a schedule for a specific train line?

 Results:
 Completed Task         User 1   Y
 (Y/N)                  User 2   Y
                        User 3   Y
                        User 4   Y
                        User 5   Y
 Obstacles              User 1   Spent a lot of time reading the navigation
 Encountered                     options trying to find the schedule page
                                 before realizing that a map existed. Once on
                                 the map page, couldn‟t read the station names
                                 - font size too small.
                        User 2   Never seemed to notice the left navigation,
                                 couldn‟t find the map to locate the train route
                                 name, went route by route and searched for
                                 Schaumburg from the schedule list. Search
                                 function on the site didn‟t turn up any results
                                 when searching for “Schaumburg.” Did not
                                 realize the system map was clickable until
                                 after the task was over.
                        User 3   First checked Schedules from top navigation,
                                 then went back and tried left navigation
                                 schedule and discovered they were the same
                                 thing. Obviously not what the user was

                                           -8-
                         expecting the schedule page to look like. Not
                         from Chicago area, hard time finding
                         Schaumburg, once she noticed the system
                         map she was tilting her head to read all the
                         stations and noticed that a piece of the route
                         was coded yellow, thought of it like the CTA
                         trains (color coded) but then realized that
                         couldn‟t be the line because the same route
                         was divided into different colored sections.
                         Then noticed the route name abbrev. MD-W
                         and went to the home page (didn‟t notice it
                         was clickable). Ended up at the station page
                         eventually finding schedule after awhile.
                User 4   This user didn‟t have much trouble locating
                         the schedule, the only struggle he had was
                         locating which line the Schaumburg station
                         was on. Found the system map right away
                         though and found Schaumburg pretty quickly.
                         This user used MD-W abbreviation on map
                         and clicked it to get to the particular line then
                         saw schedules.
                User 5   Noticed top navigation first, clicked “Quick
                         Schedules” and didn‟t know the name of the
                         line/didn‟t really spend much time reading,
                         went back to home and chose schedule from
                         left navigation after using the navigation hover
                         link expansion and reading all the options.
                         Once on schedule user didn‟t know there was
                         a system map so they just clicked on ever line
                         name until they found Schaumburg about ½
                         way down the list of lines.
Actions Taken   User 1   Mouse over top navigation, check bottom
                         navigation, decide to us left navigation go to
                         schedule, couldn‟t find the schedule because
                         line name unknown, looked at system map, hit
                         back button and found line name Milwaukee
                         District West name, clicked on it, found
                         Saturday schedule, Done.
                User 2   Top Navigation to Quick Schedule, clicked on
                         the first schedule, back to list of lines tried the
                         next line on list, no Schaumburg, back button
                         to list, next line on list, no Schaumburg, back
                         button to list, feeling frustrated located system
                         map, couldn‟t search on map, tilted head
                         sideways to read station names, finally found
                         Schaumburg, back button (didn‟t realize map

                                   -9-
                            was clickable) had a hard time with the
                            abbreviated line name from map and locating
                            the correct one on schedule went back to map
                            and then realized that the map was clickable,
                            found schedule, Done.
                   User 3   Top navigation to Quick Schedule, didn‟t know
                            what the line name was, back home, left
                            navigation to Metra system map, didn‟t know
                            where Schaumburg was so gave up after
                            scanning the map with no luck, went back
                            home. Tried SEARCH function for
                            “Schaumburg” no results. Back home, to the
                            system map again, tilting head read every
                            station on each line until Schaumburg located
                            on MD-W back to schedule page (didn‟t notice
                            map was clickable) on schedule page, clicked
                            on station name, and found the station info but
                            no schedule, back and noticed table/chart
                            with schedule, located schedule, Done.
                   User 4   Using left navigation, clicked on schedules,
                            then system map because not sure which line
                            Schaumburg was on, located MD-W and
                            clicked on the MD-W link then to schedules
                            and located schedule, Done.
                   User 5   Using Top navigation first, selected Quick
                            Search, realized no idea what line
                            Schaumburg was on so when back to home,
                            used left navigation hover over each
                            navigation option to see the options under
                            each, decides on “Schedule” started clicking
                            on each line one by one looking for
                            Schaumburg, finally clicks on Milwaukee
                            District West and locates Schaumburg, clicks
                            on it, locates Saturday schedule, Done.
Quotes             User 1   “Since I am new to Chicago, It‟s hard for me to
                            find the station by the map even”
                            “I can‟t even see the text on the map because
                            the text is so small – I don‟t have my glasses
                            with me”
                   User 2   “What does the abbreviation MD-W mean?”
                            “Map shouldn‟t open up in a new page”
                   User 3   “I‟m from Ohio so it might take me awhile to
                            find Schaumburg”
                   User 4   (didn‟t speak)
                   User 5   (didn‟t speak)
Time to Complete   User 1   2 minutes 38 seconds

                                     - 10 -
                       User 2   5 minutes
                       User 3   5 minutes 8 seconds
                       User 4   45 seconds
                       User 5   2 minutes 30 seconds
Questionnaire          User 1   Disagree
ratings                User 2   Strongly Disagree
Relevant question:     User 3   Slightly Agree
“I found the time it   User 4   Slightly Agree
took me to complete    User 5   Agree
tasks was
appropriate.”


Test Objective 2:

Will the end users be able to find the fare for their specific train
route?

Results:
Completed (Y/N)        User 1   Y
                       User 2   Y
                       User 3   Y
                       User 4   Y
                       User 5   Y
Obstacles              User 1   None
Encountered            User 2   Used top navigation for everything from home
                                page, got the schedule for Schaumburg again
                                but the fares weren‟t on there finally found the
                                correct page.
                       User 3   None
                       User 4   Used system map, clicked on MD-W, found
                                station info by clicking on “Schaumburg” but
                                didn‟t indicate fares, had trouble locating
                                fares, finally went back to map and was able to
                                click through and find the information.
                       User 5   Hovered all top navigation options had trouble
                                picking an option.
Actions Taken          User 1   Went to Fare Check on left navigation.
                                Remembered it was on Milwaukee District
                                West, clicked on the fares option selected
                                Schaumburg as starting station, and Chicago
                                as ending station, got fare, Done.
                       User 2   On the map already, got to Schaumburg
                                schedule searching for the fare information,

                                         - 11 -
                                found the check fares page on the home page
                                top navigation, clicked on it, selected
                                Schaumburg as starting station, and Chicago
                                as ending station, got fare, Done.
                       User 3   From drop down menu on current page,
                                selected “Fare Check” selected Milwaukee
                                District West Line, Schaumburg as starting
                                station and Chicago as ending station, got
                                fare, Done.
                       User 4   Back to map, the to MD-W page through
                                clicking map, found line info, clicked
                                “Schaumburg” got station info, not what
                                looking for, back to Metra Home, found “Fare
                                Check” from left navigation, clicked on
                                Milwaukee District North by mistake back to
                                Fare page, selected Milwaukee District West
                                and selected Schaumburg as the starting
                                station and Chicago as the ending station,
                                found fare, Done.
                       User 5   Hover top navigation, select Fare Check, goes
                                to Milwaukee District West link, click
                                Schaumburg as starting station, Chicago as
                                ending station, found fare, Done.
Quotes                 User 1   (didn‟t speak)
                       User 2   “Maybe the map should be more obviously
                                clickable”
                       User 3   (didn‟t speak)
                       User 4   (didn‟t speak)
                       User 5   (didn‟t speak)
Time to Complete       User 1   38 seconds
                       User 2   1 minute 17 seconds
                       User 3   33 seconds
                       User 4   1 minute 39 seconds
                       User 5   40 seconds
Questionnaire          User 1   Strongly Disagree
ratings                User 2   Slightly Disagree
Relevant Question:     User 3   Slightly Agree
“I found the content   User 4   Slightly Agree
to be organized in     User 5   Disagree
an effective
manner”




                                        - 12 -
Test Objective 3:

Will end users be able to purchase train tickets online?

Results:
Completed (Y/N)     User 1   Y
                    User 2   Y
                    User 3   Y
                    User 4   Y
                    User 5   Y
Obstacles           User 1   Didn‟t seem to understand why a user would
Encountered                  need a PIN number, it wasn‟t clear that only a
                             10-day or Monthly pass was all that could be
                             purchased online until after they filled out the
                             entire form.
                    User 2   When the user went to the online purchase
                             section, the first screen that popped up was
                             how great buying tickets online was and they
                             expected to just start filling out the information
                             to buy a ticket online. The user kept trying to
                             find a way to buy a ticket only to discover that
                             they would have to mail in a canceled check
                             and form filled out in order to receive the
                             privilege of a PIN number, which would then
                             allow to register and buy tickets online.
                    User 3   The user found the page rather quickly, but
                             spent a lot of time reading all the text in order
                             to determine that they required a PIN number
                             in order to purchase tickets online.
                    User 4   Once on the online ticket purchase page, the
                             user clicked on the form immediately. When
                             they got to the PIN requirement they gave up
                             on that and went back to the instructions page
                             and didn‟t really read through it. The user
                             went back to the form and proclaimed they
                             were done because they didn‟t have a PIN.
                    User 5   The user took awhile to find the tickets by
                             internet page, but once they discovered that
                             they didn‟t have a PIN number they just said
                             they were done.
Actions Taken       User 1   Clicked on tickets by internet from Fares
                             page, filled out the form, realized a PIN was
                             needed, continued anyway, and then realized
                             they could only purchase a 10-Day or Monthly
                             pass. Done.

                                      - 13 -
                   User 2   Drop down box for tickets by internet,
                            scanned the long text, clicked around on the
                            left navigation in the new pop up window,
                            looked at the form and couldn‟t fill it out
                            without a PIN. Done.
                   User 3   Clicked from Home Page to tickets by
                            internet, read the long message that opened
                            up, clicked on the form, to get a PIN, clicked
                            on the form to purchase tickets online once
                            you have a PIN, Done.
                   User 4   Got to tickets by internet from home page,
                            didn‟t read the long text that popped up, went
                            immediately to the Left Navigation to fill out
                            form for ticket purchase, looked at it and
                            realized no PIN, didn‟t know how to find the
                            PIN, so they said they were done because
                            without a PIN they can‟t finish the form. Done.
                   User 5   Used the bottom link on home page for tickets
                            by internet (this took awhile to locate), looked
                            at details discovered they needed a PIN by
                            filling out the form to get a PIN and then fill out
                            the tickets online purchase page, Done.
Quotes             User 1   “Do I fill out the form? What is the PIN
                            number?”
                            “Oh I can only purchase a 10-day or Monthly
                            pass?”
                   User 2   “This first page is dumb, why don‟t they
                            realize that if I clicked on this I already know
                            what the convenience of buying something
                            online is”
                            “Seems like this is where I want to be, but it is
                            not what I want to do”
                            “having to fill out a form and mail it in to get a
                            PIN number seems contradictory to the
                            convenience of buying something online”
                   User 3   “So I would fill out this form, mail it in with a
                            canceled check and then they will mail me a
                            PIN number and then I can go and fill out
                            another form to place the order?”
                   User 4   (didn‟t speak)
                   User 5   (didn‟t speak)
Time to Complete   User 1   N/A
                   User 2   N/A
                   User 3   N/A
                   User 4   N/A

                                     - 14 -
                        User 5   N/A
Questionnaire           User 1   N/A (this question was added after test user)
ratings                 User 2   Agree
Relevant Question:      User 3   Slightly Disagree
“I expected that I      User 4   Strongly Agree
could purchase a        User 5   Agree
single roundtrip fare
online.”


Test Objective 4:

Will the users know where the train stations are located (i.e.
physical address)?

Results:
Completed (Y/N)         User 1   Y
                        User 2   Y
                        User 3   Y
                        User 4   Y
                        User 5   Y
Obstacles               User 1   Tried to find this information from the
Encountered                      Navigation o the left of the home page, gave
                                 up and tried at the bottom of the home page.
                        User 2   No real obstacles, but user wanted to find the
                                 info without using the site.
                        User 3   No obstacles, found info quickly.
                        User 4   No obstacles, found info quickly
                        User 5   In order to get the info you have to go into the
                                 train line relevant to the station info. This user
                                 clicked the wrong train line (Milwaukee
                                 District North) by mistake because they forgot
                                 which line it was and they thought the line
                                 names had changed order (the didn‟t change
                                 order)
Actions Taken           User 1   Left navigation hover, no luck, went to
                                 schedule and clicked on Milwaukee District
                                 West, from there clicked on Schaumburg,
                                 found the map, then located the address.
                                 Done.
                        User 2   Started at the Home page, went to train
                                 schedules, clicked on Schaumburg station,
                                 and found address and phone number, Done.
                        User 3   Stations to MD-W station located Schaumburg

                                          - 15 -
                            station, found information by clicking on
                            Schaumburg, Done.
                   User 4   From System Map, clicked on MD-W then
                            clicked on the Schaumburg station, found
                            address and phone number, Done.
                   User 5   From the Home Page used the bottom
                            navigation for “Station Info” clicked on
                            Milwaukee District North, realized wrong train
                            line, thought they changed orders, hit the
                            „back‟ button selected Milwaukee District
                            West, clicked Schaumburg, found address and
                            phone number, Done.
Quotes             User 1   “I found the map, but not the station info, Oh
                            wait here we go”
                   User 2   “Do I have to use this site to find that info?”
                            “I‟d rather „Google‟ it”
                   User 3   (didn‟t speak)
                   User 4   (didn‟t speak)
                   User 5   “They changed the order of the rail lines on
                            me that‟s why I went to the wrong line first”
Time to Complete   User 1   33 seconds
                   User 2   30 seconds
                   User 3   15 seconds
                   User 4   10 seconds
                   User 5   15 seconds
Questionnaire      User 1   N/A
ratings            User 2   N/A
                   User 3   N/A
                   User 4   N/A
                   User 5   N/A




                                     - 16 -
Test Objective 5:

Will users be able to find other transit options that will help them
commute to the places of interest once they reach their Metra
destination station?

Results:
Completed (Y/N)     User 1   Y
                    User 2   Y
                    User 3   Y
                    User 4   Y
                    User 5   Y
Obstacles           User 1   Nothing on the Metra site seemed to give the
Encountered                  user the idea that they would get this
                             information on the first run through of all the
                             navigation items. User decided to try going
                             through the CTA, but then gave up on that site
                             and went back to the Metra page. Eventually
                             the user found a .pdf file that gave information
                             on bus routes from Union Station to Chicago
                             destinations.
                    User 2   Clicked on Travel Guides from the top
                             navigation of Metra home page, scrolled down
                             to from Union Station, got the .pdf file but
                             wanted to be able to search for “Field
                             Museum” instead had to scan through it until
                             the proper info was discovered.
                    User 3   Started from Metra home page, clicked on
                             CTA from left navigation found bus schedules
                             and train routes on that page. The user never
                             even attempted to try to get the information
                             from Metrarail.com
                    User 4   No obstacles encountered, found it quickly.
                    User 5   No obstacles encountered, found it quickly
Actions Taken       User 1   Hovered left navigation a few times, selected
                             CTA, clicked around on CTA site, gave up,
                             back to Metrarail.com, finally saw the top
                             navigation for Travel Guides, clicked on from
                             Union Station, got .pdf file, scanned through
                             that until info was discovered, Done.
                    User 2   Immediately used “Travel Guides” from home
                             page top navigation, then to the from Union
                             Station, scanned around .pdf file until bus
                             routes were discovered. Done.


                                      - 17 -
                       User 3   Started at metra Home Page, the clicked on
                                CTA from the Left Navigation, found bus
                                schedules and train routes from that site.
                                Done.
                       User 4   Clicked on Travel Guides from top navigation
                                of Home Page, opened a .pdf file, scaned that
                                info for the bus list, found bus, Done.
                       User 5   From Home page, clicked Travel Guides,
                                selected From Union Station, got .pdf file that
                                gave info on what buses to take, found bus,
                                Done.
Quotes                 User 1   (didn‟t speak)
                       User 2   “I want to search for „Field Museum‟ on this so
                                I don‟t have to read as much”
                       User 3   (didn‟t speak)
                       User 4   (didn‟t speak)
                       User 5   (didn‟t speak)
Time to Complete       User 1   3 minutes 50 seconds
                       User 2   38 seconds
                       User 3   10 seconds
                       User 4   43 seconds
                       User 5   22 seconds
Questionnaire          User 1   N/A (these questions were added after test
ratings                         user)
Relevant Questions:    User 2   (1st question):
                                Agree
“I felt that I could            (2nd question):
easily continue my              Agree
trip to the Field      User 3   (1st question):
Musem”                          Agree
                                (2nd question):
“I felt that I could            Agree
easily continue my     User 4   (1st question):
trip to other                   Agree
destinations in                 (2nd question):
Chicago.”                       Agree
                       User 5   (1st question):
                                Agree
                                (2nd question):
                                Agree




                                         - 18 -
Test Objective 6:
Will users buy train fare using this website again?
Results:
Completed (Y/N)         User 1   Y
                        User 2   Y
                        User 3   Y
                        User 4   Y
                        User 5   Y
Obstacles               User 1   N/A
Encountered             User 2   N/A
                        User 3   N/A
                        User 4   N/A
                        User 5   N/A
Actions Taken           User 1   Filled out post test survey questionnaire
                        User 2   Filled out post test survey questionnaire
                        User 3   Filled out post test survey questionnaire
                        User 4   Filled out post test survey questionnaire
                        User 5   Filled out post test survey questionnaire
Quotes                  User 1   (didn‟t speak)
                        User 2   (didn‟t speak)
                        User 3   (didn‟t speak)
                        User 4   (didn‟t speak)
                        User 5   (didn‟t speak)
Time to Complete        User 1   N/A
                        User 2   N/A
                        User 3   N/A
                        User 4   N/A
                        User 5   N/A
Questionnaire           User 1   (1st question):
ratings                          Slightly Disagree
Relevant Questions:              (2nd question):
“I would purchase a              N/A (this question was added after test user)
ticket online for       User 2   (1st question):
future trips on Metra            Strongly Disagree
trains”                          (2nd question):
                                 Strongly Disagree
“I found that           User 3   (1st question):
navigating the steps             Disagree
to complete the                  (2nd question):
online ticket                    Disagree
purchase was easy”      User 4   (1st question):
                                 Agree
                                 (2nd question):

                                          - 19 -
                               Slightly Disagree
                      User 5   (1st question):
                               Strongly Disagree
                               (2nd question):
                               Strongly Disagree

Test Objective 7:
Do users find this website helpful and convenient?
Results:
Completed (Y/N)       User 1   Y
                      User 2   Y
                      User 3   Y
                      User 4   Y
                      User 5   Y
Obstacles             User 1   N/A
Encountered           User 2   N/A
                      User 3   N/A
                      User 4   N/A
                      User 5   N/A
Actions Taken         User 1   Filled out post test survey questionnaire
                      User 2   Filled out post test survey questionnaire
                      User 3   Filled out post test survey questionnaire
                      User 4   Filled out post test survey questionnaire
                      User 5   Filled out post test survey questionnaire
Quotes                User 1   (didn‟t speak)
                      User 2   (didn‟t speak)
                      User 3   (didn‟t speak)
                      User 4   (didn‟t speak)
                      User 5   (didn‟t speak)
Time to Complete      User 1   N/A
                      User 2   N/A
                      User 3   N/A
                      User 4   N/A
                      User 5   N/A
Questionnaire         User 1   Disagree
ratings               User 2   Slightly Disagree
Relevant Question:    User 3   Slightly Agree
“I found              User 4   Slightly Agree
Metrarail.com to be   User 5   Slightly Disagree
helpful and
convenient.”


                                        - 20 -
D. FINDINGS AND RECOMMENDATIONS

   1. Finding and Explanation:
      The metrarail.com currently has a top navigation and a left navigation.
      The menus in the navigation are not clear and not consistent throughout
      the website.
      During the usability test, users did not know where to look for the
      information they needed. They looked to the top navigation and then
      tried the left navigation. If they did not find what they needed, they
      went back to the main page to search through the menus again. One
      user did not even notice the left navigation at all.

      Recommendation:
      Reorganize and combine the navigation into one more uniform. Use a
      simpler consistent navigation menu throughout the website. This will
      help the user become familiar with the navigation reducing the time
      spent searching through the top or left navigation.

   2. Finding and Explanation:
      New users or new out of town users typically had no clue what
      lines/routes are available for them to take. They would like to see a
      Map of the routes that are available. Currently, the Map can be found
      under the left Navigation on the first menu, fourth down the submenu.
      Four out of five users had difficulties finding the Map.

      Recommendation:
      In addition to having the link on the navigation, provide an easy access
      button or link on the homepage of the website that will take users to the
      Map. The button or link should be in its own box on the homepage,
      explaining what users will see if they click the button/link. For example
      “To see a system map of all the train routes, click here”.

   3. Finding and Explanation:
      The current map is big and some of the stations on the map is placed
      vertically causing the user to tilt their head to read the station names.
      Users had difficulties locating their desired stations because the map is
      so big they didn‟t realize they had to scroll to see more of the map.

      Recommendation:
      Provide a smaller map and give the user options to zoom in and zoom
      out of the map. Rotated all the names of the station that‟s vertical to
      slightly slanted, maybe a 45 degree angle or just enough so the users
      don‟t have to tilt their head.

   4. Finding and Explanation:
      The easiest way to find a specific train schedule is to click on the train
      line on the Map. It is currently not obvious to the users. There is no

                                          - 21 -
   indication that the map is clickable. The user, not knowing the map
   would take them to the train schedule, looked at the map, learned the
   train line name and went back to the schedules page.

   Recommendation:
   Add a hover effect when the user moves the mouse over a train line.
   The cursor should also change into a hand icon. The effect should be
   clear enough for the user to know that the map is clickable.

5. Finding and Explanation:
   To find a specific fare, users had to go to the homepage then navigate
   through the menu to get to the fare information page. Then the users
   had to go through a series of selections about their desired destination
   before getting to their fare information.

   Recommendation:
   Since the train system map is color coded with the intention to show
   pricing zones, there should be clear indication what those colors are
   for. Once the users find their train schedule, there should be pricing
   information listed on this page or a link to the pricing page. This will
   make it more convenient for the user to navigate around the site.

6. Finding and Explanation:
   There is a link on the website to buy fare. The users clicked on the link
   and started filling the form but then they were asked to fill in their pin
   number. If they are new users, they won‟t have a pin number. The
   users had to read on and find out they had to fill another form and then
   fax or mail it in to get a pin number so they could purchase fares online.

   Recommendation:
   If users can not conveniently purchase fares online, then such option
   should not be provided. Several of our users were frustrated and
   thought this feature was cumbersome and they would not purchase fare
   online if they had to mail in a form to get a pin number.

7. Finding and Explanation:
   The current search function does not work very well. Users tried to
   search for a train station name but no result was returned. User tried
   this twice but gave up and went back to the homepage to navigate
   through the menu.

   Recommendation:
   Add a more robust search engine to the site. Give user the option to
   enter “from” and “to” destination to the search engine and have it
   return train schedules, location and fares on the page.



                                      - 22 -
8. Finding and Explanation:
   The additional transit information seems to be readily available. 50% of
   the users used the Travel Guide accessible from the top navigation
   menu and 50% used the Transit Links option (left navigation) providing
   links to external websites, such as CTA. Some of the users seemed to
   be a little confused or lost before locating either option.

   Recommendations:
   The confusion in locating the additional transit information is related
   again to the inconsistent navigation issue. This could be eliminated by
   providing the transit information or travel guide on either left or top
   navigation.

9. Findings and Explanations:
   Based on the post-test questionnaire, half of the users found the website
   only slightly helpful and half did not see it as helpful.

   Recommendations:
   The low level of satisfaction of the website seems to be caused by the
   problems outlined in findings 1-8. Possible solutions have been
   provided in these respective items above.




                                     - 23 -
                APPENDIX (SUPPLEMENTAL MATERIALS)


Scripts: Usability Testing on Metrarail.com

      [Introduce yourself and your team members]

      Hi, my name is Anna Mikrut. This is Becky and Joanne. We are all
      DePaul students and we are doing a study where we try to find out how
      user friendly is the Metra website. First of all, we would like to thank
      you for coming and your willingness to participate in this study.

      Please have a seat here and make yourself comfortable.

      I hope you don‟t mind if I use this script during the session. It will help
      us ensure that everyone in this study gets the same information.

      We will be conducting a usability test where we will observe you
      interacting with the Metra website while we ask you to get some
      specific information. This will help us learn how the website supports its
      primary functions and whether it is easy to use.

      Let me ask you, have you ever taken a Metra train? Have you ever
      visited their website?

      [If the user was taking a Metra and used their website ask the
      following]:
      What line (or train route) did you take?

      OK. It‟s good to know.

      Before we can start, we will need your consent to participate in this
      session. We have a consent form that we would like you to read and let
      us know if you agree with what it says.

      ** present the form **

      [If the user disagrees] Thank you for your time. Have a great
      day/afternoon/evening!

      [Else]

      I would also like to let you know that during this session we will be
      taking notes that will help us keep track of your interaction with the
      website. We will also measure time used to perform specified tasks. But
      please do not be alarmed or feel like you have to rush. This will help us
      determine how well the website is designed and how easy it is to use.

                                          - 24 -
If at any time during this session you feel that you want to stop, just let
us know and you will be free to leave.

OK. So, let me give you a quick overview of the nature of this study. The
Metra‟s website is a little outdated. They are looking to improve their
website to help the current and potential Metra riders learn more about
the convenience of public transportation. We tried to determine what
areas of the website could be most useful to people interested in taking
Metra and how easily those areas can be accessed.
We created a list of five tasks that we would like you to perform. It that
OK?

Please do not feel you‟re under pressure to perform well. We are
testing the website and not you, so if you have any difficulties along the
way, it will help us identify problem areas with the website itself.

When you feel that you completed the given task, please let me know
by saying “I‟m done.” I will then present you with another task.

Are you ready?

[The Metrarail.com web page will be preloaded]

OK.

[Task 1 – If the user already has taken the Milwaukee District West
line, change the departure station from any other line that the user
did not take]
So, for our first task let‟s assume you live in [Schaumburg] and you are
planning to go downtown Chicago this upcoming Saturday. You plan to
take Metra train and you need to arrive downtown before 1 PM. OK?
Given that, I would like you to find a schedule for the train going from
[Schaumburg] to downtown Chicago on a Saturday and arriving
downtown before 1 PM. Does that make sense?
[If „No‟, explain or reword the task]

[If „Yes‟]
OK. You can begin.

[If the user notices the various navigation menus and different options
to get to the schedule and asks which one s/he should use, say]:

It doesn‟t matter. Whichever way you feel is more comfortable or
natural to you.

[If the user asks for advice or confirmation of his/her actions, say]:


                                    - 25 -
We just need your opinion on how the website works, so please do
whatever you feel is appropriate (or let us know if you don‟t think the
right option is available). OK?

[If the user cannot find the correct rail line after 2 minutes, point to the
“system map”]

** When the user indicates that the task is completed **

OK

[Task 2]
Given the same scenario as in the previous task (you live in
Schaumburg and want to take Metra downtown Chicago), your next
task is to find the fare for a train going from Schaumburg to downtown
Chicago.

[If the user doesn‟t remember the Metra rail name and is stuck at this
point for longer than 2 minutes, direct to the “System Map”]

** When the user indicates that the task is completed **

OK

[Task 3]
 For our next task, provided the same scenario as in the two previous
tasks, we are assuming that now you know your schedule and the fare
and would like to purchase a train ticket online. So, this is your task – to
purchase a train ticket by Internet. Just so you know, I don‟t want you to
actually purchase anything, but just act like you were to buy a ticket.
You do not need to finalize the transaction or provide any confidential
information. You can enter fictional information when required. OK?

[If necessary, point the user to “Application Instruction” section]

** When the user indicates that the task is completed **

OK

[Task 4]
For this task, let‟s assume you want to find out where the Schaumburg
station is located and whether there is a telephone number available in
case you need to call and get more information. So, your next task is to
find the address and telephone number for the [Schaumburg] station.

** When the user indicates that the task is completed **
OK

                                    - 26 -
[Task 5]
For this final task, let‟s assume that the purpose of your trip downtown
is to go the Field Museum to see an exciting new exhibit this Saturday.
So, we know that you will take Metra and will arrive downtown around
1PM. But, now you want to find your way around the loop. You want to
know if there are other transportation options that will take you to your
desired destinations. You‟re curious whether the Metra website
provides that additional information.
So, your final task is to find if the Metra website provides additional
information on other forms of public transportation that will help you
find your way from the Union Station to the Field Museum.

** When the user indicates that the task is completed **

OK. Thank you for completing the tasks.

Now, I would like you to complete a questionnaire that will give us
more feedback on your overall impression of the website. Is that OK?
We appreciate that, thank you!

** After the questionnaire is completed initiate the debriefing session.
You can start by saying: **

We can now have a more informal follow up discussion of the session.
Do you have any questions you‟d like ask or any concerns? …




                                   - 27 -
Post Test Questionaire: Usability Testing on Metrarail.com

Please fill out the following questions. Take as long as you need.

Name:_______________________________________

During the usability test, we studied several interactions that are frequently
attempted on the metrarail.com site. We would like your feedback on those
interactions. Please circle one response to each statement below.

Content

   1. I found the content to be clear and concise.
      Strongly Agree   Agree   Slightly Agree   Slightly Disagree   Disagree   Strongly Disagree
            1           2            3                    4            5              6

   2. I found the content to be organized in an effective manner.
      Strongly Agree   Agree   Slightly Agree   Slightly Disagree   Disagree   Strongly Disagree
            1           2            3                    4            5              6

   3. I found Metrarail.com to be helpful and convenient.
      Strongly Agree   Agree   Slightly Agree   Slightly Disagree   Disagree   Strongly Disagree
            1           2            3                    4            5              6

   4. I expected that I could purchase a single roundtrip fare online.
      Strongly Agree   Agree   Slightly Agree   Slightly Disagree   Disagree   Strongly Disagree
            1           2            3                    4            5              6

   5. I felt that I could easily continue my trip to the Field Museum.
      Strongly Agree   Agree   Slightly Agree   Slightly Disagree   Disagree   Strongly Disagree
            1           2            3                    4            5              6

   6. I felt that I could easily continue my trip to other destinations in
      Chicago.
      Strongly Agree   Agree   Slightly Agree   Slightly Disagree   Disagree   Strongly Disagree
            1           2            3                    4            5              6




                                                 - 28 -
Navigation

   1. I felt that the time it took me to complete the tasks felt appropriate.
      Strongly Agree   Agree   Slightly Agree   Slightly Disagree   Disagree   Strongly Disagree
            1           2            3                    4            5              6

   2. I found that navigating the steps to complete an online ticket purchase
      was easy.
      Strongly Agree   Agree   Slightly Agree   Slightly Disagree   Disagree   Strongly Disagree
            1           2            3                    4            5              6

   3. I would purchase a ticket online for future trips on Metra trains.
      Strongly Agree   Agree   Slightly Agree   Slightly Disagree   Disagree   Strongly Disagree
            1           2            3                    4            5              6



General Impressions:
Please give any extra comments or suggestions here.




                                                 - 29 -
[Checklists]

__ Computer or laptop is connected and turned on

__ Internet connection is established and tested

__ A timer or a watch with timer prepared

__ Printed copies of the test script

__ Printed copies of the Informed Consent form

__ Printed copies of the Log Sheet

__ Participant greeted and introduced

__ Informed Consent form presented to and signed by the participant

__ Orientation

__ Prepare to test

__ Record the start time for each task, observe the participant and collect the
   important information (complete data log)

__ Post-test questionnaire completed by the participant

__ Debrief the participant

__ Thank and release the participant

__ Organize notes and data logs

__ Prepare for the next participant




                                         - 30 -
Log Sheet: Usability Testing on Metrarail.com


User:_______________________________________________________

Metra User Level Response: ________________________________




                Succed?
    Task                     Time                        Comments and Observations
                  Y/N




 Locate Train
  Schedule


                          (target 3min)




 Determine
 Train Fare


                          (target 3min)




  Purchase
                             N/A
Ticket Online




   Locate
 Address &
   Phone
  Number
                          (target 2min)



 Determine
  route to
Museum from
   Station
                          (target 1min)




                                                - 31 -
                                Consent Form


Purpose of study
You are invited to participate in a usability test of the MetraRail.com website
for a school project. We hope to learn and define the difficulties or the
problems that a user might face with the web site. You were selected as a
possible participant in this study because you have accessibility to the
internet and you are in our age range of a possible Metrarail rider.

Description of Study and Risks
If you decide to participate, we will ask you to perform 5-8 tasks on the
metrarail.com website and ask you a few questions afterwards. We will have
a note taker to record your actions while you perform the tasks.

The test should take about 15 minutes depending on the rate of your speed to
finish each task. Some tasks might be more difficult than others, so it is
expected that you might run into a problem. You will be timed, but you are
allowed to take your time to complete each task.

If you feel discomfort during the test, you are allowed to stop but you cannot
ask us for help to complete the task.

Confidentiality and Disclosure of Information
Any information that is obtained in connection with this study and that can be
identified with you will remain confidential and will be disclosed only with
your permission or except as required by law. If you give us your permission
by signing this document, we plan to discuss or publish the results only to HCI
460 for School of CTI.

Your Consent
Your decision whether or not to participate will not prejudice your future
relations with The University of DePaul and school of CTI. If you decide to
participate, you are free to withdraw your consent and to discontinue
participation at any time without prejudice.

If you have any questions, please feel free to ask us. If you have any
additional questions later, Rebecca, Joanne, and Anna will be happy to answer
them.

You will be given a copy of this form to keep.


                         THE UNIVERSITY OF DePaul
                School of CTI for Human Computer Interaction


                                         - 32 -
  PARTICIPANT INFORMATION STATEMENT AND CONSENT FORM
                        (continued)
                      (Title of project)



You are making a decision whether or not to participate. Your signature
indicates that, having read the information provided above, you have
decided to participate.




……………………………………………………
.…………………………………………………….
Signature            of Research                               Participant
Signature of Witness



……………………………………………………
.…………………………………………………….
 (Please PRINT name)                           (Please PRINT name)



……………………………………………………
.…………………………………………………….
Date                   Nature of Witness



……………………………………………………
Signature(s) of Investigator(s)



.…………………………………………………….
Please PRINT Name




                                    - 33 -
INDIVIDUAL CONTRIBUTIONS

Becky Jones - email: bjones@depaul.edu
Test role: Data logger and note taker. Provided and set up testing room.
Report: Provided the Results section, Questionnaires and Blank and
completed log sheet.

Joanne Keung - email: joanne.keung@comcast.net
Test role: Timer and note taker.
Report: Provided Findings and Recommendations sections and the Consent
Form.

Anna Mikrut - email: annamikrut@hotmail.com
Test role: Moderator
Report: Provided Executive Summary and revised Method sections, Scripts,
and Checklists. Assisted in coming up with Test Objectives. Assisted in
organizing the entire document for submission.




                                        - 34 -

				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:13
posted:7/13/2011
language:English
pages:34