Heuristic Evaluation and Simple Usability Study
Name: Ellena Cheng
Date: September 30, 2004
1
Table of Contents
Section 1: Scenario ................................................................................................................... 3
Section 2: Methodology ........................................................................................................... 3
Heuristic Evaluation .......................................................................................................... 3
H2-1 Visibility of system status ................................................................................. 3
H2-3 User control and freedom ................................................................................. 3
H2-4 Consistency and standards ................................................................................ 3
H2-5 Error prevention ................................................................................................ 4
H2-6 Recognition rather than recall ........................................................................... 4
H2-7 Flexibility and efficiency of use ....................................................................... 4
H2-8 Aesthetic and minimalist design ....................................................................... 4
H2-9 Help users recognize, diagnose, and recover from error .................................. 5
H2-10 Help and documentation ................................................................................. 5
Task Evaluation .................................................................................................................. 5
Task 1. Book a train ticket. ........................................................................................ 5
Task 2. Finding a cheap train ticket deal. ................................................................... 5
Task 3. Reserve train and hotel together. ................................................................... 6
Task 4. A student wishes to buy a ticket using the student discount. ......................... 6
Task 5. Look at pictures or videos of the train. .......................................................... 6
Task 6. Browse for a suitable deal. ............................................................................ 6
Task 7. Book tickets for a family. .............................................................................. 7
Task 8. Cancel a reservation. ..................................................................................... 7
Section 3: Observations ........................................................................................................... 7
Section 4: Interpretation: System Strengths and Weaknesses ............................................. 7
Strengths ............................................................................................................................ 7
Weaknesses ........................................................................................................................ 8
Section 5: Suggested Improvements ....................................................................................... 8
Section 6: Conclusion ............................................................................................................... 8
Appendix 1: Comparison of Different Techniques ............................................................... 9
Appendix 2: Raw Data............................................................................................................. 9
2
Section 1: Scenario
The system is the VIARail website that allows user to obtain information and schedule
trips with VIARail. My role of evaluation is to evaluate how young people will accomplish
various tasks using their website. Online business‟ success depends a lot on how easy it is to
use their site, therefore this usability testing will gain insight to its current problems and then
improvements could be made for problematic features.
Section 2: Methodology
Heuristic Evaluation
H2-1 Visibility of system status
[Severity 1]: Book, Schedule, Fares flashes, user cannot be sure about where they are because
there is the border of the colour of the page they‟re on and the flashing icon.
Solution: Stop the flash with that unique tab, and make it consistent with the other ones
[Severity 2]: When in English site, it says „Francais‟ on the navigation bar on top.
Solution: Show the current language at the bar, and show the rest in the drop down list.
[Severity 2]: Doesn‟t tell user where in the website they are under.
Solution: Give a path of the hierarchy of how to get to the site at the top of the page.
H2-3 User control and freedom
[Severity 2]: User can‟t choose more than a 6 passengers for a trip.
Solution: Allow the user to enter a number instead of drop down list.
[Severity 4]: User can‟t return to the beginning quickly.
Solution: Add a “home” link in the navigation bar
H2-4 Consistency and standards
[Severity 2]: Different looking websites in different languages
Solution: Change other sites using the same theme, even if the content isn‟t identical.
[Severity 4]: Navigation bars are gone as soon as the user goes into lower hierarchy (eg:
selected schedule).
Solution: Add navigation bars.
[Severity 1]: Colour changes under different sections.
Solution: Use same colours.
3
[Severity 1]: “Book, Schedule, Fare” tab is different than the rest.
Solution: make it consistent with other tabs.
[Severity 2]: Two navigation bars, without clear distinction of what belongs to which.
Solution: Make it into one navigation bar.
[Severity 3]: “Fares and reservations” and “schedules” are actually the same thing, may
confuse the user.
Solution: Make it “Schedules, Fares and Reservations” instead.
[Severity 2]: Some logos go back to English site, some goes back to viarail.ca which needs to
choose English again.
Solution: Make it consistently going back to English site.
H2-5 Error prevention
[Severity 2]: Allows user to make mistake to pick dates that don‟t exist (eg. Sept 31).
Solution: Block out non-existing choices as soon as user inputs partial data.
[Severity 1]: Doesn‟t tell user immediately that the return date is before the departure date.
Solution: Do the check right when the user chooses the date instead of after submit
[Severity 1]: Allowed to choose dates without trips.
Solution: Can show a list of available dates instead of just closest date.
H2-6 Recognition rather than recall
[Severity 3]: Ask the user for exact information rather than let them browse available options.
Solution: Make another section where the user can browse based on price, dates, or
locations.
H2-7 Flexibility and efficiency of use
[Severity 3]: User must go through the whole process of searching for the trip every time.
Solution: Save the user‟s choices by cookies.
[Severity 4]: Navigation bars are gone as soon as the user selects schedules, forces the user to
click on back buttons.
Solution: Universal navigation bar.
H2-8 Aesthetic and minimalist design
4
[Severity 2]: The navigation bar at the left has too much information.
Solution: Current expanded section can have more spacing and clearer between each link.
H2-9 Help users recognize, diagnose, and recover from error
[Severity 3]: Displays message and ask user to call hotline if no trip matches criteria.
Solution: List everything in the destination, departure locations.
H2-10 Help and documentation
[Severity 4]: Help and search not visible universally.
Solution: Make them available in a universally visible navigation bar.
Task Evaluation
Number of subjects: 4
[Condition 1 - Silent Observer]: (Within Subject) Novice university student (have never
made an online transaction): Task 1, 3, 2, 4
[Condition 2 - Think aloud method]: (Within Subject) Semi-knowledgeable Computer
Science student: Task 3, 4, 1, 2
[Condition 3 - Constructive Interaction]: Novice middle-aged user, semi-knowledgeable
Computer Science student: Task 5, 6, 7, 8
Task 1. Book a train ticket.
A friend from France is visiting you, and would like to see more of Canada than just Toronto.
In particular, she would like to visit Quebec City, and would prefer to travel by train. She has
asked you to book her train ticket. She would like to leave Toronto's Union Station on 4
October 2004, stay in Quebec City for 4 nights, and then return directly to Toronto. She
would prefer the shortest duration train journey even if it is a little more expensive. She hasn't
quite specified what "little more expensive" means. Try and make this booking for her.
Reason for choosing this task. This is a reasonably typical task, but where not all the exact
details are given apriori.
Task 2. Finding a cheap train ticket deal.
You and a friend want to check the costs of a weekend trip to Chibougamau, Quebec (Fri
evening to Mon morning) from Toronto. You would like to make this trip sometime in the
Fall of 2004, before it gets typically starts snowing in Chibougamau. In part, the exact dates
will be influenced by the cost of the train ticket, as well as any additional transportation
should trains not actually go all the way to Chibougamau (i.e., you may have to take a train to
a nearby city and take a connector bus).
5
Reason for choosing this task. This is a more complex, but reasonable request. The customer
knows their travel constraints, and the required answers could easily be obtained if you called
Via Rail and spoke to a (competent) human customer service representative. Thus, this tests
whether the website provides similar levels of service as the company's presumably
competent human representatives
Task 3. Reserve train and hotel together.
A user wishes to have a round trip from Toronto‟s Union Station to Winnipeg. The user
wishes to reserve a hotel room there for at least 3 nights, and less than a week. He wishes to
have a comfortable trip, and price is not an important factor.
Reason for choosing this task. Most people needs to reserve hotel rooms while traveling, and
a lot of hotels have packages made for travelers so that it is cheaper for them.
Task 4. A student wishes to buy a ticket from Vancouver, British Colombia to Halifax,
Nova Scotia, using the student discount.
A University of Toronto student tries to reserve a spot for the Christmas Break for a 7 days
trip. The trip should be anytime from Dec. 21st to Jan. 3rd.
Reason for choosing this task. This should be a reasonably easy task, many University
students have experience doing transactions online. How easy the discounts could be used
will be important.
Task 5. Look at pictures or videos of the train so that they can imagine how the trip will
be like.
User wishes to look at the train that she will be on from Toronto to Montreal, and any
additional information that may give her more information about her trip.
Reason for choosing this task. This should be a simple task for those that are interested in
knowing more about the trains.
Task 6. Browse for a suitable deal.
Two friends has 2 weeks of holiday from Oct. 7th to Oct. 17th, she wants to go traveling by
train, but doesn‟t know where she wishes to go yet. Their budget is around $500, and tries to
maximize the time out of time in her holiday.
Reason for choosing this task. Many people don‟t actually know where they want to go, so
6
they would like to browse for their choices. They don‟t really have a fixed schedule,
destination or budget, so it depends on what the company has to offer and how easy it is to
obtain the information.
Task 7. Book tickets for a family.
A family of 5 with Father, Mother, 21 year old, 9 years old, and 6 months old from Toronto to
Montreal. The parents wish to find the prices for their children, and see how much discounts
they will have off. In specific, they wish to know if the infant will go for free or not.
Reason for choosing this task. Many people travel with the whole family, and many of them
have children. The different rates for children may confuse the user.
Task 8. Cancel a reservation.
User has decided that the trip planned in Task 1 was not appropriate and wishes to cancel
online. The user wishes to check before hand how if there is any penalty if he/she cancels the
trip, and if it will cost too much.
Reason for choosing this task. It is typical that someone‟s schedule changes, and when it
changes, the user may wish to see if there is a penalty. Or if someone else wishes to go
instead of the original user, they wish to see how difficult it is, if possible, to transfer the
ticket.
Section 3: Observations
Users generally found it looks professional and not too difficult to use
Users were pleased with its aesthetic appeal
One third of the problems come from consistency and standards, confuse users.
There is a whole range of severity; therefore the site needs to be fixed at every level.
The hierarchy of the system is not obvious to the user.
Assumed users had all the information about the trip they wish to have
Section 4: Interpretation: System Strengths and Weaknesses
Strengths
The scheduling on the main page was convenient and first thing that the users try.
The search is quite useful, except that it is not always available.
Paying method is generally easy to use.
Always easy to access the schedule which leads the user to make a reservation.
They have many promotions, but homepage not overwhelmed by advertisements.
7
Weaknesses
There were different ways that the user can reserve hotel and ticket, therefore it is
confusing for the users. Also difficult to understand which package is being sought after.
Many problems when user doesn‟t have all the specific information about their trip.
Both users had a lot of trouble to find other transportations if the destination is not
directly on the railway‟s route.
If user clicked on a wrong link, it is more difficult for them to trace their way back and
start again since they don‟t know where they are relative to other pages.
Should give user more choices of searching for their preferred trip, since the easier it is
for the user, the more probability they will use the service.
Section 5: Suggested Improvements
1) The information should be more centralized. There are two navigation bars, and much
information cannot be obtained until the user is a few levels below the homepage.
Organize the navigation bars again to avoid repetition and add in important sections to the
navigation bars (eg: trainpackages.ca, Student discounts).
2) Ensure important links are always available for user, such as search, home etc. If the
navigation bar is visible at all times, then these links could be added into the navigation
bars. This will also allow easier browsing for the user, since they can jump from section
to section directly.
3) The user should always know where they are in the homepage. The current site has a low
visibility of system status. A link at the top of the content page referring to each level
before the content page will be very useful.
4) Allow the user to make more recognition when scheduling instead of filling in exact
information from the beginning. Giving the user a list of available trips according to one
or two criteria that the user entered (eg. Date between Dec. 1st to Dec. 31st).
5) There should be consistency in the website. Make centralized guidelines for the website
designers so that the information will be organized and presented in a systematic way.
Includes both information and aesthetic appeal consistency.
Section 6: Conclusion
The site looks very professional at first glance, however, it requires much more work on
usability if VIARail wants the customers to be happy after they have tried using it. The site is
geared towards convincing the consumer to make a reservation, and puts less effort in
organizing and presenting other information which makes it easy for the consumer to make a
decision.
8
Appendix 1: Comparison of Different Techniques
I found Condition 2 to be the most useful. In Condition 3, the two users do not seem to
want to communicate, and semi-knowledgeable participant was simply asking the novice to
physically move to mouse, even after instruction that their thought process has to be more
explicit. The novice did not want to contribute, and was relying heavily on the
semi-knowledgeable user. Condition 1 did not give as much information because I cannot tell
what is going on in the user‟s thought process. While Condition 2 allowed me to know what
the user is thinking, without the instructions to move the mouse since the user ha done that
internally. However, it may be better if the novice was in Condition 2 instead of the
semi-knowledgeable user because the Computer Science student seemed to have taken many
steps for granted already and does not remember to say it out loud because it is not a unique
event anymore. The novice may give more surprising insights to the system‟s problem since
an online transaction is a new experience. Condition 3 was the least useful in this study
because it generated around the same amount of information but required two different
participants. However, if the participant was willing to explain everything explicitly and skip
the steps for directly asking the user to move the mouse over to a certain point, then it has the
potential to generate much more information than the other two conditions.
Appendix 2: Raw Data
An evaluation was run based on Jakob Nielsen‟s second set of heuristic guidelines.
Heuristic # Heuristic # of Violations
2-1 Visibility of system status 3
2-2 Match b/t system and real world 0
2-3 User control and freedom 2
2-4 Consistency and standards 7
2-5 Error prevention 3
2-6 Recognition rather than recall 1
2-7 Flexibility and efficiency of use 2
2-8 Aesthetic and minimalist design 1
2-9 Help users recognize, diagnose, and recover 1
from error
2-10 Help and documentation 1
Total 21
Level Severity Violations
0 Don’t agree that this is a usability problem 0
9
1 Cosmetic problem 5
2 Minor usability problem 8
3 Major usability problem; important to fix 4
4 Usability catastrophe; imperative to fix 4
10