Runnels—Usability Study 1
Usability Study of Cultural Awareness Websites
Kurstin Blue Runnels
EDC385G: Designs and Strategies for New Media
Professor Min Liu
April 7, 2007
Runnels—Usability Study 2
Table of Contents
Introduction 3
Methodology
Site Selection 4
User Selection 4
Test Procedure 5
Data and Results #
Pre-Test Questionnaire #
Usability Test Results #
Post-Test Questionnaire #
Analysis #
Not Just Sushi #
Africa for Kids #
Race: The Power of Illusion #
Discussion #
Appendix A: Usability Test #
Appendix B: Pre-Test Questionnaire #
Appendix C: Post-Test Questionnaire #
References #
Runnels—Usability Study 3
Usability Study of Cultural Awareness Websites
Introduction
The middle school social studies curriculum is so extensive that teachers rarely
have time to touch on topics pertaining to cultural awareness. One of the social studies
teachers at my school described her curriculum as jumping from one tip of an iceberg to
another, teaching only the basics in a very narrow framework. The standardized
curriculum leaves no time to explore concepts profoundly, nor does it allow time for
students to truly understand cultures other than their own. In other words, a middle
school social studies student may be able to identify Kenya as an African country, but he
would not be able to explain what daily life is like in Kenya.
As educators, we know that providing ways for students to connect to the
curriculum personally increases learning and comprehension, but the time it would take
to teach lessons like this would mean sacrificing other content that is needed to prepare
students for looming standardized tests. Thankfully, many websites are being created to
assist teachers in providing more of an unrestrained exploration of other cultures. What
would take teachers several weeks to cover in the classroom can be quickly yet
thoroughly covered in one or two class periods using websites. Not only would the
students become more culturally aware, but they would also find the experience more
enjoyable than, say, reading about various cultures in a textbook.
Of course, one must consider whether the sites available are user-friendly. Sites
intended to increase cultural awareness need to be interesting, educational, fun, and easy
to use by both students and teachers. If websites are difficult to use, students and teachers
will become frustrated and learning will be diminished. By conducting usability tests on
Runnels—Usability Study 4
cultural awareness websites, designers can find out what works and what does not, make
improvements on already existing sites, and discover how teachers and students are
actually using the sites. This research study focuses on three cultural awareness websites
in an effort to find out how well they meet the needs of both students and teachers and to
determine whether these sites are a useful educational tool in social studies classrooms;
therefore, the study findings will benefit both users and designers
Methodology
This usability study is guided by the methodology used by both the Nielsen
Normal Group (Nielson et al., 2000) and Rubin (1994). Like the Nielsen Norman Group
example, this study was exploratory in that I wanted to learn what makes cultural
awareness sites successful in the eyes of students and teachers in terms of educational
value.
Site Selection
For this usability research study, I chose three websites that are used by the sixth-
grade social studies teachers at my school: Not Just Sushi, Africa for Kids, and Race: The
Power of an Illusion. The Not Just Sushi site has been used for three years by the social
studies teachers, the Africa for Kids site was used for the first time this year, and the Race
site will be used at the end of this year for the first time. Though their focuses may be
different, each of the three sites seeks to enhance cultural awareness by educating users
about different cultures.
The Not Just Sushi site (Figure 1) was developed by students at the University of
Texas at Austin and can be found at http://www.edb.utexas.edu/japanese/index.php.
Runnels—Usability Study 5
Figure 1: Not Just Sushi
The purpose of the site is to introduce students to Japanese culture through food. The site
focuses on the “history, geography, nutrition, and ethnography of Japanese food” (Not
Just Sushi, 2004). Students are supposed to take on the role of a geographer, historian,
culinary specialist, or an ethnographer while completing a web quest using the sites
resources.
The Africa for Kids (Figure 2) site, which can be found at http://pbskids.org/
africa/index.html, is produced by Thirteen and distributed by the Public Broadcasting
System. The purpose of the site is to introduce students to African culture by exploring
daily life, music, art, and literature. Students can view photo albums and accompanying
text created by African students, learn to play the thumb piano, make African masks, and
read or listen to an African folktale while taking an interactive quiz.
Runnels—Usability Study 6
Figure 2: Africa for Kids
The Race (Figure 3) site is also distributed by the Public Broadcasting System and
can be found at http://www.pbs.org/race/000_General/000_00-Home.htm.
Figure 3: Race: The Power of an Illusion
The goal of the site, which was produced by California Newsreel, is to inform students
“about race in society, science and history” (Race, n.d.). Students can discover the
Runnels—Usability Study 7
significance of race through several disciplines, such as language arts, history, science,
sociology, and psychology. The site includes an enormous amount of information about
race and human diversity and their implications in society.
User Selection
Because these cultural awareness sites will be used by both students and teachers,
I selected individuals from both populations to participate in my usability study. Since I
did not want to test the sites using an individual who was familiar with them for fear prior
knowledge would affect the results, I chose a seventh-grade social studies teacher and a
sixth-grade student from a class that has not used the sites to participate in the study.
Neither of these users had ever used any of the three sites. In an effort to rank the sites in
terms of usability, I chose to use the same student and teacher for all three usability tests.
Test Procedure
Before the test, users filled out a pre-test questionnaire to determine
demographics, computer skill, and Internet experience (Appendix B). The questionnaire
was mixture between the Rubin (1994) and Nielsen, et al. (2000) examples, and I used it
to gather demographic information, to determine how experienced the users were with
computers and the Internet, and to find out what types of sites the users frequent
regularly.
During the test, the users completed four tasks: free exploration, specific
(information), open-ended, and specific (interaction). The free exploration task consisted
of the users familiarizing themselves with the site and being free to click on anything
they want for up to three minutes. I hoped to determine what elements of the site pique
user interest by observing the users during the free exploration. The second task was
specific in that it asked the user to find definite information, such as a definition or date.
Runnels—Usability Study 8
The third task was open-ended and asked users to explore an area of the site and discover
something. The final task involved specific interaction and asked users to complete a task
in an interactive section of the website. After the test, users filled out a post-test
questionnaire, based on the Site Feedback Questionnaire used by Nielsen, et al. (2000),
for each site they visited to gauge their level of user satisfaction (Appendix C). I modified
the content of the questionnaire to fit the websites used in my study.
I specifically chose to give the same tasks to both the student and teacher because
teachers will base their impressions of a site on the educational value for students
foremost rather than the materials available to teachers. From my experience as a teacher,
I have found that if a site is not deemed valuable for students by teachers, the teachers
will never even explore the teacher resources available on a site. Therefore, rather than
creating separate, incomparable tasks for the student and teacher, I chose to give both of
them the same tasks.
During the testing, I observed the users and took copious notes to document how
both the student and teacher were using the sites. I asked the users to “think out loud”
while making selections so that I could note any questions or concerns they had in
completing their tasks. I also kept track of the time it took the users to complete their
tasks and documented any places within the sites where the users became stuck, lost, or
frustrated.
Data and Results
Pre-Test Questionnaire Results
Dylan is a sixth-grade regular education student of Mexican descent who is
twelve years old. He has been using a personal computer for four years. He typically uses
Runnels—Usability Study 9
the computer approximately two hours per day at home and spends most of that time on
the Internet. He uses the Internet for social interactions with his friends, most often
visiting My Space (www.myspace.com).
Toni is a seventh-grade social studies teacher who is fifty-five years old and is
white. She holds a bachelor’s degree and has pursued some graduate education. She has
been using personal computers for eighteen years. She typically uses the computer
approximately five hours per day both at home and at work. She primarily uses the
computer to generate teaching materials, communicate with friends and family through
email, and surf the Internet. Toni mainly accesses the Internet at home to find lesson plan
ideas, gather information, or to shop. She listed Awesome Library (www.awesomelibrary
.org/social.html) and National Public Radio (www.npr.org) as the top two sites she
frequents.
Usability Test Results
The usability test results were surprising in that the student and teacher results
presented a dichotomy. Whereas the student may have found an element entertaining and
rated it high, the teacher may have found that same element distracting from the purpose
of the site. Additionally, while the teacher was pleased to see a large amount of
educational information on a screen, the student may have been “turned off” by it. Thus,
the results for the sites were very different in terms of user demographics and their roles.
Both the student and teacher users were able to complete all tasks for each site in
the relatively same amount of time. However, some tasks were considered failures
because they required substantive assistance for them to be completed. Both the student
Runnels—Usability Study 10
and teacher experienced the same task failures, which indicate a flaw in design not
relative to user demographics or roles.
Post-Test Questionnaire Results
According to combined user responses (Chart 1), the Race: The Power of an
Illusion website was the best site of the three tested. Both the student and teacher rated
their satisfaction with quality of information, ease of reading text, appearance of site,
speed of pages displaying, and educational value very high. All results were in the “very
satisfied” category; however, getting to the right part of the site, entertainment value, and
overall experience were less satisfying for the student than the teacher but were still in
the “very satisfied” range.
Post-Test Questionnaire Results
6.0
5.0
4.0
3.0
2.0
1.0
0.0
Not Just Sushi Africa for Kids Race: The Power of
an Illusion
Chart 1: Users’ Post-Test Questionnaire Results
Runnels—Usability Study 11
The Africa for Kids site also produced similar results on the post-test
questionnaire and was the second-highest rated site in the study. Both users were “very
satisfied” with the site, but where several threes were awarded to the Race site, the
highest score for the Africa site was a two. Both the student and teacher rated their
satisfaction with getting to the right part of the site, quality of information, ease of
reading text, and entertainment value the same. Appearance, educational value, and
overall experience were rated less satisfying by the student but were still in the “very
satisfied” range.
The Not Just Sushi site produced disparate results. The majority of the teacher
ratings fell into the “very unsatisfied” category while most of the student ratings fell into
the “very satisfied” category. Items that produced the greatest amount of discrepancy
were getting to the right part of the site, quality of information, entertainment value,
educational value, and overall experience. Not Just Sushi scored the lowest of the three
sites overall.
Analysis
The results of this usability study were analyzed from the design standpoint, using
the three types of design (information, interface, and interaction) as an evaluation tool.
By looking at the usability test results through a design lens, areas that need development
are emphasized. Based on the results, designers can make improvements to the sites and
avoid design issues that cause task failures.
Information design consists of organizing information so that it is concise,
scannable, and objective (Morkes and Nielsen, 1998). Information should be presented in
a visual hierarchy and organized in such a way that it is easily located and relationships
Runnels—Usability Study 12
are obvious (Krug, 2006). Interface design refers to how information is presented on a
site. According to Krug, the interface gives users a sense of scale, location, and direction.
The interface also refers to the aesthetics of the site or, rather, the way the site looks, such
as color scheme and graphics. The most important element of interface design is
navigation. The way a site looks tells the user how to get around and finally interact with
the site. Interaction design involves the ways in which users interact with, or use, a site.
Interaction design can be as simple as the use of buttons, toolbars, links, menus, and
action panels or as complex as using a web-based application to create a document.
Users were asked to rate the three sites in terms of information, interface, and
interaction design on a scale from 1 to 10, with 10 being a perfect score. The results are
depicted in Charts 2 through 4 below.
Not Just Sushi
10
8
6 Student
4 Teacher
2
0
Information Interface Interaction
Chart 2: Not Just Sushi User Rating
Runnels—Usability Study 13
Africa for Kids
10
8
6 Student
4 Teacher
2
0
Information Interface Interaction
Chart 3: Africa for Kids User Rating
Race: The Power of an Illusion
10
8
6 Student
4 Teacher
2
0
Information Interface Interaction
Chart 4: Race: The Power of an Illusion User Rating
Combined results (Chart 5) reveal that the Race site scored the highest in all three design
categories: information, interface, and interaction. The Sushi site scored higher than the
Africa site for information design but scored far below in terms of interface and
interaction.
Runnels—Usability Study 14
Combined Results
10
9
8
7
6 Information
5 Interface
4 Interaction
3
2
1
0
Sushi Africa Race
Chart 5: Combined User Rating Results
The user ratings corroborate the post-test questionnaire results and support the finding
that the Race site is the most user-friendly of the three. What follows are the particular
results for each site.
Not Just Sushi
Not Just Sushi is a site developed by the University of Texas at Austin’s Center
for East Asian Studies, which seeks “to promote research related to East Asian cultures
and societies” (2007), in conjunctions with the University’s Instructional Technology
Program whose goal is to enhance learning through technological tools (2005). The site
was designed for sixth-grade social studies students to familiarize them with Japanese
culture through an exploration of Japanese cuisine.
Information. Although the Sushi site received the lowest score of the three for
information, the score was still relatively high. Both users commented that the
information was interesting and indicated the quality of the information was very
Runnels—Usability Study 15
satisfying. However, the student commented that there was too much information,
whereas the teacher was impressed with how much information was contained on the site.
Regardless, the teacher also made a remark that she would never have enough class time
to use the site to its full potential and that, if the purpose is for students to use the site, the
information is too dense. She also stated that the information was “quite dated” and
needed “fresh” information.
Interface. Based on the results of the usability study, navigation was the primary
design flaw of the Sushi site. When asked to find out what miso is, both users
experienced task failure because they had to be directed not once, but twice, to find the
answer. When presented with the start page (Figure 4), users were confused by their
choices and did not notice or else consider the text above the map that read, “Click on
Japan to begin.”
Figure 4: Explore Tokyo Page
The student user clicked on “Explore Tokyo,” not realizing that he was already on that
page. Neither the breadcrumbs nor the page heading seemed to indicate to him that he
Runnels—Usability Study 16
was already where he was supposed to be. Next, he clicked on the instructions (“Click on
Japan to begin”) instead of on the map where Japan is located. When nothing happened,
he just sat there studying the map for approximately 30 seconds until he asked me what
he was supposed to do. I told him to click on Japan to begin, and he started moving the
mouse all over the map, rather quickly, and then complained that the map was not doing
anything. Finally, I had to point out Japan so he could move his mouse to the country and
click.
The teacher experienced the same navigation problem because, instead of clicking
on Japan, she clicked on “Learner’s Roles,” which took her to a page describing the roles
of the geographer, historian, culinary specialist, and ethnographer. She systematically
clicked on each title, read all of the questions, scrolled up and clicked on the next title,
and so forth. When she did not find her answer on the “Learner’s Roles” page, she went
to the “Teacher’s Guide” and begin to read through that page as well. For time’s sake, I
had to direct her back to the “Explore Tokyo” page and point out the directions. She had
no trouble finding Japan. The teacher commented that there was no link on the other
pages to the “Explore Tokyo” page. Users have to go back to the home page and re-
navigate to the map.
The second time the users had to be directed was on the Suzuki Residence page.
The users served the family breakfast then tried to click on all of the items on the table,
which did nothing. They also clicked on the bird and the people, which produced no
results. Both users commented that they wanted to be able to click on more objects in the
picture. The student accidentally clicked on the door and was taken back to the Tokyo
page where he had to reenter the Suzuki Residence. The teacher indicated she would have
Runnels—Usability Study 17
liked to see some text accompanying the dishes on the breakfast table like there was on
the dinner table. “The lack of consistency is frustrating,” she commented. Because
neither user even tried to click on the TV screen, I had to direct them again by pointing it
out.
While completing the task of finding out how sushi is made, the teacher
remarked, “There is too much going on” with the plates of food circling the chef, the TV
screen, and the menu (Figure 5).
Figure 5: Sushi Page
The student wanted to be able to interact with the plates of food but could not, and it took
both users approximately one minute of exploring the page to figure out they could click
on the two sushi rolls in front of the chef.
Interaction. When finally directed to the TV screen in the Suzuki Residence
during the miso task, both users commented that they had no idea they could interact with
the TV until I pointed it out. Both the TV and the buttons that control the stop, play, and
menu functions were too small, they said (Figure 6).
Runnels—Usability Study 18
Figure 6: TV Screen and Buttons Too Small
In addition, when finally clicking on the menu, the teacher user could not see the menu
listings because one of the character’s word bubbles covered up almost half of the menu
(Figure 7).
Figure 7: Word Bubble on top of Menu
She had to wait for the word bubble to disappear before she could scan the menu options,
at which point she did click on the miso link, which took her to the definition of miso
and, thus, completed her task. The student user tried to click on the rice cooker because
Runnels—Usability Study 19
the pointer turned to a hand, which would indicate a button; however, the rice cooker did
not do anything, which the user found “annoying.” The student also remarked that it was
not obvious he could click on a dinner selection (soup, rice, etc.) more than once to
change what the family ate for dinner; rather, he discovered this by accidentally clicking
twice. The student user also wanted to see the family actually eat the dinner, as in the
food disappearing, when he clicked on “start dinner,” but the only thing that happened
was the family started talking. The teacher suggested that to complete the task of defining
miso, she should be able to click on the miso soup on the table instead of having to click
on the TV.
Both the student and teacher user successfully completed the task of serving
dinner to the Suzuki family, but they both commented that they would like to “see it do
more.” However, the task of finding out how they make sushi in Japan pleased them
because they could watch a video that includes systematic directions.
Africa for Kids
Africa for Kids is a site produced by Thirteen, a “leading provider of educational,
informational, and cultural products and services using all media” (n.d.). Published
through PBS, the site offers an interactive introduction to African culture through the
eyes of students in Africa.
Information. Both the student and teacher users were very satisfied with the
quality and educational value of the information presented on the Africa site. The short
blurbs of information (Figure 8) made the site easy to read, but the teacher user would
have liked to see hyperlinks that provided additional information with a click. Both users
tried to click on words that stood out because they were a different font color, which is a
Runnels—Usability Study 20
convention that often indicates a hyperlink, but the words were highlighted only to draw
attention to them because they were culturally-specific.
Figure 8: My World Page
Neither user had any trouble locating the information for the first task (define kenkey).
The site uses a “guide” in the form of a cartoon African youth who gives the users
information on each section of the site (Figure 9), and both users relied on this
information to guide them through their tasks.
Figure 9: The “Guide”
Runnels—Usability Study 21
Interface. The limited options on the site (only four menu options) made it easy
to navigate. Neither user experienced a task failure at any time; rather, both users
completed the tasks quickly and efficiently. Krug (2006) asserts that persistent navigation
on a website needs to include five elements: the site ID, a way home, a way to search,
utilities, and sections (p. 62). Every page on the Africa site included all of the elements of
persistent navigation, except a search tool, which would probably be superfluous on this
particular site. Both users were impressed with the color scheme and graphics, though
they both mentioned it was “simple,” and the teacher commented that it looked like a site
geared toward younger students.
Interaction. “Fun” was the word both the student and teacher used when
describing the site’s interaction. It was actually difficult to get the users to stop playing
the thumb piano (Figure 10), as they were enjoying themselves so much.
Figure 10: The Thumb Piano
The student user commented that he would have liked to be able to mix the two recorded
tracks together, which was not an option the site afforded. The teacher thought it was
Runnels—Usability Study 22
great that she could print sheet music for the thumb piano and have students try to play
the songs.
During reading or listening to the folktale, neither user had any trouble
completing the task of taking the interactive quiz. The teacher was pleased that when an
incorrect answer was chosen, a thorough explanation of why the answer was wrong was
presented (Figure 11).
Figure 11: Incorrect Answer Explanation
The teacher also thought the accessibility of the site was “fantastic” because it provided
students with the opportunity to either read the story or listen to it being read by the
narrator.
Race: The Power of Illusion
Race: The Power of Illusion is a site published by PBS and produced by
California Newsreel, a “leading resource center for the advancement of racial justice and
diversity, and the study of African American life and history as well as African culture
and politics” (n.d.).
Runnels—Usability Study 23
Information. The Race site scored the highest overall for information. Although
the site contains an enormous amount of information, it is organized and presented in
such a way that it is easy for users to understand. Each menu item contains general
information but includes a link called, “Go Deeper,” which brings up additional resources
chock-full of supplementary information (Figure 11).
Figure 11: Additional Information Available
Whereas the student user disregarded the link to the additional resources, the teacher user
almost immediately clicked on the link and praised the quality of the educational
information she found.
Runnels—Usability Study 24
Interface. The Race site also scored the highest of the three for interface. The
teacher user was impressed with the “sophisticated yet modern” design of the site and
commented on the color scheme, which she saw as a metaphor for race. She explained,
“They avoided using black or white and seem to have purposefully chosen a caramel
color in between, which stands for the fact that we are all a mixture of races.” While the
student user did not seem to appreciate the color scheme in particular, he rated the
interface high because “it looks cool.” He could not elaborate on his description, so I
concluded the overall interface had a generalized positive effect on him.
Interaction. As with information and interface, the Race site also took the high
score for interaction. In fact, the site’s interaction score was the highest of all the scores.
The users attributed their high scores to the “interesting,” “fun,” and “educational”
interactive elements on the site. Neither user experienced a task failure. The tasks were
completed in a timely manner. Even though the student and teacher used two different
methods to complete the task of finding out what year minorities were denied social
security, the results were the same. The student user chose a theme on the timeline while
the teacher user chose a time period (Figure 12).
Runnels—Usability Study 25
Figure 12: Timeline
Although both users did poorly on the interactive quiz, they learned just as much
as if they had aced it because of the thorough answer explanations. After selecting an
answer, the correct answer is shown along with a detailed explanation (Figure 13).
Interacting with the quiz was simple yet highly informative.
Figure 13: Answer Explanations
The users got the most enjoyment out of interacting with the people sorter, which
asked users to sort 20 different people into race categories (American Indian, Asian,
Black, Hispanic/Latino, or White) based on their appearance alone (Figure 14). Both
users were shocked at the results, which was the point of the interactive feature. The
teacher user did complain that the thumbnail pictures were too small to clearly identify
race; however, she failed to read the directions at the top, which explained the pictures
could be enlarged by clicking on them. It appeared the student did not read the directions
either, but he immediately started clicking on the thumbnails because his expectation was
that the pictures would be enlarged. Both users spent the majority of their time on this
Runnels—Usability Study 26
task, especially on the results page (Figure 15) because they could click on individual
pictures and find out to what race the people belong.
Figure 14: People Sorter
Figure 15: People Sorter Results
Discussion
Although this usability study was small in scale, three very important design
issues emerged. The most important finding was that each of the design elements
(information, interface, and interaction) hinges on the other, so that if one is flawed it
affects the others. While the Sushi site contained excellent information, users were
Runnels—Usability Study 27
stumped as to how to retrieve the information, rendering it useless. Second, sites need to
ensure that users have no question about how to get from point A to point B on a site.
Buttons without labels, graphics that appear to be buttons but are not, and elements that
react on one page but not another only serve to confuse and frustrate users. As Krug
(2006) says, “Don’t make me think.” Users should immediately know where to click and
what to expect. Clear navigation, like that on the Africa site, allows users to focus on
actually using the site instead of how to get around on the site. Finally, websites should
not adopt the motto that more is better. Too many graphics, too many links, too much
information, and too many options leave users stranded and overwhelmed. Instead, sites
should scream, “Half as much twice as well!” Users who can easily grasp information,
navigate, and interact while enjoying the aesthetics of the site will be successful and be
more likely to return to the site in the future. The Race site is an exemplar for this type of
well thought-out design.
Because students and teachers are two very different and unique types of users,
the results of this study are not as significant as they would have been if only one type of
user had been tested. To improve on this usability study, researchers should comprise two
separate user groups, teachers and students, and test the sites independently first. The
results of the first tests should be used to fine-tune the tasks for the second test, in which
both students and teacher users would participate. The goal would be to find some
common ground on which sites could improve usability for both sets of users.
Runnels—Usability Study 28
Appendix A: Usability Test
Not Just Sushi Africa for Kids Race
Free Exploration 3 minute free-form 3 minute free-form 3 minute free-form
exploration exploration exploration
Specific Task You want to learn You want to explore You are interested
(Information) more about culture in Ghana. in finding out more
Japanese cuisine. Find out what about
Find out what miso kenkey is. discrimination. In
is. what year were
minorities denied
Social Security?
Open-Ended Task You love sushi but You are interested You want to learn
have no idea how it in African folktales. more about human
is made. Find out Read the story of diversity. Take the
how they make Prince Sadaka and quiz to see how
sushi in Japan. help him find his much you know.
brothers.
Specific Task You are interested You are curious You want to know
(Interaction) in what Japanese about musical how good you are at
families eat at instruments used in figuring out
home. Serve dinner Africa. Record and people’s race based
to the Suzuki play back music you on their appearance.
family. make using the Use the sorter to
thumb piano. sort people into
different races.
Runnels—Usability Study 29
Appendix B: Pre-Test Questionnaire
Name: _____________________________________________________
_____I am a student. _____I am a teacher.
Gender: Male Female
Your Age: ______ Your Race: _________
Please answer the questions below in order to help me understand your background and
experience.
EDUCATION (Please circle the highest grade level achieved below.)
Middle School 6 7 8 High School 9 10 11 12
College 1 2 3 4 Graduate 1 2 3 4
COMPUTER EXPERIENCE
1. What is the total length of time you have been using personal computers?
_______________________________________________________
2. On a typical day, how often do you use a computer?
_______________________________________________________
3. How often do you use the Internet?
_______________________________________________________
4. Where do you access the Internet most often?
________________________________________________________
5. What do you use the Internet for primarily?
________________________________________________________
6. What are the websites you visit most often?
_________________________________________________________
Runnels—Usability Study 30
Appendix C: Post-Test Questionnaire
Please rate your satisfaction with the following aspects of the site you have just finished
working with by circling the appropriate number. As you evaluate other sites, you may
come back and change ratings you have already made for this site.
Very Very
Unsatisfied Satisfied
1. Getting to the right part of the site 3 2 1 0 1 2 3
2. Quality of information about culture 3 2 1 0 1 2 3
3. Ease of reading the text 3 2 1 0 1 2 3
4. Appearance of site (colors, graphics, etc.) 3 2 1 0 1 2 3
5. Speed of pages displaying 3 2 1 0 1 2 3
6. Fun, entertainment value 3 2 1 0 1 2 3
7. Educational value of information 3 2 1 0 1 2 3
8. Overall experience 3 2 1 0 1 2 3
Very Very
Unlikely Likely
9. How likely are you to return to the site on your own? 3 2 1 0 1 2 3
Runnels—Usability Study 31
References
California Newsreel. (n.d.). About California Newsreel. Retrieved on April 8, 2007,
from http://www.newsreel.org/nav/aboutus.asp
California Newsreel. (n.d.) Race: The Power of an Illusion. Retrieved on April 7, 2007,
from http://www.pbs.org/race/000_General/000_00-Home.htm
Krug, S. (2006). Don’t Make Me Think: A common sense approach to web usability.
Berkeley, CA: New Riders Publications.
Morkes, J. and Nielsen, J. (1998). Applying writing guidelines to web pages. Retrieved
on February 8, 2007, from
http://www.useit.com/papers/webwriting/rewriting.html
Nielsen, J, Snyder, C., Molich, R., & Farrell, S. (2000). E-Commerce User Experience:
Methodology. Retrieved April 8, 2007 from
https://courses.utexas.edu/webapps/portal/frameset.jsp?tab=courses&url=/bin/co
mmon/course.pl?course_id=_92274_1
Rubin, J. (1994). Handbook of Usability Testing: How to Plan, Design, and Conduct
Effective Tests.
Thirteen. (n.d.) Mission statement. Retrieved on April 8, 2007, from
http://www.thirteen.org/homepage/about/index.html
Tidwell, J. (2006). Designing Interfaces. Sebastopol, CA: O’Reilly Media, Inc.
University of Texas at Austin. (2007). Center for East Asian Studies. Retrieved April 8,
2007, from http://www.utexas.edu/cola/centers/eastasia/?path[0]=eastasia
University of Texas at Austin. (2005). Instructional Technology. Retrieved April 8, 2007,
from http://www.edb.utexas.edu/CI/IT/index.php
Runnels—Usability Study 32
University of Texas at Austin. (2004). Not Just Sushi. Retrieved April 7, 2007, from
http://www.edb.utexas.edu/japanese/index.php
Runnels—Usability Study 33
This paper is written by Kurstin Blue Runnels for the course
EDC985G: Designs & Strategies for New Media
at the University of Texas—Austin.