Measuring Website Usability:
Instrument Development, Validation,
and Application
Big XII IS Research Symposium
April 5, 2003
Younghwa “Gabe” Lee
University of Colorado at Boulder
leey@colorado.edu
Agenda
Background
Previous Studies
Research Objective
Research Design
Expected Contribution
Discussion
Background
Online business failures are increasing as customers turn
away from unusable or unfriendly sites. ‘Build it and they
will come’ mentality has led to the demise of e-commerce
sites when sites are too late, too buggy, or too complex
- Becker and Mottay, 2001
In a poorly designed EC environment, users might be
uncomfortable with the uncertainty and ambiguity caused
by lack of interaction with websites.
- Jahng et al, 2000
The number of shoppers and total sales are still marginal,
mainly because of poor interfaces
- Jarvenpaa and Todd, 1997
Background
Building a usable website is important since website is the
only source for online customers to touch, feel, search,
communicate, and experience the products or services
available at the online store
Usable websites
Build positive attitude (Singh and Dalal, 1999)
Increase stickiness (Rettie, 2001)
Increase revisit rates (Klein, 1998)
Increase online purchase (Palmer, 2002)
Increase performance (Nielsen, 2000)
Provide more satisfaction (Lund, 1999)
Website Usability Research
Website
Usability
How to measure? How to verify?
Instruments Design and
Development Testing
HCI
Company-specific HCI
Industry Gurus MIS SIG HCI
Few IS researchers
Previous Studies
Measurement problems of website usability
No consensus on the definition and dimensions of
website usability
A number of single-item constructs
Intuition and experience-based: Few efforts to develop
measurement using scientific methods
HCI-oriented objective variables (error rate and
download time)
No Investigation of the relationship between
website usability constructs
No process model
Agarwal and Palmer (2002) Kim et al. (2002)
Venkatesh (2002)
• Content • Download Delay • Firmness
• Ease of Use • Navigability • Convenience
• Promotion • Information Content • Delight
• Made-for-the-medium • Interactivity
• Emotion • Responsiveness
McKinney et al. Zhang and
(2002) von Dran (2002)
• Access • Content
• Usability • Enjoyment
• Entertainment • Privacy
• Hyperlinks • User empowerment
• Navigation • Visual appearance
• Interactivity • Technical Support
• Navigation
• Credibility
• Organization
Download Download
Delay Delay
Navigability Navigability
Information Perceived Information Perceived
Content Success Content Success
Interactivity Interactivity
Responsiveness Responsiveness
Palmer (2002)
Motivation of the Study
Current inconsistency and incompleteness among
website usability measurement is the crucial
problem of website usability studies.
There is very little in the way of concrete
measurement that tells us how good a website
really is. Current guidelines, methods, and metrics
do help to design better websites, but there is room
for improvement
- Tarasewich (2000)
Research Objective
Develop measurement of website usability
18 constructs and 62 instruments have been identified
Investigate the causal relationship between website
usability constructs
Revealed Causal Mapping approach (Nelson et al., 2000)
Examine the effects of website usability constructs to
multiple dependent variables
Satisfaction, purchase intention, revisit intention, actual purchase,
affect, and loyalty
Investigate generalizability of the new measurement
and identify different causal maps under different
boundary conditions
Gender, Product, Industry and Culture
Research Design
Instruments Development
Literature review
Interviews with web usability experts
SUN, IBM, 37 Signals.com website designers
A Focus Group Study (IS-majored master-level subjects)
A Major survey to 400 Business undergraduate students
Exploratory Factor Analysis
Causal Relationship Between Website Usability
Constructs
Interviewed with experts (n = 20)
Interviewed with experienced online customers (n = 40)
Data Analysis suggested by Nelson et al. (2000)
Research Design
Effects of Website Usability Factors to Diverse
Dependent Variables
The effects of Price, Time, Scarcity, Convenience, Fun, Usefulness
will be examined together
CFA and Path Analysis will perform
Data Analysis: LISREL
Boundary Conditions
Compare websites with different gender-focused
Compare websites with different types of products (Hedonic vs
Utilitarian)
Compare websites with different cultures (U.S. vs Japan)
Compare websites with different stakeholders (Customers,
Designers, and Managers)
Data Analysis: PLS
Website Usability Constructs
Simplicity Scope Timeliness
Readability
Scanability
Consistency Relevancy
Learnability
Content
Navigability
Flexibility
Privacy Credibility Tele-
Presence
Security Interactivity
Reliability/ Community
Accessibility
Causal Relationship between
Website Usability Constructs
Simplicity Scanability Scope Timeliness
Readability
Consistency
+ Content
Relevancy
Learnability
+
Navigability
- +
-
Tele- Flexibility
Privacy Credibility - Presence
+ +
Interactivity
Reliability/ Community
Security
Accessibility
Effects of Usability Constructs to
Diverse Dependent Variables
Simplicity Scanability Scope Timeliness
Readability Actual Revisit
Purchase Intention
Consistency
+ Content
Relevancy
Learnability
+ Purchase
Satisfaction
Navigability Intention
- +
-
Tele- Flexibility
Privacy Credibility
- Presence
+ + Loyalty Affect
Price
Interactivity
Reliability/ Community Time
Security Scarcity
Accessibility
Convenience
Fun
Usefulness
Expected Contribution
Develop new measurement of website usability
provide a better means to evaluate website design quality
Identify causal relationship between website usability
constructs
Provide a deeper understanding of how online customers build their
usability perception
Examine effects of website design factors to diverse
dependent variables
justify the investment on website usability
Investigate measurement’s generalizability and causal
maps under different boundary conditions
help to perform future study using the measurement
help designers to allocate limited resources to the most important
usability factors
Discussion
Discussion
Revealed Causal Mapping Repertory Grid
Causal Relationship Identification Construct Identification
Guided Interviews Open Interviews
No requirement for multiple elements Multiple elements are required
Identify constructs and causal Identify constructs using
Relationship based on repeated Triadic Methods to identify
Interviews for identifying
Mid-theory exists No requirement for mid-theory
Discussion
Alternative ways of conducting the research
Process Model (e.g TAM) vs Multiple Dependent Variables
Boundary conditions valuable to be observed