Docstoc

Evaluation

Document Sample
Evaluation Powered By Docstoc
					Evaluation

Howell Istance
Why Evaluate?

• testing whether criteria defining success
  have been met
• discovering user problems
• testing whether a usability-related
  specification has been met
• comparing designs
Overview of approaches to
evaluation
• observation and monitoring usage
• structured expert reviewing
• usability engineering
• contextual enquiry
• experimental techniques
       Things to test
• Correct for target audience’s ability?
• Meets the design concept?
• Human-computer dialogue and
  graphics design principles adhered
  to?
• Material / information correct?
• Install correctly, and run, on target
  machines? Links work, etc?
• How easy is it for target audience to
  use?
  Approaches to Evaluation in
  Human-Computer
  Interaction
• User-centred evaluation of software in
  general usually focuses on how well
  users can complete tasks
• The idea of a ‘task’ is often less well-
  defined in multimedia applications
               Context for evaluation
• Free use - users asked to use the
 application without specific instructions
  – difficult to ensure users visit all parts of the
    application
  – may be unclear to the user what they should
  – best suited to small information presentations
• structured use - users guided in their use of
 the application
  – ensures that critical parts of application tested
    in limited time
Observation and monitoring
usage
• direct and indirect observation
• verbal protocols
• user opinions
• software logging
Structured Direct Observation
• give subjects a series of standard
  tasks to complete using a prototype
• observe subject completing tasks under
  standardised conditions
• data collection aimed at ensuring that
  qualitative descriptions of problems
  during task completion are captured
• what problems are likely in data
  recording?
    Standard tasks in structured
    direct observation
• structure tasks into incremental difficulty (easy
    ones first)
•   have a clear policy on subject becoming stuck and
    providing help
•   have a reason for including each task (avoid
    unnecessary duplication)
•   ensure (all) functional areas of interface usage are
    covered
•   ensure tasks of sufficient complexity are included
Indirect observation - video

• enables post-session debriefing 'talk-
  through' (post-event protocols)
• enables quantitative data to be extracted -
  e.g. part task timings
• serves as a diary and visual record of
  problems
• usually very time consuming to analyze
• usability laboratories
Verbal protocols
• means of enhancing direct observations
• user articulates what they are thinking during
    task completion (think-aloud protocols)
•   but…
    – doing this can alter normal behaviour
    – subject likely to stop when undertaking complex
      cognitive activities
    – user may rationalise behaviour in post-event
      protocols
• get subjects working in pairs - co-discovery can
    overcome some of these problems.
Collecting users opinions
• interview and questionnaire
• suited to both qualitative data and quantitative
    data collection
•   interviews
    – structured interviews (fixed sequence of questions)
    – semi structured (allows disgressions, but all
      questions covered)
    – flexible (exploration of topic governed by users
      views)
• what are the advantages and disadvantages of
    these?
Questionnaires
• contain closed questions (attitude scales)
  and open questions
• pre- and post questionnaires obtain ratings
  on an issue before and after an design
  change
• can be used to standardise attitude
  measurement of single subjects following
  direct observation
• can be used to survey large user groups
Types of rating scales

 Can you use the following edit commands?
               yes no don't know
 duplicate

 paste



     A simple checklist
Multipoint checklist

 Rate the usefulness of the duplicate
  command on the following scale?




 very                               of no
 useful                             use
 Likert Scale
• statement of opinion to which the subject
  expresses their level of agreement

Computers can simplify complex problems




  very much     agree    slightly   neutral       slightly   disagree    strongly
  agree                 agree                 disagree                  disagree
Caution!
The help facility in system A is much better than the help facility in
  system B




   very much   agree   slightly   neutral   slightly     disagree   strongly
   agree                 agree                  disagree            disagree




what does 'strongly disagree' mean?
 Semantic differential Scale
uses a series of bi-polar adjectives and obtains
  ratings which respect to each

Rate the Beauxarts drawing package on the following
  dimensions
           extremely   quite   slightly   neutral   slightly   quite   extremely

easy
   difficult

clear
    confusing

fun
      dreary
Rank Order

  Place the following commands in order of
    usefulness (rank the most useful as 1,
    the least useful as 4)



   paste           duplicate      group
   clear
    Do and Don'ts with
    Questionnaire evaluation
• have a clear idea of what specifically you want
    information about and ensure there are questions
    that directly address these issues
•   don't risk subjects being demotivated
    – not interested in the questionnaire
    – questionnaire is too long
• don't be lazy
    – focus questions to the specific interface
    – avoid 'not applicable' responses
• provide specific task reference for questions
Structured Expert Reviews

• uses ‘experts’ in HCI and task domain to
  review design rather than subject-based
  testing
• methods vary according to how the review
  is structured
• two popular methods
  – heuristic evaluation
  – cognitive walkthrough
    Heuristics in Heuristic
    Evaluation
• use simple and        provide clearly
  natural language       marked
• speak the users    

                     
                         exits
                         provide shortcuts
  language              provide good error
• minimise users        messages
  memory load           prevent errors
• be consistent
• provide feedback
S: Enter 1 for account information, 3 for transfers
   between accounts..
U: 3#      (interrupts)
S: Enter account to transfer from
U: 1234567890#
S: Enter account to transfer to
U: #      (default)
S: Enter amount in cents
U: 100000#
S: From account number 1234567890 to account
   number primary account, transfer of 1000 dollars
   is to be made. Press 1 to confirm, 0 to cancel
U: 1#
S: You do not have access to use this function
S: Enter 1 for account information, 3 for transfers between accounts..
U: 3#     (interrupts)
S: Enter account to transfer from
U: 1234567890#
S: Enter account to transfer to
                                  read number before
U: #    (default)                 menu item description
S: Enter amount in cents          (minimise users memory load)
U: 100000#
S: From account number 1234567890 to account number
primary account, transfer of 1000 dollars is to be made.
Press 1 to confirm, 0 to cancel
U: 1#
S: You do not have access to use this function
S: Enter 1 for account information, 3 for transfers between accounts..
U: 3#     (interrupts)
S: Enter account to transfer from
U: 1234567890#
S: Enter account to transfer to   Avoid gap in menu numbers
U: #    (default)                 between 1 and 3
S: Enter amount in cents          (Simple and Natural Language)
U: 100000#
S: From account number 1234567890 to account number
primary account, transfer of 1000 dollars is to be made.
Press 1 to confirm, 0 to cancel
U: 1#
S: You do not have access to use this function
S: Enter 1 for account information, 3 for transfers
   between accounts..
U: 3#     (interrupts)
S: Enter account to transfer from
U: 1234567890#
                              Error message appears
S: Enter account to transfer to
U: #     (default)            too late
                              (simple and Natural Dialogue
S: Enter amount in cents
U: 100000#
S: From account number 1234567890 to account
   number primary account, tansfer of 1000 dollars is
   to be made. Press 1 to confirm, 0 to cancel
U: 1#
S: You do not have access to use this function
S: Enter 1 for account information, 3 for transfers
   between accounts..
U: 3#      (interrupts)
S: Enter account to transfer from not require dollar
                                Do
U: 1234567890#                  amounts to entered in cents

S: Enter account to transfer to (speak the users language)
U: #      (default)
S: Enter amount in cents
U: 100000#
S: From account number 1234567890 to account
   number primary account, transfer of 1000 dollars
   is to be made. Press 1 to confirm, 0 to cancel
U: 1#
S: Enter 1 for account information, 3 for transfers
   between accounts..
U: 3#      (interrupts)
S: Enter account to transfer from message is imprecise
                               Error
U: 1234567890#
                               (precise and constructive
S: Enter account to transfer toerror messages)
U: #      (default)
S: Enter amount in cents
U: 100000#
S: From account number 1234567890 to account
   number primary account, transfer of 1000 dollars
   is to be made. Press 1 to confirm, 0 to cancel
U: 1#
S: Enter 1 for account information, 3 for transfers
   between accounts..
U: 3#      (interrupts)
S: Enter account to transfer from
                                Use alternative and proper
U: 1234567890#                   English phrase
S: Enter account to transfer to
U: #      (default)             (speak users language)
S: Enter amount in cents
U: 100000#
S: From account number 1234567890 to account
   number primary account, transfer of 1000 dollars
   is to be made. Press 1 to confirm, 0 to cancel
U: 1#
S: You do not have access to use this function
Cognitive Walkthrough
• developed on the basis of Cognitive Theory of
    Initial Learning
•   intended for systems where the user ‘guesses
    their way’ through an interaction sequence
•   the task is decomposed into paths to
    successful task completion consisting of
    individual actions
•   method provides
    – set of guide lines to support development
    – procedures and checklist for evaluation
Cognitive Walkthrough Checklist
• Problems forming correct goals
  –   failure to add goals
  –   failure to drop goals
  –   addition of spurious goals
  –   premature loss of goals
• problems identifying action
  – correct action does not match goal
  – incorrect actions match goals
• problems performing action
  – physical difficulties
  – timeouts
Example of correct goal
structure
  • Program video for timed recording
    – Press timed recording button
    – Set Stream
       • type stream number
       • press ‘timed recording’ button
    – Set start time
       • type start time (24 hour clock)
       • press ‘timed recording’ button
    – …..

				
DOCUMENT INFO
Categories:
Tags:
Stats:
views:8
posted:9/30/2011
language:English
pages:32