Docstoc

Usability Test Report

Document Sample
Usability Test Report Powered By Docstoc
					4004-549, 749 Usability Testing                           Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja




Master of ORION II Battle at ANTARES

CIF - Usability Test Report

Version 1.0, May 11, 2004




4004-549, 749 Usability Testing

Team 4 members

    - Chris Lister
    - Larry Conrow
    - Steve Shapiro
    - Shiranchal Taneja




     Confidential                              Page 1                                 9/25/2011
4004-549, 749 Usability Testing                              Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

Contents

    1. Executive Summary
          a. Name and description of product
          b. Methods
                    i. Numbers
                   ii. Types of Participants
                  iii. Task
          c. Reason and nature of test
          d. Tabular summary of performance results
          e. Probability of no chance

    2. Introduction
           a. Full Product Description
                     i. Product name and release or version
                    ii. Parts of product evaluation
                   iii. User population
                  iv. Testing environment game should be used in
                    v. Type of game
           b. Test Objectives
                     i. Test objectives
                    ii. Functions and components user directly indirectly interacted


    3. Method
          a. Participants
                     i. Number of Participants tested
                    ii. Segmentation of User Tested
                   iii. Key characteristics of users
                   iv. Participant sample and use population difference
                    v. Table of participants
          b. Product Use in test
                     i. Task scenarios
    1. why task were selected
    2. Source of task
    3. Task date given to participants
    4. Completion of performance criteria for task

            c.   Task Facility
                       i. Setting and type of space
                      ii. Relevant circumstances that affected test
            d.   Participant‟s Computing Environment
                       i. Computer Configuration
            e.   Display Devices
                       i.
            f.   Audio Devices
                       i. Audio volume
            g.   Manual Input devices
                       i. Model of device
            h.   Test Administrator Tools
                       i. Questionnaire
                      ii. Hardware and software used to recode test
            i.   Experimental Design
                       i. Logical design of test
                      ii. Independent variables and control variables
                     iii. Measure of data recoded for ach set of conditions



     Confidential                                 Page 2                                 9/25/2011
4004-549, 749 Usability Testing                            Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja



            j. Procedure
                     i. Operational definitions of measure
                    ii. Description of independent variables or control variables
                   iii. Time limits on task
                   iv. Policies and procedures for interaction between tester(s) and subject
                    v. Sequence of events from greeting to dismissing
                   vi. Details of non-disclosure agreements, from completion, wrap-ups, pre-task training,
                        and debriefing
                  vii. Verification participants knew and understood their rights
                 viii. Steps followed to execute the test sessions and record data
                   ix. Number of roles during test that interacted with participants
                    x. Other individuals present at test
                   xi. Payment and compensation
            k. Participant general Instructions
            l. Participant Task Instructions
                     i. Task instructions summary
            m. Usability Metrics
                     i. Effectiveness
                    ii. Completion rate
                   iii. Errors
                   iv. Assists
                    v. Efficiency
                   vi. Completion rate/Mean time on task
                  vii. Satisfaction

    4. Results
          a. Data Analysis
          b. Presentation of the Results
          c. Performance Results
          d. Satisfaction Results

    5. Appendices




     Confidential                                Page 3                                  9/25/2011
4004-549, 749 Usability Testing                               Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja



    1. Executive Summary

Master of Orion II Battle at ANTARES is the second version in a series of three games. The game was
released in 1996 by Micropros a video game development company. Master of Orion 2 was the successful
follow up to Master of Orion. Although, it was a complex game it sold well and was thought of very highly.
The third version of this game was also complex but, did not do as well as the previous games.

Master of Orion II is a “4x” turn-based strategy video game, were your mission is to explore, expand,
exploit, and exterminate. You start out with a home planet and a couple of ships. You need to explore the
star system, gather resources and build buildings and research technology to build up an army to dominate
the universe.

We will be focusing on the learn-ability of the game itself. We will be testing users on mandatory primary
tasks in the game. We will also be testing their abilities to accomplish certain secondary tasks.

Types of participants –

Our recruitment attempts gave us a good number of people filling out the questionnaire website that was
listed on the flyers. Because of the location of the flyers, we were reaching populations that were heavier in
our first two age categories. The result was that the first two age groups made up two thirds of the
population.


Break Down of participants

                       15-19                  20-23                  24-28                  29-45
Beginner               2                      3                      2                      2
Expert                 4                      4                      2                      0

Unusable               0                      1                      1                      0




     Confidential                                  Page 4                                   9/25/2011
4004-549, 749 Usability Testing                                 Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


Task –

Task 1 – list of task handed to participants

        Script: Start a new game that has the following attributes:

           - Difficulty: Tutor
           - Galaxy Size: Medium
           - Galaxy Age: Average
           - Players: Five
           - Tech Level: Average
           - Tactical Attack: Off
           - Random Events: Off
           - Antarans Attack: Off
           - Race: Human

         You now have five minutes to play around with the game in order to get used to the interface.

        Script: Using your resources build a „Frigate‟. A „Frigate‟ is a type of attack ship. Please think aloud
         throughout the process.


        Script: Send two of the „Frigates‟ in your fleet to the nearest solar system. Please think aloud
         throughout the process.

        Script: Create a colony in the newly discovered solar system. Since there are multiple planets in
         the solar system, you are to determine which one is most inhabitable and favorable. Please think
         aloud throughout the process.

        Script: Using the in-game help system, find out about the capabilities of a „Nuclear Missile.‟ Please
         think aloud throughout the process.

        Script: Using the instruction manual, find help about “Building Ships.” Please think aloud
         throughout the process.



Reason and nature of test – The primary purpose of this testing is to asses the usability of Master of Orion
II. The usability test will measure learn-ability, accessibility, and navigability involved in accomplishing the
mandatory primary task of the video game.




     Confidential                                   Page 5                                     9/25/2011
4004-549, 749 Usability Testing                                 Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja



Tabular summary of performance results –


                                  Time on   Time on       Time on       Time on       Time on       Time on
         Participant              Task 1    Task 2        Task 3        Task 4        Task 5        Task 6

           Mean                    44.3      389.3          252.8        156.9         52.6          194.2
          Median                    45        330           212.5        112.5          44            135
           Mode                     45       #N/A            360           60           60            120
       Standard Dev                18.14     310.7         243.48        118.67        26.83         175.69


           Highest                  75        1040            840         390           120           660
           Lowest                   10         50              20          40            20            47

           Range                    65        990             820         350           100           613

         Quartile 1                 33        120              38          60            40           120
         Quartile 3                 60        590             360         240            60           185
   Inter Quartile Range             27        470             322         180            20            65

            Mean                   44.3      389.3         252.8         156.9          52.6         194.2
      Trimmed Mean
           (10%)                   44.75    350.375        208.5        142.375        48.25        154.375
    Standard Dev after
         trimming                  10.97     213.18        130.28        90.27         10.16         61.49




Probability of no chance –

Learn-ability:

Task 3 and task 4 of the usability test were extremely similar. Task 3 asked the user to move two combat
ships from one star system to another. Task 4 asked the user to colonize a planet in another star system.
This is done by sending a “colony ship” to that star system. In essence, the user is performing the same
function with two different types of ships. Therefore, we can assume that completion time for task 4 should
be significantly lower than that of task 3 if the game is learnable. This hypothesis is true. Mean task time for
task 3 was 252.8 second. The mean task time for task 4 was 156.9 seconds – a drop of over 60%.




     Confidential                                    Page 6                                    9/25/2011
4004-549, 749 Usability Testing                                            Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja



Below is the comparison between Task3 and Task 4


                                                    Learnability

                     900                                                     840
                     800
                     700
   Time ( in secs)




                     600
                     500
                                     390                   360                                      360
                     400                                     300 295
                     300           245                                         240
                           180                                                         185
                                                                                     160
                     200     120                                       105
                                              69    3860                                               60
                     100                    20                                               3040
                       0
                             1        2       3       4       5        6        7      8       9      10
                                                               Users

                                                           Task 3    Task 4


                     Figure2. Comparison between time taken for task 3 and task 4 of all users.

Null Hypothesis Test: It takes more time to complete Task 4 than it takes to complete Task 3

Confidence Level: 90%

 t-Test: Paired Two Sample for Means

                                                       Variable
                                        Variable 1         2
 Mean                                          252.8      156.9
 Variance                              59279.51111 14081.66
 Observations                                     10          10
 Pearson Correlation                   0.450495252
 Hypothesized Mean Difference                       0
 df                                                 9
 t Stat                                1.393965211
 P(T<=t)                               0.098392043
 t Critical                            1.383028803
Note: Variable 1 represents Task 3 and Variable 2 represents Task 4.

In the data above, we could note that 90% confidence level we can see that P-value is lower than 0.09. So
we can reject the null hypothesis can conclude that Task 4 takes less time than task 3 and this leads to that
the game is Learnable.




           Confidential                                       Page 7                                   9/25/2011
4004-549, 749 Usability Testing                               Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


    2. Introduction

            b. Full Product Description – CHris
                      i. Product name and release or version - Master of Orion II Battle at Antares
                     ii. Parts of product evaluation – Video game software and instruction manual
                    iii. User population - The population of user ranged from the ages of 19 to 35 and the
                         level of video game experience was from novice to expert. The criteria for the
                         setting a participant as a novice, intermediate, or expert was based on pre-test
                         questionnaire and surveys.

                      iv. Testing environment game should be used in – The environment that this product
                          should be used in is a home environment or with in avideo game environment such
                          as a LAN party.

                      v. Type of game – Master of Orion II is a “4x” turned-based strategy video game.


            c.   Test Objectives
                      i. Test objectives    – The specific questions that need to be answered:
1. Can end users perform the four primary tasks within established benchmarks?

2. Does the product contain major usability problems that prevent the completion of the most common task?

3. Is their and appropriate balance of ease of use and ease of learning?

4. Is Master of ORION III harder to play than Master of ORION II?


                      ii. Functions and components user directly indirectly interacted -




     Confidential                                  Page 8                                  9/25/2011
4004-549, 749 Usability Testing                            Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


    3. Method

Participants

1. Age: (15-19): 2
        (20-23): 5
        (24-28): 1
        (29-Up): 2


                                               Group by Age

                100%
                 90%
                 80%
                 70%
                 60%                            50%
                 50%
                 40%
                 30%          20%                                             20%
                 20%                                          10%
                 10%
                  0%
                              15-19            20-23          24-28           29-up


2. Gender:
      __0__ - Female
      __10_ - Male


                                           Group by Gender
                                  100%
           100%
            90%
            80%
            70%
            60%
            50%
            40%
            30%
            20%
            10%                                                       0%
             0%
                                    Male                           Female




     Confidential                                 Page 9                               9/25/2011
4004-549, 749 Usability Testing                               Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


3. What is your first language?

        __8__ - English
        __1__ - Spanish
        __1__ - Other – (please indicate) ___Chinese_____


                                            Group by First language

               100%
                90%                80%
                80%
                70%
                60%
                50%
                40%
                30%
                20%                                     10%                      10%
                10%
                 0%
                                  English              Spanish                 Chinese


4. How would you rate your skills in playing video games?
      (Please circle one of the numbers below.)

   None             Beginner          Average (8)        Expert (2)
   | --------------- | ----------------- | --------------- |
   1                2                    3                 4


                                     Group by skills in Video Games

               100%
                90%                      80%
                80%
                70%
                60%
                50%
                40%
                30%                                                      20%
                20%
                10%
                 0%
                                       Average                          Expert




     Confidential                                   Page 10                               9/25/2011
4004-549, 749 Usability Testing                               Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


5. Based on a one week time frame, how many hours do you spend playing video games?

        _____    -   Less than 30 minutes
        __1__    -   30 minutes to an hour
        __3__    -   1 to 5 hours
        __4__    -   5 to 10 hours
        __2__    -   More than 10 hours
        _____    -   Not applicable


                                         Group by no. of Hrs spent

                     100%
                      90%
                      80%
                      70%
                      60%
                      50%                                         40%
                      40%                             30%
                      30%                                                    20%
                      20%                 10%
                      10%         0%                                                     0%
                       0%
                                  < .5   .5 to 1     1 to 5      5 to 10     >10         NA
                                                        (in Hours)




     Confidential                                  Page 11                                9/25/2011
4004-549, 749 Usability Testing                                 Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


6. In the last 30 days, how many times have you played video games?

        _____    -   This is my first time
        __1__    -   2 to 3 times
        __1__    -   3 to 5 times
        __3__    -   5 to 10 times
        __5__    -   More than 10 times



                            Group by no. of times played in last month

               100%
                90%
                80%
                70%
                60%                                                                  50%
                50%
                40%                                                     30%
                30%
                20%                          10%       10%
                10%           0%
                 0%
                              O          2 to 3        3 to 5          5 to 10       > 10




     Confidential                                  Page 12                                  9/25/2011
4004-549, 749 Usability Testing                            Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


7. What types of video games go you like to play?

        __6__ - Role-Playing Games (RPG) (Star Wars Galaxies)
        __6__ - Strategy Games
                      __6__ - (TB) Turn Based (Master of Orion)
                      __6__ - (RTS) Real Time Strategy (Final Fantasy)
        __1__ - Simulation Games (Flight Simulator 2004: A Century of Flight)
        __4__ - Action (Battlefield 1942)
        _____ - Adventure Games (Unreal Tournament)
        _____ - First-Person Shooters (Gran Turismo)
        _____ - Puzzle games (Yu-Gi-Oh! The Dawn of Destiny)
        _____ - Sports games (Madden NFL)
        _____ - Other (please explain)



                                  Group by Type of Games experience

               100%                                        90%
                90%
                80%
                70%          60%        60%        60%                 60%
                60%
                                                                                   Novice
                50%      40%          40%       40%                      40%
                40%                                                                Expert
                30%
                20%                                           10%
                10%
                 0%
                           Role     Turn Based Real Time   Simulator   Action
                          Playing    Strategy  Strategy




     Confidential                               Page 13                                9/25/2011
4004-549, 749 Usability Testing                                  Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja




8. What platforms do you like to play electronic games?

        __8__    -   Sony's PlayStation2,
        __4__    -   Nintendo's GameCube
        __2__    -   Microsoft's Xbox
        __9__    -   personal computers,
        __3__    -   handheld game units,
        __1__    -   mobile phones,
        __7__    -   public arcade game machines and
        _____    -   other game units (please explain)



                                            Group by Platforms

                      100%                               90%           90%
                       90%        80%           80%
                       80%                                      70%             70%
                       70%              60%
                       60%                40%                                            Novice
                       50%                                       30%           30%
                       40%   20%                 20%                                     Expert
                       30%                             10%               10%
                       20%
                       10%
                        0%
                                                  be


                                                  ox




                                                   ts
                                                   rs




                                                  es
                                                   n




                                                  es
                                                 io




                                                te


                                                ni
                                              Xb
                                              Cu




                                              on


                                             am
                                              at




                                              U
                                             pu




                                          Ph
                                           st




                                          t's




                                            e
                                           e




                                          G
                                          m

                                       am
                                        ay


                                       am


                                        of


                                      Co




                                      ile


                                        s
                                     Pl




                                     de
                                     os
                                     G




                                     G


                                   ob
                                   al




                                 ca
                                 icr
                     's




                                 ld
                                   s




                                 M
                                on
                  ny


                                o'




                              he




                              Ar
                               M
                            nd




                             rs
                So




                          nd




                           ic
                         Pe
                         te




                        bl
                       Ha
                         n




                      Pu
                      Ni




     Confidential                                     Page 14                                9/25/2011
4004-549, 749 Usability Testing                               Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja




Why task were selected

Source of task - The performance test consists of a series of tasks that the participants will be asked to
carry out while being observed. Prior to each task, the test monitor will load the game to a predetermined
location and read a script to the participant. Both the script and the user manual will be available to the
participant at all times. The scenarios are as follows:

Task date given to participants –
Starting a New Game

       Scenario: The game is at the initial screen – no new game has been started yet.

       Script: Start a new game that has the following attributes:

          - Difficulty: Tutor
          - Galaxy Size: Medium
          - Galaxy Age: Average
          - Players: Five
          - Tech Level: Average
          - Tactical Attack: Off
          - Random Events: Off
          - Antarans Attack: Off
          - Race: Human

        You now have five minutes to play around with the game in order to get used to the interface.

       Proper Steps: Player selects all correct options and starts a new game.

       Evaluation Methods: This task will be evaluated using two methods. First, performance time will
        be recorded. Second, the number of errors, or incorrect steps, made by the user will be recorded.




     Confidential                                 Page 15                                   9/25/2011
4004-549, 749 Usability Testing                                Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja



Building a Ship

       Scenario: The game is loaded to the „galaxy‟ screen by the test monitor. At this point the
        participant only has control of one colony (Sol II) in one solar system (Sol). The participant‟s fleet
        consists of two (2) „Frigates‟ and one (1) „Colony Ship.‟

       Script: Using your resources build a „Frigate‟. A „Frigate‟ is a type of attack ship. Please think aloud
        throughout the process.

       Proper Steps: There are two ways to correctly complete this operation. First, the user may click on
        the „Sol‟ solar system and then click on the inhabited planet. This will take the user to the colony
        window. At this point the user needs to click the “change” button in the top-right portion of the
        screen. This will take the user to the build window. At this point the user must click „Frigate‟ and
        then „Ok‟. The user will then receive a prompt stating “Trade Goods is above a product. Do you
        want to delete trade goods before exiting?” The user must click yes in order to give ship building
        the temporary priority. This can be accomplished using a different process. From the galaxy screen,
        the user may click „Colonies‟ in the bottom-left corner of the screen. This will take them to a screen
        that lists all of their current colonies and provides some information about them. This includes
        construction projects, which is represented by the last column. By clicking “Trade goods” the user is
        taken to the same build window. From this point the same procedure needs to be followed as
        before. The operation will take two turns

       Evaluation Methods: This task will be evaluated using two methods. First, performance time will
        be recorded. Second, the number of errors, or incorrect steps, made by the user will be recorded.




     Confidential                                  Page 16                                     9/25/2011
4004-549, 749 Usability Testing                               Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja



Moving a Ship

       Scenario: If the participant completed the task successfully, the game does not need to be
        modified. If the participant was unable to complete the task or made errors that affected the game
        residually, the test monitor will load the game to a predetermined location. At this point, the
        participant‟s fleet will consist of three (3) „Frigates‟ and one (1) „Colony Ship‟.

       Script: Send two of the „Frigates‟ in your fleet to the nearest solar system. Please think aloud
        throughout the process.

       Proper Steps: There are two ways to correctly complete this operation. First, the user may click on
        the small ship icon next to the „Sol‟ solar system. This will bring up the fleet window which contains
        three ships. The proper ship must be selected and the given orders by clicking on the solar system
        of choice. This task may also be completed by clicking on the „Fleets‟ button that is located at the
        bottom of the screen. This will bring up a screen that will allow the user to select the proper ship.
        Similarly, the user needs to select the destination using a mini-map of the galaxy. The operation will
        take 2 turns.

       Evaluation Methods: This task will be evaluated using two methods. First, performance time will
        be recorded. Second, the number of errors, or incorrect steps, made by the user will be recorded.


Exploring and Colonizing a Planet

       Scenario: If the participant completed the task successfully, the game does not need to be
        modified. If the participant was unable to complete the task or made errors that affected the game
        residually, the test monitor will load the game to a predetermined location. At this point, the
        participant still only has one colony but is able to see the planets in a nearby solar system. Note
        that both of the nearby “reachable” solar systems had multiple planets. This was done to create a
        situation in which the participant needs to choose the most “inhabitable” planet by exploring its
        resources. The participant‟s fleet consists of three (3) „Frigates‟ and one (1) „Colony Ship‟.

       Script: Create a colony in the newly discovered solar system. Since there are multiple planets in
        the solar system, you are to determine which one is most inhabitable and favorable. Please think
        aloud throughout the process.

       Proper Steps: The user needs to click on the small ship icon next to the „Sol‟ solar system. From
        there the user must click „colonize planet‟ and select the planet they wish to colonize. There is
        another way of doing this task. The user may access the build menu as if they were building a ship.
        From there they can build a “Colony Base” which will take 34 turns. At that point the user is asked
        which planet within the solar system they wish to colonize. Using the first method will make a
        „Colony Ship‟ disappear from the fleet. The second method does not do this.

       Evaluation Methods: This task will be evaluated using two methods. First, performance time will
        be recorded. Second, the number of errors, or incorrect steps, made by the user will be recorded.




     Confidential                                 Page 17                                    9/25/2011
4004-549, 749 Usability Testing                              Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja



Obtaining Help via Game

       Scenario: The current game situation does not affect this test. The participant is simply asked to
        find help about a particular topic in-game.

       Script: Using the in-game help system, find out about the capabilities of a „Nuclear Missile.‟ Please
        think aloud throughout the process.

       Proper Steps: The user needs to click on the info button at the bottom of the screen. From there
        the user must select „Tech Review‟ from the menu on the left hand side of the screen. The
        „weapons‟ submenu must then be selected – this button is located at the bottom of the screen.

       Evaluation Methods: This task will be evaluated using two methods. First, performance time will
        be recorded. Second, the number of errors, or incorrect steps, made by the user will be recorded.

Obtaining Help via Manual

       Scenario: The current game situation does not affect this test. The participant is simply asked to
        find help about a particular topic in the user manual.

       Script: Using the instruction manual, find help about “Building Ships.” Please think aloud
        throughout the process.

       Proper Steps: The user is to flip through the manual until the item is found.

       Evaluation Methods: This task will be evaluated using one method -performance time will be
        recorded.




     Confidential                                 Page 18                                   9/25/2011
4004-549, 749 Usability Testing                                 Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja



Completion of performance criteria for task

Task Facility

The primary purpose of this exploratory testing is to assess and compare the usability of Master of Orion II
with Master of Orion III. The usability test will measure learn-ability, accessibility, and navigability involved
in accomplishing the mandatory primary tasks of the game.

Test Environment and Equipment Requirements:

We will need to create a gaming environment approximating a home including a desk, computer, speakers,
and chair.

The computer will be a PC running Windows 95, Windows 98, or Windows XP installed and running on the
computer. The processor has a minimum requirements of a “486” or “Pentium” and be at least 100 MHz
speed fast computer. The computer must have at least a 16 MB of RAM installed. Their must also be
enough storage space on the computers hard drive to store the necessary files form the install. The
graphics video card must be a Super VGA quality (640x480x256 colors) or better. A sound Blaster or
compatible sound card is supported. The machine will contain with software Master of Orion II, compact
disc of the game inside the CD-ROM, and all the necessary additional software such as Direct.X.5 or more.

We will also need some screen capture software such as snag-it running in the background.


The lighting should approximate a home gaming environment and at the same time, allow for clear
videotaping. Other than the computer, following items will be on the table.

    1) A printout of the Master of Orion II user instructor‟s manual guide. The paper should have clearly
       printed characters in order to ascertain if the participant want to refer for some strategy options.

    2) Pencil, if participant want to mark something in the manual which he/she thinks is useful or
       confusing.

The game should be installed properly, should have the privilege to save the status of the game on to the
hard disk, and be able to run the all the task mentioned in the task list. It is imperative that at least one
evaluator remain on site to quickly manage should the game abruptly terminates.




     Confidential                                   Page 19                                     9/25/2011
4004-549, 749 Usability Testing                                Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja




            d. Experimental Design -
                     i. Logical design of test
                    ii. Independent variables and control variables
                   iii. Measure of data recoded for ach set of conditions
            e. Procedure - Steve
                     i. Operational definitions of measure
                    ii. Description of independent variables or control variables
                   iii. Time limits on task
                  iv. Policies and procedures for interaction between tester(s) and subject
                    v. Sequence of events from greeting to dismissing
                  vi. Details of non-disclosure agreements, from completion, wrap-ups, pre-task training,
                        and debriefing
                  vii. Verification participants knew and understood their rights
                 viii. Steps followed to execute the test sessions and record data
                  ix. Number of roles during test that interacted with participants
                    x. Other individuals present at test
                  xi. Payment and compensation

             f. Participant general Instructions – Larry
The participants will receive a short, verbal, scripted introduction to the test. This information about what is
expected of them. They will be assured and reminded that the product is the center of the evaluation and
not themselves, and that they should perform in the way that is typical and comfortable to them. The
participants will be informed that they are being observed, videotaped, audiotape, and that their actions are
being recorded.

            g. Participant Task Instructions –

Task 1 – Start a new game
Task 2 – Build a ship (Frigate)
Task 3 – Moving a ship
Task 4 – Colonize a planet
Task 5 – Obtaining In game help
Task 6 – User manual – create a ship

                      i. Task instructions summary –

       Script: Start a new game that has the following attributes:

          - Difficulty: Tutor
          - Galaxy Size: Medium
          - Galaxy Age: Average
          - Players: Five
          - Tech Level: Average
          - Tactical Attack: Off
          - Random Events: Off
          - Antarans Attack: Off
          - Race: Human

        You now have five minutes to play around with the game in order to get used to the interface.




     Confidential                                  Page 20                                     9/25/2011
4004-549, 749 Usability Testing                                Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

       Script: Using your resources build a „Frigate‟. A „Frigate‟ is a type of attack ship. Please think aloud
        throughout the process.


       Script: Send two of the „Frigates‟ in your fleet to the nearest solar system. Please think aloud
        throughout the process.

       Script: Create a colony in the newly discovered solar system. Since there are multiple planets in
        the solar system, you are to determine which one is most inhabitable and favorable. Please think
        aloud throughout the process.

       Script: Using the in-game help system, find out about the capabilities of a „Nuclear Missile.‟ Please
        think aloud throughout the process.

       Script: Using the instruction manual, find help about “Building Ships.” Please think aloud
        throughout the process.



After completing our data analysis, we were able to draw several strong conclusions about the
video game. These interpretations are based on quantitative data obtained from the analysis
and qualitative data obtained from user comments. It can best be categorized into three
sections: playability, learn-ability, and usability.

1. Playability

Video games present a unique challenge when it comes to usability testing - the playability
factor. Standard usability tests focus on a user’s ability to complete a task successfully. Video
game usability testing, on the other hand, also needs to focus on the user’s experience while
completing the task and not just the success or failure. The ability to advance a level or beat
the game is not reason enough for the user to continue using the product. This concept
encapsulates enjoyment (or frustration) and ease of use.

       T-test Analysis: T-tests were performed on four correlating questions in the pre-test
        and post-test questionnaires. These questions evaluated a user’s opinion on the
        following topics: visual appeal, ease of navigation, ease of locating information, and
        ease of play. These questions were first asked after a user completed their exploration
        time. They were asked a second time after the user completed all of the tasks. We
        determined that “Ease of Play” was the only evaluator that dropped after task
        completion with a 95% confidence. This led us to believe that the game’s playability
        was below satisfactory levels.

       User Comments: Participant’s feedback was critical in evaluating the game’s
        playability. First, a user’s level of enjoyment needed to be assessed beyond what the
        numbers unveiled. User comments about the topic followed a solid trend - virtually
        every user expressed some level of frustration during the test. This is likely to be a
        leading factor in lowering a user’s enjoyment level. Second, many users were unclear
        about the goal of the game or how progress was measured. This added to user
        frustration because the game did not provide adequate guidance. By guidance, we
        refer to a game’s ability to help a player intuitively understand what “the next step”
        should be.




     Confidential                                  Page 21                                    9/25/2011
4004-549, 749 Usability Testing                           Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

2. Learn-ability

Once again, video games present a unique situation for usability testing. Standard products
need to be easy to learn and easy to master. A video game, on the other hand, needs to be
easy to learn but difficult to master. In order to evaluate a game’s mastery difficulty, more
extensive test would need to be performed. These tests would analyze a user’s performance
over an extended period of time and should yield a logarithmic graph in which a user’s rate of
progress slows as the game progresses. For purposes of this report, we are able to analyze
the short-term learn-ability of the game by comparing user time on similar tasks and by
comparing expert vs. novice performance.

       Task Analysis: Task 3 and task 4 of the usability test were extremely similar. Task 3
        asked the user to move two combat ships from one star system to another. Task 4
        asked the user to colonize a planet in another star system. This is done by sending a
        “colony ship” to that star system. In essence, the user is performing the same function
        with two different types of ships. Therefore, we can assume that completion time for
        task 4 should be significantly lower than that of task 3 if the game is learnable. This
        hypothesis is true. Mean task time for task 3 was 252.8 second. The mean task time
        for task 4 was 156.9 seconds – a drop of over 60%.

       Expert vs. Novice: From out analysis we found that expert gamers completed tasks
        within benchmark times at a much higher rate than novice users. At the highest
        extreme 100% of expert users completed task 3 in benchmark time, while 0% of
        novice users attained the same goal. To explain this, we revert to the client interview.
        Professor Bierre explained that, “people with exploration game experience... will
        probably find the game intuitive and will probably pick the game up in 5-10 minutes.
        Novice users who don’t have experience with this genre of games will probably take an
        hour to learn the game.” This presents a problem. While expert users were drawn in
        by their understanding of the complex structure of the game, novice users began to
        feel disengaged because of their lack of understanding. In this sense the game is not
        learnable because the initial spike in progress is not there. A novice user will struggle
        in the very beginning of the game – the part of the game that is supposed to be the
        easiest.




     Confidential                              Page 22                                9/25/2011
4004-549, 749 Usability Testing                           Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


3. Usability

A product’s usability is one of the core factors in determining the product’s overall success. In
order to evaluate this aspect of Master of Orion 2, we relied heavily on questionnaire results
and verbal user feedback.

       Visual Component: Users were mostly satisfied with the game’s visual appeal.
        However, numerous functionality issues surfaced as a result of interface flaws. Users
        became confused by the general layout of the main screen. The buttons along the
        bottom of the screen were overlooked by many users at first. Several users even said
        that they thought the buttons were simply labels. Similarly, the game status labels on
        the right side of the screen were thought to be buttons. This error can be attributed to
        poor button designed and placement.

       Feedback: Usability was also dwindled by a lack of responsiveness – the game
        provided an inadequate level of feedback. Several labels had mouse-over effects and
        most buttons did not. This caused confusion with users as they were unable to figure
        out which was which. Actions were also left unconfirmed. When a user instructed a
        ship to move they were often unsure if the action had taken place or not. Random
        windows popping up added to the confusion because users were unaware of whether
        they were the cause of the event.




     Confidential                              Page 23                                9/25/2011
4004-549, 749 Usability Testing                                Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


    4. Results

Data Analysis, Interpretation, and Recommendations


Summarize Performance Data


                                  Time on   Time on      Time on       Time on      Time on       Time on
         Participant              Task 1    Task 2       Task 3        Task 4       Task 5        Task 6

           Mean                    44.3      389.3         252.8       156.9          52.6         194.2
          Median                    45        330          212.5       112.5           44           135
           Mode                     45       #N/A           360          60            60           120
       Standard Dev                18.14     310.7        243.48       118.67         26.83        175.69


           Highest                  75       1040           840          390           120          660
           Lowest                   10        50             20           40            20           47

           Range                    65        990           820          350           100          613

         Quartile 1                 33        120            38           60           40           120
         Quartile 3                 60        590           360          240           60           185
   Inter Quartile Range             27        470           322          180           20            65

            Mean                   44.3      389.3          252.8       156.9         52.6         194.2
      Trimmed Mean
           (10%)                   44.75    350.375         208.5      142.375        48.25       154.375
    Standard Dev after
         trimming                  10.97    213.18        130.28        90.27         10.16        61.49




The primary tasks performed by the user can be subdivided into two categories: mission-critical tasks and
support tasks. Critical tasks include creating a ship, sending a ship to another star system, and colonizing a
planet. Support tasks include starting a new game and finding help within and outside the game.

With the above table in hand, we can note the following:-

    1) All participants were able to complete the tasks.
    2) Measure of Central Tendency:-
           a. Mean: While considering the complete data(including the outlier –if any), we can say the
                average time taken in seconds per task was
                     i. Task 1 : 44.3
                    ii. Task 2 : 389.3
                   iii. Task 3 : 252.8
                   iv. Task 4 : 156.9
                    v. Task 5 : 52.6
                   vi. Task 6 : 194.2




     Confidential                                   Page 24                                   9/25/2011
4004-549, 749 Usability Testing                               Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


            b. Trimmed Mean: As mean are highly sensitive to the outliers, trimmed mean will give us the
               better picture of the average time taken by the participant.
                     i. Task 1 : 44.75
                    ii. Task 2 : 350.375
                   iii. Task 3 : 208.5
                   iv. Task 4 : 142.375
                    v. Task 5 : 48.25
                   vi. Task 6 : 154.375

                 As we one can notice, trimmed means are much more accurate than normal mean, which
                 were disturbed by outliers.

    3) Measure of Dispersion:
          a. Range: Below are the range in seconds for each task
                   i. Task 1 : 65
                  ii. Task 2 : 990
                 iii. Task 3 : 820
                 iv. Task 4 : 350
                  v. Task 5 : 100
                 vi. Task 6 : 613
             The above data prompted us to investigate why there is a big difference between highest
             and lowest completion times of task 2, 3 and 6.

            b. Inter quartile Deviation: As we can see in the data above, extremes render the range un-
               descriptively. Therefore, calculating Inter Quartile Range will be more appropriate in order
               to get the times of the majority of the participants
                     i. Task 1 : 27
                    ii. Task 2 : 470
                   iii. Task 3 : 322
                   iv. Task 4 : 180
                    v. Task 5 : 20
                   vi. Task 6 : 65

                 We could see that task 6 in the previous point was affected by an outlier. But other tasks
                 still need be investigated of why they were producing such a long range.

            c.   Standard Deviation: Below is the standard deviation after trimming the mean by 10%.
                      i. Task 1 : 10.97
                     ii. Task 2 : 213.18
                    iii. Task 3 : 130.28
                    iv. Task 4 : 90.27
                     v. Task 5 : 10.16
                    vi. Task 6 : 61.49

                 Standard deviation gives us a more accurate and detailed estimate of the dispersion
                 because an outlier can greatly exaggerate the range (as true in our case). It allows us to
                 reach some conclusions about the time taken. Assuming that the distribution is normal or
                 bell-shaped (or even close to it!), we can conclude the following

                 For Task 1,
                        1. Approx. 69% of the users took time between 44.75 (+ or -) 10.97 i.e. between
                             33.78 and 55.72 seconds
                        2. Approx. 95% of the users took time between 44.75 (+ or -) 21.94 i.e. between
                             22.81 and 66.69 seconds.
                 For Task 2,



     Confidential                                 Page 25                                   9/25/2011
4004-549, 749 Usability Testing                            Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

                        1. Approx. 69% of the users took time between 350.375 (+ or -) 213.18 i.e.
                             between 137.195 and 563.55 seconds
                 For Task 3,
                        1. Approx. 69% of the users took time between 208.5 (+ or -) 130.28 i.e. between
                             78.22 and 338.78 seconds
                 For Task 4,
                        1. Approx. 69% of the users took time between 142.375 (+ or -) 90.27 i.e.
                             between 52.105 and 232.64 seconds
                 For Task 5,
                        1. Approx. 69% of the users took time between 48.25 (+ or -) 10.16 i.e. between
                             38.09 and 58.41 seconds
                        2. Approx. 95% of the users took time between 48.25 (+ or -) 20.32 i.e. between
                             27.93 and 68.57 seconds.
                 For Task 6,
                        1. Approx. 69% of the users took time between 154.375 (+ or -) 61.49 i.e.
                             between 92.89 and 215.87 seconds
                        2. Approx. 95% of the users took time between 154.375 (+ or -) 122.98 i.e.
                             between 31.395 and 277.355 seconds.




     Confidential                               Page 26                                9/25/2011
4004-549, 749 Usability Testing                           Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


       1200




       1000




        800

                                                                                   Q1
                                                                                   Lowest
        600                                                                        Median
                                                                                   Highest
                                                                                   Q3

        400




        200




           0
                 Task1       Task2      Task3    Task4      Task5     Task6
     Q1            33             120    38       60         40        120
     Lowest        10             50     20       40         20         47
     Median        45             330   212.5    112.5       44        135
     Highest       75         1040      840       390        120       660
     Q3            60             590   360       240        60        185




     Confidential                               Page 27                                 9/25/2011
4004-549, 749 Usability Testing                                      Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

Histograms of tasks are provided in order to obtain the trend.
Task 1
                                          Task 1 (Time vs Participant)

                   80



                   70



                   60



                   50
  Time (in secs)




                   40



                   30



                   20



                   10



                   0
                         1        2   3     4         5         6        7      8        9      10
                                                      Participant




                   Confidential                           Page 28                                9/25/2011
4004-549, 749 Usability Testing                                              Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

Task 2
                                          Task 2 (Time vs Participant)


                   1200




                   1000




                    800
  Time (in secs)




                    600




                    400




                    200




                      0
                          1       2   3     4          5                 6       7       8       9       10
                                                           Participant




                   Confidential                            Page 29                                       9/25/2011
4004-549, 749 Usability Testing                                              Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

Task 3
                                           Task 3 (Time vs Participant)


                    900



                    800



                    700



                    600
  Time ( in secs)




                    500



                    400



                    300



                    200



                    100



                      0
                          1        2   3     4         5                 6      7        8       9       10
                                                           Participant




                    Confidential                            Page 30                                      9/25/2011
4004-549, 749 Usability Testing                                             Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

Task 4
                                          Task 4 (Time Vs Participant)


                   450



                   400



                   350



                   300
  Time (in secs)




                   250



                   200



                   150



                   100



                    50



                     0
                         1        2   3     4         5                 6      7        8       9       10
                                                          Participant




                   Confidential                            Page 31                                      9/25/2011
4004-549, 749 Usability Testing                                              Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

Task 5
                                           Task 5 (Time Vs Participant)


                    140




                    120




                    100
  Time ( in secs)




                     80




                     60




                     40




                     20




                      0
                          1        2   3     4         5                 6      7        8       9       10
                                                           Participant




                    Confidential                            Page 32                                      9/25/2011
4004-549, 749 Usability Testing                                             Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

Task 6
                                          Task 6 (Time Vs Participant)


                   700




                   600




                   500
  Time (in secs)




                   400




                   300




                   200




                   100




                     0
                         1        2   3     4         5                 6      7        8       9       10
                                                          Participant




                   Confidential                            Page 33                                      9/25/2011
4004-549, 749 Usability Testing                                                                        Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

Overall Performance – grouped by tasks
                                                          Performance Grouped by Tasks

                 1200




                 1000




                 800
  Time in Secs




                 600




                 400




                 200




                   0
                           1                       2                       3                       4                       5                   6
                                                                                    Tasks

                               Series1   Series2       Series3   Series4       Series5   Series6       Series7   Series8       Series9   Series10




                 Confidential                                                     Page 34                                                           9/25/2011
4004-549, 749 Usability Testing                                Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


Task Accuracy:


Following figure shows the combined summary of task timings and task accuracy scores.

                                                                                     Finding
                         Start a                                           Using     help in
                         new            Building   Moving     Colonizing   in-game   User
 Tasks Title             game           a Ship     a ship     a Planet     help      manual

   Mean Time                44.3         389.3      252.8       156.9       52.6      194.2
 10% Trimmed Mean           44.75       350.375     208.5      142.375      48.25    154.375

 Tasks Benchmarks            55           125        90          135         75        150


 % of users
 performing
 successfully
 within the time
 benchmark                        70%      30%        30%          60%        90%        60%
 % of Novice
 forming the
 successful
 users                      28.57%      33.33%       0.00%      16.67%     33.33%     33.33%
 % of Experts
 forming the
 successful
 users                      71.43%      66.67%     100.00%      83.33%     66.67%     66.67%
 % of users
 performing
 successfully,
 regardless of
 time
 benchmark                    100%        100%       100%         100%       100%       100%




     Confidential                                   Page 35                                    9/25/2011
4004-549, 749 Usability Testing                           Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja



               Users who completed the task in benchmarks

   100%                                                90%
    90%
    80%        70%
    70%                                      60%                   60%
    60%
    50%                                                                       Users (%)
    40%                    30%    30%
    30%
    20%
    10%
     0%
                 1           2      3          4         5          6

Figure1. Users who completed their tasks in benchmarks (in %)




     Confidential                              Page 36                                9/25/2011
4004-549, 749 Usability Testing                               Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja




                           Break up of successfull users

   120
                                      100
   100
                                              83.33
    80       71.43
                         66.67                           66.67     66.67
                                                                               Novice Users
    60
                                                                               Expert Users
    40 28.57          33.33                           33.33      33.33

                                            16.67
    20
                                  0
      0
              1           2           3        4         5          6

Figure2. Break down of users who completed their tasks in respective benchmarks.




     Confidential                                   Page 37                               9/25/2011
4004-549, 749 Usability Testing                             Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


Summarize Reference Data

Pre-Test Questionnaire

1. Age: (15-19): 2
         (20-23): 5
         (24-28): 1
         (29-Up): 2


                                               Group by Age

               100%
                90%
                80%
                70%
                60%                             50%
                50%
                40%
                30%           20%                                              20%
                20%                                            10%
                10%
                 0%
                              15-19            20-23          24-28            29-up


2. Gender:
       __0__ - Female
       __10_ - Male


                                           Group by Gender
                                  100%
           100%
            90%
            80%
            70%
            60%
            50%
            40%
            30%
            20%
            10%                                                       0%
             0%
                                    Male                           Female




     Confidential                                 Page 38                               9/25/2011
4004-549, 749 Usability Testing                                             Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


3. What is your first language?

         __8__ - English
         __1__ - Spanish
         __1__ - Other – (please indicate) ___Chinese_____


                                                  Group by First language

                  100%
                   90%                   80%
                   80%
                   70%
                   60%
                   50%
                   40%
                   30%
                   20%                                                10%                      10%
                   10%
                    0%
                                        English                      Spanish                 Chinese


4. How would you rate your skills in playing video games?
       (Please circle one of the numbers below.)

  None                Beginner             Average (8)        Expert (2)
  | --------------- | ----------------- | --------------- |
  1                  2                     3                  4


                                            Group by skills in Video Games

                  100%
                   90%                           80%
                   80%
                   70%
                   60%
                   50%
                   40%
                   30%                                                                 20%
                   20%
                   10%
                    0%
                                               Average                                Expert




     Confidential                                                 Page 39                               9/25/2011
4004-549, 749 Usability Testing                               Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

5. Based on a one week time frame, how many hours do you spend playing video games?

        _____ - Less than 30 minutes
        __1__ - 30 minutes to an hour
        __3__ - 1 to 5 hours
        __4__ - 5 to 10 hours
        __2__ - More than 10 hours
        _____ - Not applicable


                                         Group by no. of Hrs spent

                    100%
                     90%
                     80%
                     70%
                     60%
                     50%                                          40%
                     40%                              30%
                     30%                                                     20%
                     20%                  10%
                     10%          0%                                                     0%
                      0%
                                  < .5   .5 to 1     1 to 5      5 to 10     >10         NA
                                                        (in Hours)


6. In the last 30 days, how many times have you played video games?

        _____ - This is my first time
        __1__ - 2 to 3 times
        __1__ - 3 to 5 times
        __3__ - 5 to 10 times
        __5__ - More than 10 times




     Confidential                                  Page 40                                9/25/2011
4004-549, 749 Usability Testing                                  Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


                           Group by no. of times played in last month

               100%
                90%
                80%
                70%
                60%                                                                      50%
                50%
                40%                                                       30%
                30%
                20%                         10%         10%
                10%          0%
                 0%
                              O          2 to 3         3 to 5           5 to 10         > 10


7. What types of video games go you like to play?

        __6__ - Role-Playing Games (RPG) (Star Wars Galaxies)
        __6__ - Strategy Games
                        __6__ - (TB) Turn Based (Master of Orion)
                        __6__ - (RTS) Real Time Strategy (Final Fantasy)
        __1__ - Simulation Games (Flight Simulator 2004: A Century of Flight)
        __4__ - Action (Battlefield 1942)
        _____ - Adventure Games (Unreal Tournament)
        _____ - First-Person Shooters (Gran Turismo)
        _____ - Puzzle games (Yu-Gi-Oh! The Dawn of Destiny)
        _____ - Sports games (Madden NFL)
        _____ - Other (please explain)



                                  Group by Type of Games experience

               100%                                           90%
                90%
                80%
                70%          60%        60%         60%                      60%
                60%
                                                                                           Novice
                50%      40%          40%         40%                              40%
                40%                                                                        Expert
                30%
                20%                                                10%
                10%
                 0%
                           Role     Turn Based Real Time      Simulator       Action
                          Playing    Strategy  Strategy




     Confidential                                 Page 41                                       9/25/2011
4004-549, 749 Usability Testing                                  Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

8. What platforms do you like to play electronic games?

        __8__ - Sony's PlayStation2,
        __4__ - Nintendo's GameCube
        __2__ - Microsoft's Xbox
        __9__ - personal computers,
        __3__ - handheld game units,
        __1__ - mobile phones,
        __7__ - public arcade game machines and
        _____ - other game units (please explain)



                                            Group by Platforms

                    100%                                 90%           90%
                     90%          80%           80%
                     80%                                        70%             70%
                     70%                60%
                     60%                  40%                                            Novice
                     50%                                         30%           30%
                     40%    20%                  20%                                     Expert
                     30%                               10%               10%
                     20%
                     10%
                      0%
                                                 be


                                                 ox




                                                  ts
                                                  rs




                                                 es
                                                  n




                                                 es
                                                io




                                               te


                                               ni
                                             Xb
                                             Cu




                                             on


                                            am
                                             at




                                             U
                                            pu




                                         Ph
                                          st




                                         t's




                                           e
                                          e




                                         G
                                         m

                                      am
                                       ay


                                      am


                                       of


                                     Co




                                     ile


                                       s
                                    Pl




                                    de
                                    os
                                    G




                                    G


                                  ob
                                  al




                                ca
                                icr
                     's




                                ld
                                  s




                                M
                               on
                  ny


                               o'




                             he




                             Ar
                              M
                           nd




                            rs
                So




                         nd




                          ic
                        Pe
                        te




                       bl
                      Ha
                        n




                     Pu
                     Ni




     Confidential                                     Page 42                                9/25/2011
4004-549, 749 Usability Testing                                                    Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

Analyze Data:
Above was the complete detail of the pre-test and post-test questionnaire. Now we will do the comparison
on some of the questions which were present in both. This way we can see how the user‟s perception
changed after playing the game Master of Orion II.




                                                            Pre-test       Pre-test       Post-test    Post-test
                                                            Data /40       in %           Data /40     in %

           Visual appeal                                              21          52.5            21        52.5
           Ease of navigation                                         16            40            14          35
           Easy to locate information                                 16            40            13        32.5
           Easy to play                                               21          52.5            14          35




                                                        Pre n Post Test Comparison

                                       100
                                        90
                Preference score (%)




                                        80
                                        70
                                        60   52.552.5                                     52.5
                                                                                                           Pre-test
                                        50                  40               40
                                        40                       35               32.5           35        Post-test
                                        30
                                        20
                                        10
                                         0
                                             Visual         Ease of          Easy to   Easy to play
                                             appeal        navigation         locate
                                                                           information


        Figure1. Pre n Post test comparison of 10 users preferences.




     Confidential                                                     Page 43                                      9/25/2011
4004-549, 749 Usability Testing                               Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

T-Test Analysis

We can easily see that there is fall between 18% to 0% between
To further strengthen the statement that after playing the game, users impression about the game lower
down, we will perform a t-test for comparing the data

Null Hypothesis Test: There is no change in user‟s opinion before and after playing the game.

Confidence Level: 95%

Visual Appeal:
 t-Test: Paired Two Sample for Means

                              Variable 1         Variable 2
 Mean                                   2.1                2.1
 Variance                     0.766666667         0.544444444
 Observations                            10                 10
 Pearson Correlation          0.842700972
 Hypothesized Mean
 Difference                              0
 Degree of freedom                       9
 t Stat                                  0
 P(T<=t) one-tail                      0.5
 t Critical one-tail          1.833113856
 P(T<=t) two-tail                        1
 t Critical two-tail          2.262158887

Ease of Navigation:
 t-Test: Paired Two Sample for Means

                              Variable 1       Variable 2
 Mean                                   1.6            1.4
 Variance                     0.266666667     0.266666667
 Observations                            10             10
 Pearson Correlation          0.666666667
 Hypothesized Mean
 Difference                              0
 Degree of freedom                       9
 t Stat                                1.5
 P(T<=t) one-tail             0.083925328
 t Critical one-tail          1.833113856
 P(T<=t) two-tail             0.167850656
 t Critical two-tail          2.262158887




     Confidential                                 Page 44                                 9/25/2011
4004-549, 749 Usability Testing                               Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

Easy to locate information:
 t-Test: Paired Two Sample for Means

                                   Variable 1        Variable 2
 Mean                                      1.6                 1.4
 Variance                         0.266666667         0.266666667
 Observations                               10                  10
 Pearson Correlation              0.666666667
 Hypothesized Mean
 Difference                                  0
 df                                          9
 t Stat                                    1.5
 P(T<=t) one-tail                 0.083925328
 t Critical one-tail              1.833113856
 P(T<=t) two-tail                 0.167850656
 t Critical two-tail              2.262158887


Easy to Play:
 t-Test: Paired Two Sample for Means

                                   Variable 1     Variable 2
 Mean                                      2.1            1.4
 Variance                         0.322222222    0.266666667
 Observations                               10             10
 Pearson Correlation              0.606478435
 Hypothesized Mean
 Difference                                 0
 df                                         9
 t Stat                           4.582575695
 P(T<=t) one-tail                 0.000661475
 t Critical one-tail              1.833113856
 P(T<=t) two-tail                 0.001322951
 t Critical two-tail              2.262158887


From all the data above, we could see, only “Easy to play” have a P-value lower than 0.05. . So we cannot
reject the null hypothesis for “Visual Appeal”, “Ease of Navigation” and “Easy to locate information”. But for
the last case, which is “Easy to play”, we can reject the null hypothesis and conclude that users found the
more difficult that theirs expectations.




     Confidential                                  Page 45                                   9/25/2011
4004-549, 749 Usability Testing                                            Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

Learn-ability:

Task 3 and task 4 of the usability test were extremely similar. Task 3 asked the user to move two combat
ships from one star system to another. Task 4 asked the user to colonize a planet in another star system.
This is done by sending a “colony ship” to that star system. In essence, the user is performing the same
function with two different types of ships. Therefore, we can assume that completion time for task 4 should
be significantly lower than that of task 3 if the game is learnable. This hypothesis is true. Mean task time for
task 3 was 252.8 second. The mean task time for task 4 was 156.9 seconds – a drop of over 60%.

Below is the comparison between Task3 and Task 4


                                                    Learnability

                     900                                                     840
                     800
                     700
   Time ( in secs)




                     600
                     500
                                     390                   360                                      360
                     400                                     300 295
                     300           245                                         240
                           180                                                         185
                                                                                     160
                     200     120                                      105
                                              69    3860                                               60
                     100                    20                                               3040
                       0
                             1        2       3       4       5        6        7      8       9      10
                                                               Users

                                                           Task 3    Task 4


                     Figure2. Comparison between time taken for task 3 and task 4 of all users.




           Confidential                                      Page 46                                   9/25/2011
4004-549, 749 Usability Testing                              Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


Null Hypothesis Test: It takes more time to complete Task 4 than it takes to complete Task 3

Confidence Level: 90%

 t-Test: Paired Two Sample for Means

                                                       Variable
                                        Variable 1         2
 Mean                                          252.8      156.9
 Variance                              59279.51111 14081.66
 Observations                                     10          10
 Pearson Correlation                   0.450495252
 Hypothesized Mean Difference                       0
 df                                                 9
 t Stat                                1.393965211
 P(T<=t)                               0.098392043
 t Critical                            1.383028803
Note: Variable 1 represents Task 3 and Variable 2 represents Task 4.

In the data above, we could note that 90% confidence level we can see that P-value is lower than 0.09. So
we can reject the null hypothesis can conclude that Task 4 takes less time than task 3 and this leads to that
the game is Learnable.




     Confidential                                 Page 47                                   9/25/2011
4004-549, 749 Usability Testing                           Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja


Recommendations


Based on the quantitative and qualitative data, we are able to make several recommendations
that will improve the game’s ease of use and user satisfaction with the product. These
suggestions concentrate on feedback, navigation, immersion, and support.

1. Feedback

Feedback is one of the most critical aspects of game play. It can make or break a user’s
experience with the product by either providing, or not providing, the user with intuitive clues
that help them progress.

       Tool Tips: Because of the level of complexity that inherently exists in a game such as
        this, providing explanations of key components of the interface in a non-intrusive
        manner is critical. To accomplish this, we recommend a toolbox that appears when you
        roll over a component. This toolbox needs to provide relevant information about the
        component in question and disappears when the user moves the mouse away.

       Use of Sound: The game made very little use of sound as a feedback mechanism.
        The only sound present throughout the game is a repeating background sound clip. No
        sounds are used to provide any kind of feedback. We highly recommend changing this.
        Adding a sound for button clicks or task completion (such as the completion of a new
        ship) would help engage the user and provide a more fulfilling game experience.

       Production Status: One of the most commonly encountered problems was task
        completion awareness. Users were often unsure of whether or not they completed a
        task that required turns (such as building or relocating a ship). This should have been
        a foreseeable problem because of the nature of the game. Very few games are turn
        based and developers should have incorporated an added level of feedback to make
        the user comfortable with such an environment. Currently, research progress,
        production progress, and travel progress are all displayed in different areas of the
        interface and on different screens. We recommend adding a “Progress Status” window
        that would be accessed via a button from the main menu. This window will list all
        projects and the number of turns each project has until completion. This will give users
        a centralized place to monitor all turn-based activity. Such a component would prove
        even more valuable as the game progresses because of the increase in production and
        travel.

2. Navigation

Throughout the usability tests, users consistently encountered navigation problems. Several
users commented about feeling “lost” in the interface. When this occurs a user’s interest in
the game severely decreases as a result of the frustration felt by the user.

       Button Consistency: Many users confused buttons with labels and vice versa. This
        happened for several reasons. First, buttons often blended into the interface because
        they were the same color as everything else – grey. One user was even unable to find
        the “Turn” button – the most important component of the game. We recommend
        choosing a color that stands out more to draw the user’s attention to the button.
        Second, buttons in different areas of the interface did not always have the same



     Confidential                              Page 48                                9/25/2011
4004-549, 749 Usability Testing                           Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja

        design. This needs to be corrected. By providing the user with the same design every
        time, you lower the odds of a button being misinterpreted to be a label.

       Mouse-over: In order to isolate buttons from the interface even further, a mouse-
        over effect should be added. This could be as simple as a change in test or background
        color. More importantly, there are non-button clickable items. For example, a star may
        be click in order to bring up information about that star system. Users are not made
        aware of what aspects of the interface are clickable and what aspects aren’t. To
        reprimand the situation, we recommend a change in the mouse cursor whenever it is
        brought over a clickable item (similar to how the cursor in a web browser changes
        when you move it over a link).

       Flexibility: MOO2 presented a unique problem when it came to task design – there
        are multiple ways to complete each task. For example, ships may be relocated using
        functionality in the main screen and also in the “Fleets” screen. Because of the game’s
        complexity adding redundant ways to complete a task increases the likelihood of a
        user finding the solution. We feel that this needs to be expanded. For example, one
        user tried moving ships by dragging them from one star system to another. This did
        not work because the proper way of completing the task is to first click the ships you
        wish to send and then click the system you wish to send them to. However, the action
        seems logical and should be an alternate way of completing the task. We recommend
        a follow-up study, the purpose of which is to determine the most common ways in
        which users attempt to complete tasks on their first attempt. By figuring out the most
        common logical approaches, the developers would create a game that is much more
        intuitive and learnable.

3. Immersion

As has been stressed earlier, engaging or “sucking in” a user is absolutely critical to making a
successful video game. This can only be accomplished by keeping the user interested in
uncovering a future state of the game. The desire to reach that state is what keeps them
coming back. This entails creating a catchy storyline, presenting the user with a clear mission,
and keeping the user informed of their progress. A game like MOO2 should have no problem
achieving this state because of its genre and intricate storyline. Unfortunately, this is not the
case.

       Introduction Movie: Surprisingly, almost all of the users watched the introductory
        trailer in its entirety. The feedback about the video was fairly consistent - “it looks nice
        but it doesn’t tell me anything about the game.” The trailer should introduce the user
        to the game, but should also provide information about the storyline and game
        objective. This will help engage the user from the beginning by giving them a goal to
        strive towards.

       Game Pace: Novice users often complained about being bored. They attributed the
        feeling to frustration with tasks and a slow game pace. The pace issue is inherent in
        strategy games since they are concentrated around thoughtful decision making rather
        than reactive hand-eye coordination or timing. The fact the MOO2 is turn-based makes
        this even more of an issue because time stands still when a decision is being made or
        orders are being given. Real-time strategy games (such as WarCraft) don’t have as
        much of an issue with game pace because there are always events happening. We
        recommend blending the turn-based concept with several real-time components (such
        as production or population growth) to keep the user on their toes and engaged.




     Confidential                              Page 49                                9/25/2011
4004-549, 749 Usability Testing                           Team 4: Master of ORION II Battle at Antates
Chris Lister, Larry Conrow, Steve Shapiro, and Shiranchal Taneja




4. Support

MOO2 is a very complex game. Because of this, the game’s support systems need to be very
effective in providing the user with information. While several usability tasks specifically
tested this portion of the game, many users also resorted to using help systems on earlier
tasks. These systems can be significantly improved.

       Tutorial Mode: When a user starts a new game, they are presented with several
        difficult levels: Tutor, Easy, Medium, and Hard. The concept of tutor mode is deceptive
        because it is not an actual tutorial. It is just the name they gave to the easiest play
        level. Perhaps they were on the right track. Once a user started a new game and took
        a look at the extensive screen of buttons, stars, and status labels, many users
        responded with, “now what do I do?” A game of such complexity should have a 5-10
        minutes tutorial level that takes the user, step by step, through the basic functions of
        the game. It will make the goal of the game more clear by suggesting what initial
        steps should be taken after starting a new game.

       Terminology Tutorial: In order to achieve an immersive atmosphere in a game such
        as this, developers need to concentrate on functionality and a compelling story line. In
        essence, you are creating a virtual world – one that needs currency, measuring
        standards, and other details that add to the realism of the situation. This result is an
        enormous amount of jargon that needs to be understood by the user. For example, the
        currency in MOO2 is “B.C.” - a term that one user was confused by. Similarly, distance
        is measure by “parsecs,” which is also a term that is unique to this game. In order to
        assist users in learning and understanding this terminology, we recommend a glossary
        section be added to both the in-game help and the user manual.

       User Manual: Several issues were identified with the user manual. First, it had no
        table of contents or index. When a user was asked to look an item up, the only way to
        complete the task was to flip through the entire manual, which is over 100 pages.
        These two elements must be added for the user manual to be effective. Second, users
        complained that the manual was not well organized. Aspects of game-play are
        described throughout the manual rather than in one organized section. We recommend
        reorganizing the manual so that all game-play components are explained in the same
        portion of the manual.

       In-Game Help: In-game help provides valuable information but the interface for the
        system is very poor. A list on the left has items that help can be obtained on. A box on
        the right shows help about the currently selected item. To select an item, a user
        simply needs to hold their mouse over it. However, the label changes color at that
        point making most users think that it is a button. Many users attempted to click the
        label but received no feedback. Creating a standard look for all labels and buttons will
        solve this problem. The in-game help section also lacked search capability – an option
        that is crucial in helping users find the information they are looking for.




     Confidential                              Page 50                                9/25/2011

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:10
posted:9/25/2011
language:English
pages:50