Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

ds_report

VIEWS: 14 PAGES: 96

									CSCW to Go




         SI 689 Final Report
        Nintendo DS Evaluation
         Professor Gary Olson

              Prepared by
           CSCW to Go Team
Josh Morse, Xiaomin Jiang, Cheng-Lun Li,
   Chris Demeniuk, Satyendra Nainwal
SI 689 Final Report                                                                                      Nintendo DS Evaluation


                                                   Table of Contents
Table of Figures ............................................................................................................... 4
Table of Tables ................................................................................................................. 5
Overview ........................................................................................................................... 6
  Summary of the Report................................................................................................... 6
  History of the DS............................................................................................................ 7
  DS Features ................................................................................................................... 8
Literature Review .......................................................................................................... 10
Comparative Evaluation ............................................................................................... 11
  Overview ...................................................................................................................... 11
  Findings ....................................................................................................................... 11
Narrowing Our Scope.................................................................................................... 14
Interviews ....................................................................................................................... 15
  Overview ...................................................................................................................... 15
  Results .......................................................................................................................... 15
       Level of DS experience ....................................................................................................... 15
       Fun ...................................................................................................................................... 15
       Frustration ........................................................................................................................... 17
Heuristic Evaluation ...................................................................................................... 19
 Overview ...................................................................................................................... 19
 Heuristics ..................................................................................................................... 19
       Gaming Experience Heuristics............................................................................................ 19
       Mobility Heuristics.............................................................................................................. 21
       Gameplay Design Heuristics............................................................................................... 21
   Heuristic Evaluation for Selected Games .................................................................... 24
       Mario Kart:.......................................................................................................................... 24
       Metroid Prime Hunters:....................................................................................................... 29
       Elite Beat Agents: ............................................................................................................... 34
       Clubhouse Games: .............................................................................................................. 39
User Testing.................................................................................................................... 44
  Overview ...................................................................................................................... 44
  Test design: .................................................................................................................. 44
  Data collection: ........................................................................................................... 45
  Subject Recruitment ..................................................................................................... 46
  User Test Setup ............................................................................................................ 46
  User Test Procedure .................................................................................................... 48
Results and Analysis ...................................................................................................... 50
  Overview ...................................................................................................................... 50
  Quantitative Data Analysis .......................................................................................... 50
       Effects to Fun ...................................................................................................................... 50
       Effects to Performance ........................................................................................................ 52
  Qualitative Data Analysis ............................................................................................ 53
Conclusion ...................................................................................................................... 55
  Recommendations ........................................................................................................ 55
  What We Learned......................................................................................................... 56
  Final Words ................................................................................................................. 56


 CSCW to Go                                                                                                               Page 2 of 96
SI 689 Final Report                                                                Nintendo DS Evaluation


Appendix A: Interview Protocol................................................................................... 57
Appendix B: CSCW To Go Interview Result ............................................................... 59
Appendix C: Usability Testing Script and Questionnaires....................................... 75




 CSCW to Go                                                                                     Page 3 of 96
SI 689 Final Report                                                                     Nintendo DS Evaluation



                                           Table of Figures

Figure 1: Original Game Boy ............................................................................................ 8
Figure 2: Game Boy Color................................................................................................. 8
Figure 3: Game Boy Advance SP ...................................................................................... 8
Figure 4: Original DS and DS Lite (Images on this page from Wikipedia) ...................... 8
Figure 5: Sony PSP (Picture from Wikipedia)................................................................. 12
Figure 6: Nokia N-Gage QD (Picture from Wikipedia) .................................................. 12
Figure 7 - Mario Kart DS (images from IGN.com) ......................................................... 24
Figure 8 - Metroid Prime Hunters (images from IGN.com)........................................... 29
Figure 9 - Elite Beat Agents (images from IGN.com)..................................................... 34
Figure 10 - Clubhouse Games (images from IGN.com).................................................. 39
Figure 11: population (yellow area) and sample frame (pink + blue) of test .................. 46
Figure 11: Our "Mobile Usability Lab" ........................................................................... 47
Figure 12: Positioning of cameras for user testing .......................................................... 47
Figure 13: Positioning of test subject and facilitator ....................................................... 48
Figure 14: Interaction between Game Type and Distance............................................... 51




 CSCW to Go                                                                                           Page 4 of 96
SI 689 Final Report                                                            Nintendo DS Evaluation



                                        Table of Tables

Table 1: Comparison of Features (DS, PSP, N-Gage)..................................................... 13
Table 2: Reasons for having fun when playing Single-player Game .............................. 16
Table 3: Reasons for having fun when playing Multi-player Games .............................. 17
Table 4: Reasons for being frustrated when playing Multi-Player Games...................... 18
Table 5: Heuristic Severity Rating for Mario Kart .......................................................... 25
Table 6: Heuristic Severity Rating for Metroid Prime Hunters....................................... 29
Table 7: Heuristic Severity Rating for Elite Beat Agents................................................ 35
Table 8: Heuristic Severity Rating for Clubhouse Games............................................... 40
Table 9: The Effects of Independent Variables to Fun .................................................... 51
Table 10: The Effects of Independent Variables to Performance.................................... 52
Table 11: Themes of Participants’ Conceptual Models................................................... 53




 CSCW to Go                                                                                 Page 5 of 96
Overview
The Nintendo DS is currently the most popular mobile video game system in the world.
At first glance, the Nintendo DS may not seem like a system that fits the paradigm of
Computer Supported Cooperative Work. However, as mentioned in the first class
lecture, each letter in the CSCW acronym isn’t necessary for a system to fall under the
CSCW umbrella. The DS would probably be best classified as Computer Supported
Cooperative Play, and it is a rich source of investigation for CSCP.


Summary of the Report
To evaluate the Nintendo DS, our team used a number of user-centered evaluation
methods familiar to SI 622. Initially, we focused on gaining a better understanding of
the Nintendo DS, building on our previous knowledge and refining our scope until the
final usability tests. We used the following evaluation methods:

   •   Literature Review: we investigated a number of academic and non-academic
       sources to learn more about mobile gaming, the Nintendo DS, and how current
       users use their DS systems.

   •   Comparative Evaluation: we compared the DS with its closest competitor, the
       Sony PSP, and with the Nokia N-Gage. These results gave us an overview of
       how the DS fit in with its competition and why it is so successful.

   •   Contextual Interviews: we interviewed several current DS users to find out how
       they used their Nintendo DS system. This gave us some useful information, and
       helped us narrow our scope to focus on multiplayer games.

   •   Heuristic Evaluation: we used 15 Nokia heuristics for mobile gaming to
       evaluate four different DS games in depth. This helped us choose the best game
       for our usability testing, and to inform us as we designed the usability tests.

   •   Usability Testing: the culmination of our study, our usability tests focused on
       multiplayer gaming with the DS. Users completed a number of specified tasks
       with the DS, and the quantitative and qualitative results of the testing informed
       our final recommendations and conclusion.

After conducting our usability tests and analyzing the quantitative results, we found
several interaction effects between two variables. Our qualitative results and interviews
helped explain these effects, and we made several recommendations based on our
findings, including the following:

   •   Ensure that users of similar skill levels are matched against each other
   •   Release more games with a cooperative multiplayer element
SI 689 Final Report                                                  Nintendo DS Evaluation


      •   Create better ways for players at a distance to communicate, such as synchronous
          text and perhaps audio message.


History of the DS
Nintendo made its first venture into the portable video game market in 1989 with its
release of the Gameboy. The original Gameboy was large and heavy by today’s mobile
gaming standards, and was powered by four AA batteries. It had a mere 8 kilobytes of
RAM, a 4 MHz processor, and a 160 x 144 pixel screen that could display for shades of
green1. It retailed for $89.99, and, along with the Gameboy Color, had sold over 118
million units by 2005.

Nintendo relied on the Gameboy for almost 10 years, but as competition for competitors
like Sega increased, Nintendo began releasing more updates. The first was the Game
Boy Color in 1998, which sported a color screen, and 8 MHz processor, and a more
compact design. This was followed in 2001 by the Game Boy Advance, which had a
16.8 MHz processor and a higher resolution screen2. The Game Boy Advance had two
purely cosmetic updates: The Game Boy Advance SP in 2003, which had a built in
lithium ion battery, a backlit screen, and a smaller, clamshell design, and the Game Boy
Micro in 2005, which weighed a mere 80 grams but still had the same hardware as the
original Game Boy Advance3.

While these releases had allowed Nintendo to dominate the portable gaming market, it
was losing in the home console market, where Sony’s Playstation 2 had achieved a large
market share compared to Nintendo’s Gamecube. As Sony prepared its own mobile
system, Nintendo went back to the drawing board, and eventually released the Nintendo
DS in 2004. In 2006, Nintendo released the DS Lite, which added a brighter screen and
a more compact design4. Sales of both DS systems have totaled over 58 million5.




1
  “Game Boy”. Wikipedia. Accessed December 7, 2007. <http://en.wikipedia.org/wiki/Gameboy>
2
  “Game Boy Advance”. Wikipedia. Accessed December 7, 2007.
<http://en.wikipedia.org/wiki/Game_Boy_Advance>
3
  “Game Boy Micro”. Wikipedia. Accessed December 7, 2007.
<http://en.wikipedia.org/wiki/Game_Boy_Advance>
4
  “Nintendo DS”. Wikipedia. Accessed December 7, 2007. <http://en.wikipedia.org/wiki/Nintendo_DS>
5
  “Hardware Sales”. VG Chartz. Accessed December 7, 2007. <http://vgchartz.com/>


    CSCW to Go                                                                  Page 7 of 96
 SI 689 Final Report                                                 Nintendo DS Evaluation




Figure 1: Original Game Boy   Figure 2: Game Boy Color     Figure 3: Game Boy Advance SP




              Figure 4: Original DS and DS Lite (Images on this page from Wikipedia)




 DS Features
 The Nintendo DS has a number of features that set it apart from its predecessors. It
 borrows the clamshell design sported by the Game Boy Advance SP, but it adds a
 second screen to the bottom of the clamshell, and moves the speakers to the top screen.
 Visual feedback from the two screens and auditory feedback from the speakers allow the
 user to interact with the system.

 The most interesting features of the DS involve its input mechanisms. Like most
 portable gaming systems, the DS includes a control pad on the left and several buttons
 on the right. There are also two buttons on the outer edge of the system. More
 interestingly, the bottom screen is touch sensitive, allowing users to interact with the


  CSCW to Go                                                                    Page 8 of 96
SI 689 Final Report                                                       Nintendo DS Evaluation


system in very precise ways. The DS also sports a small microphone, and can recognize
simple voice commands, though it does not have the processing power to handle
complex speech. Put together, “touch screen and voice recognition technology made for
a style of play unlike anything else on the market6”. Using these input styles, users can
train virtual pets to recognize their name in the game Nintendogs, or allow users to
quickly scribble answers to puzzles in Brain Age.

The DS also has wireless networking capabilities. There is a proprietary network
protocol that allows the DS to communicate with other DS systems at a short range.
This protocol serves a variety of purposes. DS users can use a built in program called
Pictochat to scribble messages and drawings and send them to other DS users in range.
Users can also connect with other DS users in range to play a variety of different games
together. Using a feature called Download Play, users who don’t own a particular game
can download a smaller version of the game and play together. Finally, DS owners can
bring their DS to Nintendo kiosks and download limited versions of new DS games to
try out before purchasing the full game7. Combined with Download Play, these kiosks
allow Nintendo to advertise their games without spending a significant amount of
resources. Thus, the wireless capability supports communication, entertainment, and
marketing.

The DS also has the ability to connect to a Wi-fi network8. Various games, such as
Mario Kart DS, can be played over the Wi-Fi connection service, a free service provided
by Nintendo9. This allows users to connect with anyone around the world. In the game
Animal Crossing, Wild World, users create a virtual village which their avatar can
interact with. Other players can then visit another user’s village with their own avatar
through the Wi-Fi Connection, creating a sort of limited Massively Multiplayer Online
Game. The DS also supports a specialized version of the Opera Web browser, which
has been adjusted to support the DS hardware. Users can view the actual-size version of
a web page on the top screen, and a miniaturized version on the bottom screen, which
then can interact with using the stylus.

All of these hardware capabilities would be meaningless without software, but the DS
sports a library of over 500 games and applications, many of which take advantage of
the hardware in interesting ways. We have already mentioned several games and
applications, such as Nintendogs and Pictochat, which take advantage of the hardware
platform. Many of our evaluation methods, including the literature review, interviews,
and heuristic evaluation, were used to explore which games and applications provided
the most enjoyable experience for our users.

6
  Blakely, Rhys. “Wii are swimming in a clear blue ocean”. Times Online, July 12, 2007.
http://business.timesonline.co.uk/tol/business/industry_sectors/technology/article2063714.ece. Accessed
27, September 2007.
7
  “Nintendo DS”. Wikipedia. http://en.wikipedia.org/wiki/Nintendo_DS. Accessed 27, September 2007.
8
   “Nintendo Wi-Fi service reaches 5 million users”. Cnet News, May 30, 2007.
http://www.news.com/8301-10784_3-9723617-7.html?part=rss&subj=news&tag=2547-1_3-0-5
9
   “Nintendo DS gets VoIP”, Engaget.com, Sep 13, 2007
http://www.engadget.com/2007/09/13/nintendo-ds-gets-voip


    CSCW to Go                                                                        Page 9 of 96
SI 689 Final Report                                                                Nintendo DS Evaluation



Literature Review
Since the DS is not the typical workplace CSCW application, we had to explore other
avenues to discover how users were using the system. A particularly fruitful area was
online forums, particularly for major gaming sites like IGN and Gamespot. These
forums had many postings related to common gaming activities like multiplayer gaming,
and allowed us our first glimpse into what games were popular. We could also search
the forums for topics like “Pictochat” to find specific information on how users were
using specific applications and features of the DS. This information was important to us
as we drew up our interview script so that we knew what to ask DS users about their
playing habits.

Throughout our testing, we often consulted Wikipedia. While Wikipedia proved to be a
useful resource for general information about the DS and related systems, it was also the
best listing we found of different games for the DS. Wiki pages were available that both
listed all currently released DS games and all DS games that supported multi-player
through the Wi-Fi network. These lists were especially useful as we looked to evaluate
games later in our heuristic analysis.

Finally, we did spend some time looking for academic articles related to our study. We
used the ACM Digital Library for most of this research, as it had a collection of useful
articles and conference proceedings related to mobile devices. Unfortunately, the only
articles we found that related directly to the DS were very technical in nature, dealing
with issues like the underlying technology in the DS’s wireless networking system. We
did find several useful articles on mobile gaming in general, however, which helped
influence the setup and analysis for our usability tests later on1011.




10
   Methodological considerations: Experience clip: method for user participation and evaluation of mobile concepts,
Minna Isomursu, Kari Kuutti, Soili Väinämö, Proceedings of the eighth conference on Participatory design: Artful
integration: interweaving media, materials and practices
11
   Towards the perfect infrastructure for usability testing on mobile devices, Rudy Schusteritsch, Carolyn Y. Wei,
                     07
Mark LaRosa. CHI ' extended abstracts on Human factors in computing systems


 CSCW to Go                                                                                     Page 10 of 96
SI 689 Final Report                                             Nintendo DS Evaluation



Comparative Evaluation
Overview
As previously mentioned, the Nintendo DS was released to compete with Sony’s first
portable system, the Playstation Portable. As the Sony PSP is the only successful
dedicated gaming device currently on the market, it seemed a natural choice for our
comparative evaluation. Released only months after the Nintendo DS, it seemed like a
straight-forward, head-to-head market match-up.

Based on our reviews of literature on mobile gaming, particularly from the ACM digital
library, it was obvious that mobile phones were seen by many as an important portable
gaming platform. Indeed, even the most basic mobile phones usually support playing
simple games, and more powerful smartphones are capable of running games similar to
the Nintendo DS. Our team debated briefly about what type of phone to test, but we
decided, in the end, to focus on the Nokia N-Gage. Though the N-Gage was
discontinued in 2005, it was a fully functional smartphone that was designed to both
make calls and play games. Furthermore, it was released before both the DS and the
PSP, though it never had the market traction of either system.


Findings
Our team researched the specifications of each system online, including size, memory
capacity of games, and CPU speed. We also found sale price and units sold to date,
which have been summarized in Figure 7 below. We also obtained both a Nokia N-
Gage QD and a Sony PSP. The QD was a revision of the original N-Gage. There is also
a revised version of the Sony PSP known as the PSP Slim, but we were not able to
obtain a PSP Slim for our comparative evaluation.

Looking at the specifications, it seems that the DS is the underdog, particularly in
relation to the PSP. While the PSP has a single screen, it has a much higher resolution
than either of the Nintendo DS screens. The PSP has a processor that is five times as
fast as the DS’s fastest CPU, and its UMD disks can hold more than 10 times as much
information as the DS flash cartridges. The N-Gage has components more in line with
the DS, with a smaller screen but a slightly faster processor. While the DS is only able
to run Pictochat and play games, however, the PSP and DS have many more functions.
The PSP can play videos and music, and has a built in web browser. The N-Gage can be
used to make phone calls, and has a built-in calendar, web browser, alarm, calculator,
and many other small applications that are often found on phones.

The fact that the DS has less features and less processing power than its competitors may
make it seem like it would lose out in the marketplace, but sales figures tell a different
story. With 57 million sales, the DS has sold twice the number of units as the PSP, and
has left the N-Gage, with just 2 million sales, in its dust. Though the DS is priced more



 CSCW to Go                                                               Page 11 of 96
 SI 689 Final Report                                                 Nintendo DS Evaluation


 competitively than the PSP, we didn’t believe the price told the whole story, so we
 decided the try the units out for ourselves.

 One thing we quickly noticed when holding the PSP was its size; even though it is only
 about an inch longer, the size difference is very noticeable. In our literature review, we
 encountered other users who said that the extra inch made the difference between fitting
 it in one’s pants and having to use a jacket pocket. Ergonomically, the PSP was less
 comfortable to hold than the DS. More issues arose as we tried to use the PSP, and
 found a complicated series of menus. These menus were obviously designed to allow
 the user to access the multimedia functionality, but we had a difficult time simply
 starting up our game. The DS, on the other had, immediately begins a game if a
 cartridge is inserted.

 The N-Gage QD was actually quite comfortable ergonomically, though online users said
 that the original N-Gage was an ergonomic nightmare, and poor initial reviews may
 have made it difficult for the redesigned N-Gage QD to gain traction. The menu system
 for the N-Gage was actually easier to use than the menus on the PSP, and inserted game
 cards could be loaded with a single button press from the home screen. Unfortunately,
 the screen on the N-Gage was too small to take advantage of features like web browsing,
 and the size of the device, while small, was still larger than many cellphones on the
 market today.

 In the end, it seemed to us that the DS succeeded because of the lack of additional
 features. It played games, and played them very well, particularly with its touch screen
 and microphone inputs. The PSP and N-Gage both suffered from clunky menus that
 were necessary to some degree to access their additional features. The DS was also easy
 to fit into a pocket, and the clamshell protects the screens when traveling. The DS
 seemed to be both a better gaming system and more portable, which seems to have
 helped its sales. The DS also has very compelling software titles, though we were not
 able to conduct surveys to do a statistical comparison of how DS software ranked
 against PSP or N-Gage software titles.




Figure 5: Sony PSP (Picture from Wikipedia)   Figure 6: Nokia N-Gage QD (Picture from Wikipedia)




  CSCW to Go                                                                   Page 12 of 96
SI 689 Final Report                                                  Nintendo DS Evaluation



                   Nintendo DS Lite          Sony PSP Slim              Nokia N-Gage QD
Release Date       November 2004             March 2005                 October 2003
Screen Size        Two 256x190 pixel,        480x272 pixel,             176x208 pixel,
                   3 inches each             4.3 inches
Weight             275 grams                 189 grams                  143 grams

Wireless           802.11b WiFi              802.11b WiFi               GPRS cellular
Processor          ARM at 67&33 MHz MIPS at 333 MHz                     ARM at 104 MHz
Storage            Custom flash              UMD – 1.8 GB               MMC flash –
                   Up to 128 MB              Flash – up to 8GB          up to 512 MB
Extras             Pictochat                 Videos, music, web         Phone, WAP web
                                             browser                    browser
Total Sales12      57 million                27 million                 2 million
Current Price      $129                      $169.99                    $99 (discontinued)

                       Table 1: Comparison of Features (DS, PSP, N-Gage)




12
  “Hardware Sales”. VG Chartz. Accessed December 7, 2007. <http://vgchartz.com/>. Other
information from Wikipedia.


 CSCW to Go                                                                     Page 13 of 96
SI 689 Final Report                                            Nintendo DS Evaluation



Narrowing Our Scope
By this point in our study, our team had begun to narrow our project focus. We quickly
realized that evaluating the Nintendo DS was akin to trying to evaluate a PC; while we
could simply focus on the hardware, it is the software that is truly compelling.
Consequently, we narrowed our focus to three possible software applications for the DS
that fit the CSCW paradigm:

   •   Pictochat, a built-in program similar to today’s instant messaging systems.
       Pictochat allows DS users to communicate with other DS users over the short-
       range wireless network using both text and sketches drawn on the touchscreen.

   •   DS web browser. Opera created a web-browsing application that does several
       interesting things to try to overcome the limited resolution of the DS’s screens,
       such as providing a zoomed-out view on one screen and a zoomed-in view on the
       other. The web browser uses the Nintendo DS Wi-Fi connection.

   •   Multiplayer games. There are over 500 games available for the DS, and many of
       them support some degree of multiplayer functionality, either through the short-
       range wireless network or over the Internet using Wi-Fi.

Our team decided not to make our final decision until after conducting our interviews,
when we would focus our heuristic evaluation and usability testing on one of the issues
above. Still, the results of our comparative evaluation and literature review had us
leaning towards multiplayer games. As a result, we decided that if we eventually chose
multiplayer games, we wanted to use our usability testing to measure how different
variables affected the level of fun reported by users. In particular, we selected the
following three variables initially:

   •   Distance, particularly as a binary variable. The values would be co-location in
       the same physical space vs. being located in different areas.

   •   User interaction in the game. This would also be a binary variable, with values
       of competitive games, where the users play against each other, or cooperative
       games, where they work together on a team.

   •   Learning curve of the game. At first we decided to use this as a variable to
       measure users’ enjoyment, but quickly realized that it would only really work if
       users did not have significant experience with the game to begin with. Later on,
       we adjusted this to simply be a user’s level of experience with the game.

These issues strongly influence the interview script that we used for our interviews.
After our interviews, we decided that Pictochat and the web browser were not nearly as
commonly used as multiplayer games. As multiplayer gaming on the DS seemed to be a
little-studied activity, we eventually decided to focus our scope on that issue.

 CSCW to Go                                                              Page 14 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


Interviews
Overview
To examine how the variables we decided to focus on would have effect on users’ level
of enjoyment and frustration, we decided to conduct interviews in the last week of
October. We designed an interview protocol (see Appendix A) emphasizing on
following four aspects of DS use:

   •   Level of DS experience
   •   Type of games usually played
   •   Our multiplayer variables (Internet vs. short-range network play in particular)
   •   Use of non-Game features: PictoChat and Opera Browser

We sent out recruiting emails to si.all.open@umich.edu in mid October and scheduled
seven interviews in the last week of October, yielding 6 students in the School of
Information and 1 middle school student. Our results from interviews are analyzed
below and complete interview results can be found in Appendix B.


Results



Five of our interviewees had own his/her DS for more than six months, including two
people who owned two DS systems for more than one year. All of them knew at least
five people with whom they could play, those who were in more of an advanced level
knew more than ten other people who owned a DS. Their frequency of playing DS
varied, but regularly according to individual own schedule. In all, almost 60% of our
interviewees were advanced users, from which we could optimistically assume that we
were able to obtain rich information about uses of DS.

Moving on to non-gaming feature use of the DS, we found out only two of those
interviewed have ever used PictoChat and four of them didn’t even know what
PictoChat was. Only two have ever used the Opera web browser, but not often. From the
interview data, it seems likely that people want to use a gaming device primarily for
game playing rather than other uses.



Single-player Game

Interviewees spent more than 60% of their time playing single-player games on the DS,
and the games they chose are easy to learn and play. Why interviewees thought playing
single-player games is fun is summarized in the table below.



 CSCW to Go                                                               Page 15 of 96
SI 689 Final Report                                                     Nintendo DS Evaluation


Question                                 Reason             Quote
Where are you most likely to use your    Interactive        U04: Mario kart is the favorite—
DS?                                                         interactive, more incentive to play

What single game is your favorite to                        U06: Now, Zelda. The most is Sudoko,
play single-player? Why?                                    EBA. I like the action you can do with the
                                                            touch screen. There are many neat ways
What aspects of single-player play do                       for input that you want to try. Also used
                              t
you enjoy? What aspects don'you                             the mic in the games
enjoy?
                                                            U07: Mario kart is the favorite to play
Could you describe the last time you                        individually because it is fun and
played a game by yourself?                                  competitive
                                         Convenience        U03: I don’t have much time this
    i.   What was fun about the                             semester. So I like games with low
         experience? Why?                                   commitment.

                                                            U04: One is the space. I can play
                                                            wherever I want. I’m not constrained by
                                                            the environment.

                                                            U05: (I play) On bus, after studying
                                                            (break). Faster than starting a console
                                                            like PS3.

                                                            U06: WaIting for something. Like when I
                                                            as in the bank. It’s a long process.
                                                            I’d say it’s good that we can play right
                                                            away, anywhere
                                         Skill training     U02: Not always to kill time, sometimes
                                                            dedicate an hour or two to achieve a
                                                            specific goal.

                                                            U03: I don’t have much time this
                                                            semester. So I like games with low
                                                            commitment. I like brain age because it
                                                            helps me in memory.

                                                           U04: About the brain age—it’s good to
                                                           test yourself how fast you can remember
                                                           things—kinda skill training thing.
                 Table 2: Reasons for having fun when playing Single-player Game




Multi-player Game

All of our interviewees have ever played multi-player games. However, the advanced
users spend more time on playing multi-player games than average users or novice
users. The factors that affect their level of enjoyment are varied but being social and
interacting with other players are two main reasons. Sample questions and the reasons
for fun are summarized in the following table.



 CSCW to Go                                                                         Page 16 of 96
SI 689 Final Report                                                    Nintendo DS Evaluation


Question                                   Reason          Quote
What games have you played with            Social          U01: Yes. Brother also has it.
another person over the short-range
wireless network?                                          U03: Yes, 2~3 times (overall) before. I
                                                           played that with my friends’ kids.
What single game is your favorite to
play with another person over the short-                   U07: You can play with friends and
range wireless network? Why?                               interact with them.
                                           Interaction     U01: You get to see their expression. Get
What aspects of multi-player play the                      to see their experience when they lose! Its
short-range wireless network do you                        fun to see them
                         t
enjoy? What aspects don'you enjoy?
                                                           U02: Enjoy seeing the reactions of other
Could you describe the last time you                       people – adds more enjoyment to the
played a game with another person over                     experience. Can make things quite funny.
the short-range wireless network?                          Also can see people being impressed.
                                                           Usually him cursing and the other person
      i.   What was fun about the                          laughing, still a fun experience.
           experience? Why?
                                                           U04: Mario Kart. It is competitive. I
What do you find to be different when                      curse when playing. I think it’s fun to say
playing single-player rather than multi-                   “I BEAT U”. I like competitive games
player?                                                    when playing with others

                                                           U07: You can play with friends and
                                                           interact with them.

                                           Easy-to         U01: It (Animal crossing) is really simple
                                           Learn/Play      to set up.

                                                           U05: Easy – don’t have to set up router
                                                           and access – just have to turn on and
                                                           connect. Very convenient. Need more
                                                           short range games

                  Table 3: Reasons for having fun when playing Multi-player Games




Single-player game

The reasons for being frustrated when playing single-player game are also varied, with
examples being that a user’s skill level was so low that the user couldn’t progress fast,
the game itself is time-consuming, the game itself doesn’t allow saving at the midway
point, lack of interaction with other players, etc. It is likely that why users felt frustrated
usually depends on the games they were playing.

Multi-player game

The reasons for being frustrated when playing multi-player games can basically grouped
into social factors and technical factors. Similar to the reasons for “having fun when


 CSCW to Go                                                                         Page 17 of 96
SI 689 Final Report                                                        Nintendo DS Evaluation


playing single-player games”, the lack of social interaction with other players is a major
reason that users talked about in their frustrating experience. Another major reason is
found that users prefer to play with someone with a similar skill level, not only the
advanced users, but also beginner users as well. Sample questions and the reasons for
frustration are summarized in the following table.

Question                                    Reason             Quote
Have you ever played a multi-player         Interaction        U02: Can’t really chat/communicate over
game over the short-range wireless                             the Internet. Only interacting through
network (with a nearby person)?                                your character (make your character jump
                                                               and down to show happiness). Can’t
If so, how often do you play multi-player                      communicate your excitement about
games over the short-range wireless                            playing together.
network?
                                                               U04: I curse when I play=) playing with
What aspects of multi-player play the                          someone I don’t know can’t let me have
short-range wireless network do you                            more reactions than that
                         t
enjoy? What aspects don'you enjoy?
                                                               U07: In short range you know the person
Could you describe the last time you                           and can interact with him. Over the
played a game with another person over                         internet you mostly never know who you
the short-range wireless network?                              are playing against
                                            Skill Level        U01: When people leave if they are
      ii. What was frustrating about                           getting easily beaten. If you are the only
          the experience? Why?                                 one who is not good. Metroid is this
                                                               common and not very nice experience.
What do you find to be different when
playing single-player rather than multi-                       U02: Losing. That’s about it.
player?
                                                               U07: Will be much better to get a
                                                               compete profile of the person doing it.
                                                               Also important to find someone close to
                                                               your skill level
                                            Technical:         U01: Mario cart, wi fi animal crossing.
                                            Waiting time,      You need friend codes for it which can be
                                            Friend code, etc   a little ….. I traded a bunch when I had
                                                               pokemon

                                                               U02: Downside – online play doesn’t
                                                               have progression, and since you can’t see
                                                               opponent, you don’t have the same
                                                               experience as when playing short range.
                                                               Also waiting to play.

                                                               U03: I don’t like to wait for the system. I
                                                               like to be able to do other stuff or another
                                                               game before the game is ready. The
                                                               loading time.

                                                             U07: Waiting time for connecting to the
                                                             game.
               Table 4: Reasons for being frustrated when playing Multi-Player Games




 CSCW to Go                                                                             Page 18 of 96
SI 689 Final Report                                                         Nintendo DS Evaluation


Heuristic Evaluation

Overview

Mobile Games are played in a variety of contexts and hence we need heuristics designed
to study the effectiveness of the device and the game design in supporting those contexts
of use. These context include outdoor and indoor play, handing interruptions while play,
handling connectivity issues during play and various other factors. In addition there are
other issues which arise because of the combination of the game and the device – for
example the game screen will shake and move because the controls are embedded in the
device as opposed to TV or computer based games. Also there have to be indicators both
for the device status –e.g. battery life, connectivity as well as indicators for game
statistics which make for some interesting challenges13.

We studied various heuristics and decided to focus on a subset of Nokia’s Mobile Game
Playability Heuristics14 which we felt would best test our platform in the context of the
parameters we had set out to study.

After making a long list of heuristics we got together and rated each heuristic with
respect to the relevance to our study and decided to finally focus on a subset of 15
heuristics. Since the experience of the device changed with each game and the
parameters we wanted to study – co-location vs. distance, competitive vs. co-operative,
learning curve – were all more functions of the games than the platform so we decided
to focus mainly on heuristics for game design.

The 15 heuristics we selected were distributed across Game Playing Experience,
Mobility and Game Design Experience. Given below are the heuristics and their
description taken from the Nokia Guidelines:

Heuristics



     1. Screen layout is efficient and visually pleasing

        Designing a good layout is not always easy. The layout should present all
        necessary information for the player, but on the other hand, if the screen is filled

13
   Korhonen, H. and Koivisto, E. M. 2006. Playability heuristics for mobile games. In Proceedings of the
8th Conference on Human-Computer interaction with Mobile Devices and Services (Helsinki, Finland,
                                      06,
September 12 - 15, 2006). MobileHCI ' vol. 159. ACM, New York, NY, 9-16. DOI=
http://doi.acm.org/10.1145/1152215.1152218
14 Nokia Forums: Mobile Game Playability Heuristics

 http://sw.nokia.com/id/5ed5c7a3-73f3-48ab-8e1e
631286fd26bf/Mobile_Game_Playability_Heuristics_v1_0_en.pdf


 CSCW to Go                                                                            Page 19 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


       with all kinds of information, it starts to look crowded. It is important that the
       player finds the navigation controls and they should not be mixed with the
       information that needs to be visible on the screen. In general, this heuristic is
       specifically important for mobile games due to limited screen space. Designing
       the layout for a mobile-phone screen can be challenging, but a good rule of
       thumb is that information that is frequently needed should be visible to the player
       all the time.

   2. Indicators are visible

       The player should see the information that is required for being able to play the
       game. An example of this kind of information could be the status of a game
       character. Information that is frequently needed should be visible for the player
       all the time — if possible. The player should always know the current state of the
       game, for example, whose turn it is to make the next move. Indicators for critical
       game play information should be presented to the player clearly enough – e.g.
       energy of the character or the time left for game to finish.


   3. Navigation is consistent, logical, and minimalist

       The navigation on a mobile device is not easy because of the small screen
       and limited input devices. Mobile devices have two kinds of navigation
       controls: permanent and temporary navigation keys. Permanent
       navigation keys should be used primarily for navigation. Temporary
       navigation keys are often related to applications or to a specific user
       interface style. Since the games do not necessarily need to follow the
       device’s user interface style, the use of these keys can be more flexible.
       In the menu the navigation should be intuitive and have shortest paths to
       options because they are easier to remember and reach.


   4. Control keys are consistent and follow standard conventions & flexible

       Using common conventions reduces the time that is needed to learn to use any
       software application including game navigation. Game devices usually have
       specific keys for certain actions and every game should follow them. Failure of a
       game to implement this results in frustration as users automatically use the
       wrong key from their habit of the platform.


   5. The game gives feedback on the player’s actions

       A good user interface has a low response time on the player’s actions. An action
       can be either a single key press or a more complicated input sequence. The
       player should notice immediately that the game has recognized the action by


 CSCW to Go                                                               Page 20 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


       providing feedback. The most common way of providing feedback is to present it
       graphically. Other alternatives are to use audio or tactile feedback. Providing
       only auditory feedback is not acceptable since a player may be playing the game
       without sounds. In mobile devices, the network connection is usually
       considerably slower than in other game platforms and it can create latency in the
       response time, which can worsen the problem.


   6. The player cannot make irreversible errors (Can players still be competitive
      after a mistake)

       The game should confirm actions that can cause serious and irreversible damage.
       Also, when mistakes happen, it is helpful to enable recovery. In games, making
       errors is often part of the game play. However, this heuristic deals with errors
       that are related to the bad usability of the game user interface. Errors often
       happen when the player deletes game objects, such as avatars or items.
       Sometimes, however, the errors can be related to positive things that must be
       done, but the time is just not right.

   7. The game contains help

       The game teaches the player what he or she needs to know to start playing the
       game. The players do not often read manuals, and a mobile game does not
       usually even have a paper manual. Even though a tutorial mode at the beginning
       of the game is usually helpful, having a complete tutorial is not well suited for a
       mobile game.



       The game and play sessions-including multiplayer- can be started quickly

       The players should be able to start game sessions quickly and easily, preferably
       in less than five seconds. A play session of a mobile game is usually shorter than
       a play session of a computer or console game. Browsing in a game menu
       consumes precious game-playing time and actions that need to be done
       frequently in the game should not be hidden behind a long navigation path either.



   1. The game provides clear goals or supports player-created goals

       The players should be able to understand the goals that exist in the game. The
       goals can be either set by the game or the players themselves. Some games
       encourage player-created goals, and some allow the players to choose their goals
       from a set of pre-defined goals. An example of supporting player-created goals
       could be providing the player information that makes him or her curious enough


 CSCW to Go                                                                Page 21 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


       to want to do something, for instance, seeing a distant island and figuring out that
       there must be a way to get there somehow. There should be the right distribution
       of short and long term goals.

   2. The players are rewarded and rewards are meaningful

       The players should be rewarded as they progress in the game. The most
       important rewards should be meaningful for the player. The reward should be
       adjusted to the challenge that the player had to face in order to get the reward. If
       the player expects a bigger reward based on previous experience, and if he or she
       gets a smaller one, he or she will be disappointed. The rewards should be
       frequent and yet not too predictable.

   3. The first-time experience is encouraging

       The first five minutes in the game create the first impression of the game, which
       is very difficult to change. The players should feel that they have accomplished
       something and be rewarded. If the first-time experience is discouraging, the
       player may never play the game again. The first play session should make the
       player desire for the next play session and this is intimately tied to a right
       learning curve for the game.

   4. There are no repetitive or boring tasks

       The game should not require repetition of tasks without changing any conditions
       if the game is not in the training phase. Repeating the same tasks over and over
       again is often called tread milling or grinding, and it is usually a guaranteed way
       of killing the fun in the game. However, this may be necessary in the training
       phases to develop skills.

   5. The players can express themselves

       The players should be able express themselves by, for instance, customizing their
       characters, acting in a certain way in the game world, or modifying the game
       world. Allowing the player to change the game world increases the feeling of
       ownership. Players should also be able to express their state of mind through
       some actions of the character in multiplayer games.

   6. The game supports different playing styles

       The players of a game can vary a lot in terms of both experience and preferred
       play styles. In multiplayer games it also becomes important whether the game
       supports co-operative play, competitive play or both and how does the
       experience change with both of these.




 CSCW to Go                                                                Page 22 of 96
SI 689 Final Report                                               Nintendo DS Evaluation


   7. The game uses orthogonal unit differentiation

        The different game objects should have different kinds of purposes. Harvey
        Smith has introduced the term "Orthogonal Unit Differentiation". This means
        that the units, for instance character classes, in the game should be designed so
        that they are functionally different.

        A simple example of orthogonally different items would be arrows that do
        normal damage and poison damage. Orthogonally similar items would be arrows
        that do 1-3 points of damage or 2-4 points of damage. Orthogonal unit design can
                                                        s
        encourage strategic play and expands the game' possibility space.

Based on these heuristics we studied four popular Nintendo DS Games and assigned
them ratings based on a 5 point severity scale. It is important to note that most of the
ratings were on the lower side of the severity scale because these four games are
amongst the most popular and enduring Nintendo DS games and as such usually do a
good job in creating a usable and enjoyable gaming experience.

The Games we studied were:

   1.   Mario Kart
   2.   Metroid Prime Hunters
   3.   Clubhouse Games
   4.   Elite Beat Agents.

We defined our severity scale levels (SL) as follows:


                        Severe usability problem; immediate action needed.


                        Major usability problem; high priority.


                        Minor usability problem; low priority.


                        Tertiary usability problem; priority depends on resource
                        availability.

                        Not a usability problem.




 CSCW to Go                                                                 Page 23 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


Heuristic Evaluation for Selected Games


Mario Kart is a racing game where players can pick up and use a variety of colorful
items on their opponents.




                      Figure 7 - Mario Kart DS (images from IGN.com)




 CSCW to Go                                                               Page 24 of 96
SI 689 Final Report                                                          Nintendo DS Evaluation


                                         Heuristics                                             Severity
Screen layout is efficient and visually pleasing                                                   0
Indicators are visible                                                                             0
Navigation is consistent, logical, and minimalist                                                  0
Control keys are consistent and follow standard conventions & flexible                             1
The game gives feedback on the player’s actions                                                    1
The player cannot make irreversible errors (Can players still be competitive after a mistake)      2
The game contains help                                                                             1
The game and play sessions can be started quickly especially in multiplayer mode                   0
The game provides clear goals or supports player-created goals                                     0
The players are rewarded and rewards are meaningful                                                1
The first-time experience is encouraging                                                           0
There are no repetitive or boring tasks                                                            0
The game supports different playing styles (focus competitive vs. cooperative)                     0
The players can express themselves in multiplayer games to others                                  2
The game uses orthogonal unit differentiation                                                      1
                            Table 5: Heuristic Severity Rating for Mario Kart




    1. Screen layout is efficient and visually pleasing

(SL=0)

     • Use big and cute buttons. Easy to understand
     • It is Mario! Of course it is visually pleasing



    2. Indicators are visible

(SL=0)

     • Good, you can see back arrow and okey buttons mode

    3. Navigation is consistent, logical, and minimalist

(SL=0)

     • The flow of the game and navigation is easy.



    4. Control keys are consistent and follow standard conventions & flexible

(SL=1)


 CSCW to Go                                                                              Page 25 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


    • Similar to other racing game. Therefore it is easy to learn
    • Players cannot change control schemes.
    • The game didn’t provide when to use the stylus



   5. The game gives feedback on the player’s actions

(SL=1)

    • Yes, there is clear transition and sound indicating the changes made by players
    • Popup windows shown for confirmation about players’ actions.



   6. The player cannot make irreversible errors (Can players still be competitive
      after a mistake)

(SL=2)

    • In all courses of racing mode, there is no way for players to restart a certain race.
       The “quit” option leads users to restart the whole game. However, there is no
       warning signs and the action is irreversable



   7. The game contains help.

(SL=1)

    • No, there is no help although there are time trials which help you get used to the
      game and its controls and the game is easy and intuitive to get used to.

       The game and play sessions can be started quickly especially in multiplayer
       mode.

(SL=0)

   •     Yes, the game can be started quickly in both the single and the multiplayer mode.
         It has a search online friend option. The game automatically searches and adds
         players who are searching for opponents too and notifies you if the multiplayer
         mode is on. There are two modes for adding multiple players Simple and
         Normal.

   •     In the normal mode you compete only with players who have a Mario DS.


 CSCW to Go                                                                 Page 26 of 96
SI 689 Final Report                                                Nintendo DS Evaluation


   •     In the Simple mode you can compete even with players who do not have a Mario
         Kart DS.

   9. The game provides clear goals or supports player-created goals.

(SL=0)

   •     The game supports the setting and clear goals. The goal is to finish the race and
         the number of laps in the race is defined.
   •     There is also a clear indication of about the time when each round will end.
   •     The game ends when the time is over and the winner is the person with the
         highest number of points wins.

   10. The players are rewarded and rewards are meaningful.

(SL=1)

   •     There are not many rewards periodically (at least in the basic race version). The
         primary reward is just when you complete the race.
   •     In other variations of the race you have objects which you can collect which are
         shining stars which give you points. You can also collect other items like banana
         peels and shells which you can use on your opponents.
   •     The reward structure is such that even if a player is ahead and he is not careful he
         can lose the stars and thus makes the game more exciting.

   11. The first-time experience is encouraging.

(SL=0)

   •     The first time experience is very encouraging. It is a simple game to start playing
         immediately and in the easy level very easy to beat the computer bots which
         makes the play encouraging for novices.
   •     The game controls are natural and thus feeling comfortable in the very first race
         is not difficult that makes it an instant hit with new players as well as
         experienced ones.

   12. There are no repetitive or boring tasks.

(SL-0)

   •                  t
         There aren'repetitive task as the whole game is a race and you have to be alert at
         all moments in case you lose your lead if you are coming first and otherwise to
         try to beat the person ahead of you. In both cases you cannot afford to slack off
         and the game is not boring.



 CSCW to Go                                                                  Page 27 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


   •   In other variations of the game you have to be alert about the things others might
       throw at you and at the same time try to get the coins from the others, making no
       instant boring.

   13. The game supports different playing styles (focus competitive vs.
       cooperative).

(SL=0)

   •   Mario Kart does a good job of supporting different player styles. Players can
       race against other AI bots (or players in multi-player) in several grand prix races.
   •   Different classes (50cc, 100cc, etc) allow players of different experience levels to
       be challenged. Players can also race against the clock, with no other karts
       present.
   •   Finally, players can participate in a battle mode against other AI units or players.
       Players can choose from three difficulty levels for the AI bots. In the battle
       mode, players can also battle on teams which introduce a cooperative aspect.
       These three modes and different difficulty levels allow for a variety of racing
       enthusiasts to participate in the game.

   14. The players can express themselves in multiplayer games to others.

(SL=2)

   •   The players can do some basic movements like jumping up and down, but no
       actual communication during multiplayer game over the Internet.
   •   In the single player game, players can choose from different Mario characters for
       their driver, and choose tracks to race on.
   •   Overall, not a lot of customization.

   15. The game uses orthogonal unit differentiation.

(SL=1)

   •   Many of the items that players can use against other players have some
       affordance. For example, banana peels are dropped on the ground and cause
       other players to spin out, while shells can be thrown and speed along the ground
       at other players.
   •   Other items are still differentiated, though have weaker affordance – for
       example, mushrooms cause a burst of speed.

Different characters have slight differences – for example, larger characters are slower to
accelerate and poor at taking corners, but have high max speed and can cause damage to
smaller characters.




 CSCW to Go                                                                Page 28 of 96
SI 689 Final Report                                                          Nintendo DS Evaluation




Metroid Prime Hunters is a first person shooter and adventure game where players take
the role of a bounty hunter to fight alien space pirates.




                      Figure 8 - Metroid Prime Hunters (images from IGN.com)




                                         Heuristics                                             Severity
Screen layout is efficient and visually pleasing                                                   0
Indicators are visible                                                                             1
Navigation is consistent, logical, and minimalist                                                  0
Control keys are consistent and follow standard conventions & flexible                             3
The game gives feedback on the player’s actions                                                    0
The player cannot make irreversible errors (Can players still be competitive after a mistake)      0
The game contains help                                                                             1
The game and play sessions can be started quickly especially in multiplayer mode                   2
The game provides clear goals or supports player-created goals                                     2
The players are rewarded and rewards are meaningful                                                1
The first-time experience is encouraging                                                           3
There are no repetitive or boring tasks                                                            0
The game supports different playing styles (focus competitive vs. cooperative)                     0
The players can express themselves in multiplayer games to others                                  2
The game uses orthogonal unit differentiation                                                      2
                     Table 6: Heuristic Severity Rating for Metroid Prime Hunters

    1. Screen layout is efficient and visually pleasing

 CSCW to Go                                                                              Page 29 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


(SL=0)

    • Yes, generally the layout is like all other first-person shooting games. The
      graphic is nice



Indicators are visible(SL=1)

    • There is no indicator about what each meter on the screen means. The game
      doesn’t tell players how to use them either

   2.
   3. Navigation is consistent, logical, and minimalist

(SL=0)

    •



   4. Control keys are consistent and follow standard conventions & flexible

(SL=3)

    • The control scheme of the game is consistent
    • The mapping of the button to the actions is unintuitive. It is really hard to make
      complex actions
    • Players cannot change control scheme



   5. The game gives feedback on the player’s actions

(SL=0)

    • Yes, the game provide good feedback through popup screen and text windows
      (shown in the “commend window”)



   6. The player cannot make irreversible errors (Can players still be competitive
      after a mistake)

(SL=0)


 CSCW to Go                                                               Page 30 of 96
SI 689 Final Report                                                Nintendo DS Evaluation


    • Players are allowed to quit the mission if they choose to. Therefore they can get
      out if stuck at anywhere in the game.



   7. The game contains help.

(SL=1)

    • No, there is no explicit help but there is a “controls” options which gives you an
       overview of all the controls used in the game which is a useful aid.
    • In case of an error in wi-fi connection, the game suggests calling the Nintendo
       helpline in the US or Canada or visiting the general Nintendo website (not even
       a special help page) which is not very useful.

       The game and play sessions can be started quickly especially in multiplayer
       mode.

(SL=2)

   •     Not very easy to add multiple players. The game goes into wi-fi setup mode
         when going into multiplayer mode which is quite confusing.
   •     If there is a problem in the wi-fi connection then says it cannot connect, gives the
         helpline number and a website address for non US users!

   9. The game provides clear goals or supports player-created goals.

(SL=2)

   •   Clear goals are not supported in the game. The main objective of the game is
       clear but intermediate goals are difficult to figure out for a beginner.

   10. The players are rewarded and rewards are meaningful.

(SL=1)

   •   It is hard to figure out the rewards at first. However, there are bonuses that boost
       your energy power, and there are pitfalls that can make you lose that power. The
       rewards in this sense are meaningful once you can make sense of them.

   11. The first-time experience is encouraging.

(SL=3)

   •     The first time experience is not very encouraging. It is difficult to figure out what
         to do next as a novice players keep moving around in the same area not knowing

 CSCW to Go                                                                   Page 31 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


       what to do in the game.
   •   When the player does nothing meaningful for a while except for navigating in the
       maze no hints are provided by the system which leaves the new player with
       nothing to do and with no way to learn how to play the game.

   12. There are no repetitive or boring tasks.

(SL=0)

   •   Being a first person shooter game the game is initially not repetitive as you try to
       increase your level in the game. However, after a while if you are only playing
       with the computer the stages you have crossed can get a little repetitive.
   •   This is however not a problem if you are playing with multiple players since that
       makes the game exciting as moves of other human players are not as predictable
       as the moves and positions of computer bots.

   13. The game supports different playing styles (focus competitive vs.
       cooperative)

(SL=0)

   •   Metroid Prime offers 2 major options for players – a single player adventure
       mode and a “multiplayer” mode. In the adventure mode, the player controls the
       character, Samus, in the first person view. Going through a pre-constructed
       story, the player fights enemies, solves puzzles, and gathers information to
       complete the game. The difficulty level is fixed – the adventure mode might be
       too easy or too difficult.
   •   The “multiplayer” mode isn’t necessarily multiplayer, as a player can substitute
       the computer-controlled opponents, AI bots (whose difficulty level can be
       selected from 4 options). While the multiplayer modes are confined to smaller
       areas, there are a number of types of games to choose from, from simple death
       matches where the players try to kill each other, to capture the flag, which
       introduces an element of defense.

        Players can also team up in two vs. two battles. In online play, only the death
       match option is available.

   14. The players can express themselves in multiplayer games to others

(SL=2)

   •   In multiplayer, limited options to communicate. You can choose your name and
       select from a list of avatars – either Samus (the hero) or one of the bounty
       hunters you fight in the single player game.
   •   In single player, you play as Samus, who is a popular character that many players
       identify with.

 CSCW to Go                                                                Page 32 of 96
SI 689 Final Report                                            Nintendo DS Evaluation


   15. The game uses orthogonal unit differentiation

(SL=2)

   •   Energy power-ups are represented by small circles, but it is easy to forget what
       does what.
   •   Certain switches that allow doors to open can blend in with the background and
       be missed.
   •    Weapon power-ups are hard to recognize at times, though they can be
       differentiated from energy and other power-ups.




 CSCW to Go                                                              Page 33 of 96
SI 689 Final Report                                           Nintendo DS Evaluation


     !     "

Elite Beat Agent is a rhythm game where the user taps the DS screen in time with beats
of songs with colorful stories playing in the background.




Figure 9 - Elite Beat Agents (images from IGN.com)




 CSCW to Go                                                             Page 34 of 96
SI 689 Final Report                                                          Nintendo DS Evaluation



                                         Heuristics                                             Severity
Screen layout is efficient and visually pleasing                                                   0
Indicators are visible                                                                             1
Navigation is consistent, logical, and minimalist                                                  1
Control keys are consistent and follow standard conventions & flexible                             1
The game gives feedback on the player’s actions                                                    0
The player cannot make irreversible errors (Can players still be competitive after a mistake)      1
The game contains help                                                                             1
The game and play sessions can be started quickly especially in multiplayer mode                   0
The game provides clear goals or supports player-created goals                                     3
The players are rewarded and rewards are meaningful                                                1
The first-time experience is encouraging                                                           3
There are no repetitive or boring tasks                                                            0
The game supports different playing styles (focus competitive vs. cooperative)                     2
The players can express themselves in multiplayer games to others                                  2
The game uses orthogonal unit differentiation                                                      0
                         Table 7: Heuristic Severity Rating for Elite Beat Agents

    1. Screen layout is efficient and visually pleasing

(SL=0)

    •    The displaying screen and the touch screen shows the avatars as one screen
         without feeling the game objects are broken into two separate parts.
    •    Each agent is assigned appealing appearance and each game session are
         colored attractively
    •    The "mission selection" page is designed as a dragable screen to display several
         game sessions (3D view)

    2. Indicators are visible

(SL=1)

    •    The energy bar locating on the top of the screen in each game session reminds
         the player his/her possible remaining time of current game session, but players
         may always focus on touching the spots in a timely manner and find the session’s
         end so soon without noticing how many spots he/she has missed.
    •    The avatar("agent") works better in demonstrating the energy level than the
         energy bar since it will show as being exhausted when the energy falls quite
         closed to zero

    3. Navigation is consistent, logical, and minimalist

(SL=1)



 CSCW to Go                                                                              Page 35 of 96
SI 689 Final Report                                                 Nintendo DS Evaluation


   •     The depth of the navigation before a game session is reasonable. Although each
         game session goes with a scenario which will take several seconds to
         display/describe before actually starting the session, it helps set up a context for
         players. A "skip" message locating at the lower right corner allows users
         to choose to go into the game directly.

   4. Control keys are consistent and follow standard conventions & flexible.

(SL=1)

   •     The Game requires touching the screen to take certain actions so traditional
         control keys are not actually used.

   5. The game gives feedback on the player’s actions.

(SL=0)

   •     In addition to graphically showing the points gained when players hit a given
                                               s
         spot in a timely manner, the avatar' reactions also demonstrate the successful
         level of players' actions
   •     Auditory feedback plays particularly well in this game since players need to
         listen to the music to follow the right tempo for receiving feedback for their
         status
   •     There seem to be several scenarios get involve in each game session. For
         instance, if the player plays well, the scenario will continue with some positive
         small stories between "chapters", otherwise, the scenario will show some
         negative scenes
   •     A "Mission Failed" message shows at the end of each lost game session
   •     A "review" function is provided to show the player the last few seconds in the
         game session he/she just finished.

   6. The player cannot make irreversible errors (Can players still be competitive
      after a mistake)

(SL=1)

   •     Making errors is part of the gameplay. No major errors relating to the bad
         usability of the game user interface are found.

   7. The game contains help.

(SL=1)

   •     EBA does contain a tutorial, however it’s not readily apparent to a novice user,
         who would gain the most benefit from it.



 CSCW to Go                                                                   Page 36 of 96
SI 689 Final Report                                               Nintendo DS Evaluation


   8. The game and play sessions can be started quickly especially in multiplayer
      mode.

(SL=0)

   •     The game starts rapidly from the time the DS is powered on, until the time the
         game loads.
   •     Not intuitive that a user must touch the touch-screen with the stylus to end the
         intro video

   9. The game provides clear goals or supports player-created goals.

(SL=3)

   •     It is not intuitive to a novice user what they should be doing during the game
   •     There is little “forgiveness” or “play” with the beat-tapping
   •     A user is informed that something is wrong because the agents are killed off, but
         it’s not clear what they are doing wrong.
   •     The demo, which addresses these issues, isn’t very noticeable from the startup
         screen

   10. The players are rewarded and rewards are meaningful.

(SL=1)

   •     The rewards for a novice would be not killing the agents, if not attaining a high
         score. It is not clear that a user can also unlock new levels by doing well.

   11. The first-time experience is encouraging

(SL=3)

   •     For a novice user, the game objectives and goals are not intuitive or clear.
   •     The rewards are so difficult to obtain, even at an easy level, that a user may want
         to give up.
   •     The tutorial is difficult to find, and may often be looked over

   12. There are no repetitive or boring tasks.

(SL=0)

   •     The storyline and songs are constantly changing by level, keeping the repetition
         to a minimum.
   •     The tapping task can be repetitive; however this is the point of the game



 CSCW to Go                                                                  Page 37 of 96
SI 689 Final Report                                                Nintendo DS Evaluation


   13. The game supports different playing styles (focus competitive vs.
       cooperative)

(SL=2)

   •     Out of the games reviewed, EBA probably has the least flexibility. There are a
         total of four difficulty levels, though the two hardest have to be unlocked by
         beating the game at a lower difficulty level.
   •     In terms of game play, there are a number of different songs, but the basic
         motions of taping the screen in time with beats are maintained.
   •      The songs do encompass different time periods and styles. In the multiplayer
         mode, players can team up against another player(s), but only if there are 3 or
         more players.

   14. The players can express themselves in multiplayer games to others.

(SL=2)

   •     In multiplayer, not much you can do to communicate with the other player,
         though you can select from 9 different agents to represent your avatar.
   •     In single player, also not a lot of options to express yourself/identify with your
         avatar.

   15. The game uses orthogonal unit differentiation.

(SL=0)

   •     As a rhythm game, there are really only three major objects that a player interacts
         with – circles that are tapped, path’s with a ball that the player drags with their
         stylus, and spinners that a player spins by making a circular motion on the touch
         screen.
   •     The player taps as a larger circle closes in to surround each circle on the screen
         in time with the beats. Beat circles are numbered in the order they must be
         tapped, and their distance from each other corresponds to the shortness of the
         beat.




 CSCW to Go                                                                   Page 38 of 96
SI 689 Final Report                                                          Nintendo DS Evaluation


#     $

Clubhouse Games allows players to play a number of traditional card games like Hearts
and Poker, and other games like darts and battleship.




Figure 10 - Clubhouse Games (images from IGN.com)

                                         Heuristics                                             Severity
Screen layout is efficient and visually pleasing                                                   0
Indicators are visible                                                                             0
Navigation is consistent, logical, and minimalist                                                  1
Control keys are consistent and follow standard conventions & flexible                             1
The game gives feedback on the player’s actions                                                    1
The player cannot make irreversible errors (Can players still be competitive after a mistake)      0
The game contains help                                                                             0
The game and play sessions can be started quickly especially in multiplayer mode                   0
The game provides clear goals or supports player-created goals                                     0
The players are rewarded and rewards are meaningful                                                1
The first-time experience is encouraging                                                           0
There are no repetitive or boring tasks                                                            0
The game supports different playing styles (focus competitive vs. cooperative)                     0
The players can express themselves in multiplayer games to others                                  0
The game uses orthogonal unit differentiation                                                      0



 CSCW to Go                                                                              Page 39 of 96
SI 689 Final Report                                                  Nintendo DS Evaluation


                      Table 8: Heuristic Severity Rating for Clubhouse Games




   1. Screen layout is efficient and visually pleasing

(SL=0)

   •     The games under each category are arranged in rows scattering in the center area
         of the screen
   •     The players in each game session are arranged appropriately on the displaying
         screen and take up the full screen to let the players best view the action feedback
                         s
         and each other' status
   •     The color and visualization of items(e.g. bowling lanes, billiards board.
         etc) looks closed to actual items
   •     The "rules" page displays textual instruction with scrolling down and breaking
         down into several sub-pages

   2. Indicators are visible

(SL=1)

   •     A highlighting circle/square will demonstrate who is currently in his/her turn
   •     A recommended button(e.g. "play again", "next", "continue", etc) will be
         flashing to indicate the player what to click on
   •                                                               s
         In the Word Balloon/Bowling/Memory game, each player' progress is viewable
         during the whole game session so that the player can estimate where he/she
         would be in the ranking.

   3. Navigation is consistent, logical, and minimalist

(SL=1)

   •     Three buttons--Options, Chat and Rules--always locate on the bottom of the
         screen for each game session and occupy less than 10% of the screen.
   •     When selecting mini games, the game categories locate on the top of the screen
         and take no more than 20% of the screen.

   4. Control keys are consistent and follow standard conventions & flexible

(SL=1)

   •     The Game requires touching the screen to take certain actions so traditional
         control keys are not actually used.

   5. The game gives feedback on the player’s actions


 CSCW to Go                                                                    Page 40 of 96
SI 689 Final Report                                                Nintendo DS Evaluation


(SL=0)

   •                                                           s
         Bowling/Word Balloon/Balance game has each player' current status to keep the
                                      s
         player informed other people' progress.
   •     An immediate pop-out message, such as "success!" in the Balance game,
         "Match!" in the Memory game, shows up to notify the player whether his/her
         action is successful or not
   •                                    s/other players'
         The response time on the player'               actions is short when playing
         with the computer(the DS device itself)

   6. The player cannot make irreversible errors (Can players still be competitive
      after a mistake)

(SL=0)

   •     Making errors is part of the game play. No major errors relating to the bad
         usability of the game user interface is found.

   7. The game contains help.

(SL=0)

   •     The “Rules” button, providing the help information for the specific game is
         currently played, locates in the lower left corner, which is quite easy for players
         to find out.

   8. The game and play sessions can be started quickly especially in multiplayer
      mode.

(SL=0)

   •     Very traditional menu structure. Easy to figure out where you were going and
         what you were doing.

   9. The game provides clear goals or supports player-created goals.

(SL=0)

   •     The goals relate to the game you are playing. Because most of the games have
         clear, pre-defined rules, there is an assumption that a user knows how to play the
         game before playing it.

   10. The players are rewarded and rewards are meaningful.

(SL=1)



 CSCW to Go                                                                   Page 41 of 96
SI 689 Final Report                                               Nintendo DS Evaluation


   •     The rewards here are in points/chips. A numerical score isn’t very rewarding,
         though if playing against a non-computer opponent, it may mean more (think:
         bragging rights)

   11. The first-time experience is encouraging

(SL=0)

   •     If a user knows how to play the games contained in clubhouse games, the first-
         time experience can be rewarding, if the user can play well.
   •     Because clubhouse games keeps track of a user’s record, a user can build upon
         previous success the next time the game is started, instead of starting over.

   12. There are no repetitive or boring tasks.

(SL=0)

   •     Like EBA, there are repetitious tasks involved with some of the games (such as
         poker, blackjack, darts, and so on). However, this is seated in the rules of the
         original games themselves.

   13. The game supports different playing styles (focus competitive vs.
       cooperative)

(SL=0)

   •     Club House games offers over 30 different card, board, and puzzle games, so
         most players will be able to find something they can enjoy.
   •     The difficulty level for the computer-controlled players can be changed between
         3 settings to allow for players of different experience levels. Some games, such
         as hearts, allow for 2 players/computer-controlled players to play against 2 other
         players/AI bots, introducing a level of cooperation.

   14. The players can express themselves in multiplayer games to others

(SL=0)

   •     In multiplayer, have a basic ability to send predefined chat messages that seem to
         work well for basic games.
   •     In single player, you can choose your name and a picture to represent you from a
         list. This avatar is also present in multiplayer and identifies you.

   15. The game uses orthogonal unit differentiation

(SL=0)


 CSCW to Go                                                                 Page 42 of 96
SI 689 Final Report                                            Nintendo DS Evaluation


A variety of different games are represented, and they have differing levels of
affordance. Cards are generally well represented by suits and colors, and players’ hands
are organized in corners. As there aren’t really “enemies” or “power-ups” in the game,
this heuristic isn’t as applicable here.




 CSCW to Go                                                              Page 43 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


User Testing

Overview

To accurately determine how different factors (such as distant vs. co-located play, or
competitive vs. cooperative play) affected the overall experience with the Nintendo DS,
we decided to carry out a usability test, which would test the factors listed above, in
addition to other variables, such as experience.

We used the results of the heuristic evaluation to select Mario Kart DS as the game
which we would allow test subjects to play, while members of our team observed and
documented the interaction. After thorough examination of the game, we decided to use
the “Shine Racers” part of the game as the task. More detail about Shine Racers
(including objectives and methods) follows.



Test design:
Goal of the test:
We decided to focus on the effects of game type, distance, and the familiarity of DS
gamers to their experience of fun and performance. In addition to the main effect of
these variables, we are also interested in examining the interaction effects between these
variables.

Except for measuring the effects of the variables, we were also interested in identifying
all the possible factors that might influence the user experience and performance of DS
players. Therefore, we performed text analysis to the responses of our questionnaires.
The variables:

For the purpose of the test, we will have 3 independent variables and 2 dependent
variables. They are:

Dependent variables:
The dependent variables we wanted to measure were:

Fun: We used Likert scale rating (1~10) as the measurement of fun. Participants of the
testing sessions were asked to rate their level of fun after each task.

Performance: We used the outcome record of each round of the tasks as the
measurement of performance. There were 2 round of each task so we have 3 levels of
performance: 0 (lost both round), 1 (won 1 round), 2 (won both rounds).
Independent variables:
The independent variables we wanted to measure were:



 CSCW to Go                                                                Page 44 of 96
SI 689 Final Report                                              Nintendo DS Evaluation



Game type: competitive or cooperative game. (2 levels)
In our test, we controlled the game (Mario Kart DS) for the test. Mario Kart DS supports
both competitive and cooperative modes in the battle settings. In competitive mode,
players competed with each other in the same stage. In cooperative mode, they were
teamed up and play against the computer characters. We believed that game type is an
important factor to indicate DS players’ experience of fun and performance and made it
one of our independent variables.

Distant: co-located or co-distant (2levels)
We wanted to test the effect of the distance. Co-located means our participants played
the game in the same room. Co-distant means they played the game without seeing each
other and didn’t know who their partner or opponents were, either.

Familiarity: beginner or advanced (2 levels)
We believed that familiarity of the DS players is an influential factor to indicate their
experience of fun and performance. It is intuitive that advanced players would perform
better than beginners. As for fun, we wanted to test whether the beginners enjoy more
(or less) than advanced players.



Data collection:
As mentioned above, we managed to conduct 6 testing sessions. That gave us data of
inputs from 12 participants. From each participant, we gathered their responses for each
of the 5 tasks.
Questionnaires: we designed questionnaire of be used before, between and after the test.

1. Pre-test questionnaire: in pre-test questionnaire, we tried to understand our
   participants’ familiarity to Nintendo DS and also to multi-player games. We
   obtained our measures of the familiarity variable from this questionnaires
2. Post-task questionnaire: for each of the tasks, we asked the participants to
   rate their experience in terms of fun and also their own performance. We get
   our measures of fun here. Besides, the participants were asked to briefly
   describe their experience in the previous task. We analyzed their responses
   and identified all the possible factors that might affect the participants’
   experience of fun.
3. Post-test questionnaire: Participants were asked to compare the difference
   they perceived among all the previous tasks. We analyze the response here
   in order to get a better interpretation of the results from statistical analysis.

Video: we recorded all the testing sessions in order to capture the verbal and non-verbal
cues our participants expressed during the sessions.

Notes: in addition to the video, the moderators take notes during each of the tasks.

 CSCW to Go                                                                Page 45 of 96
SI 689 Final Report                                                  Nintendo DS Evaluation




Subject Recruitment
Subjects for our testing were recruited initially through an advertisement placed on the
popular social networking site facebook.com, as well as an email to the SI mailing list
si.all.open@umich.edu. At first, a $5.00 gift certificate to amazon.com was used as
compensation for participation in our study. After this yielded a less-than-desirable
number of participants, the amount of the gift certificate was increased to $10.00, at
which point another round of advertisements and email went out. The response from
this change was very favorable, and we were able to recruit the number of participants
we desired.The population, sample frame are illustrated below:




           Figure 11: population (yellow area) and sample frame (pink + blue) of test

Since we are recruiting students on Facebook, our sampling technique would be
convenience sampling. We tried to get all the respondent of our advertisements to do the
test. However, due to the confliction of the schedules among respondents and
moderators, we had to reject some of the respondents. 6 testing sessions were conducted.



User Test Setup

Subjects were able to complete the usability test at locations on either Central Campus
or North Campus. Because of the interaction effects we wished to study, two user tests
were carried out simultaneously, allowing each user to play with or against another user
in a nearby room (before moving to the same room). All user testing took place in study
rooms that were close enough for the DS short-range network to communicate
adequately between two units. On Central Campus, the location of the test was often in
student study rooms in Shapiro Library, and on North Campus, they took place in
student study rooms in the Duderstadt Center. All testing and recording equipment was
contained in a large travel roller-luggage, which was dubbed our “mobile usability lab”
(See figure 11) and brought to the users, at whichever location they were testing at.




 CSCW to Go                                                                      Page 46 of 96
SI 689 Final Report                                                  Nintendo DS Evaluation




                          Figure 12: Our "Mobile Usability Lab"

Before subjects arrived for user testing, we placed two digital video cameras on tripods
(one in each room). These cameras were out of the way, yet in a position to record the
user’s upper body and face as they interacted with a DS (See figure 12). These were
powered by AC adapters plugged into electrical outlets in the study rooms. Video data
was recorded to MiniDV tape, and later transferred to DVD media for analysis. Initially,
we had also intended to record the screens of the DS, through use of a document camera
or similar unobtrusive recording device. However, after such equipment presented a
problem to obtain, it was decided that the data from the screens of the DS was not as
important as the reactions and body language of the user, which was the dependent
measure of fun we used for this study. In light of this, the attempts to obtain document
cameras were abandoned.




                      Figure 13: Positioning of cameras for user testing



 CSCW to Go                                                                   Page 47 of 96
SI 689 Final Report                                                    Nintendo DS Evaluation




User Test Procedure

As the subjects were led into the study room by a facilitator, they were seated at a
position where they would be in view of the camera. The facilitator sat next to them
(See figure 13), and from a script, explained the purpose of the study, and what the
subject should expect. The subject was then given an informed consent document to
sign, which explained that their image and voice would be recorded, that this data would
not be used outside the study and all related presentations and reports, and that they had
the ability to stop the test at any time, should they feel uncomfortable.




                      Figure 14: Positioning of test subject and facilitator

After the subject signed the informed consent waiver, they were given a pre-test
questionnaire, which provided baseline data about the subject’s experience and attitudes
toward portable gaming systems. After this was completed, the user watched a two-
minute video tutorial on the Shine Racer game, part of the Mario Kart DS application.
The tutorial explained the purpose, objectives, and controls for the game, which are
described in the following paragraph.

Mario Kart DS’s “Shine Racer” game is a multiplayer game in which players control
virtual go-karts, and attempt to gather “Shines”, which are medallion-like objects
scattered throughout a finite virtual space, the number of which is fixed. As players
gather shines, they are added to a tally. After all shines on the map have been collected,
players may attempt to gather shines from other players by collecting and using various
objects on the map to hit another player or make another player lose control of their kart,
which in turn releases one of their shines back onto the map, available for other players
to take. Every 30 seconds, half of the players are eliminated (the half with the lowest
number of shines). The last player on the map (or the last team left, if playing as part of
a team) is declared the winner.



 CSCW to Go                                                                     Page 48 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


The user then moved on to perform the test tasks, each of which was performed twice.
The first task involved playing Shine Racers against seven computer-controlled
opponents. After this, the user was subject to one of two conditions involving another
human player (another test subject performing a usability test concurrently in another
room). One of the two conditions could be where both users played on the short-range
network on the same team with two other computer-controlled players, against a team of
four computer-controlled opponents. The other condition included each test subject
being teamed up with three computer-controlled players, and being pitted against the
other subject/computer team. After one of the tasks was finished, the users performed
the other. The order was randomized to minimize learning effects. Facilitators on
notebook computers used an Instant Messaging client to coordinate and synchronize the
testing from the two separate rooms.

After the distant condition, both test subjects moved to one room, where they again
played Shine Racers in one of two conditions (randomized order). Users could play
individually against each other (one-on-one, with no computer controlled opponents), or
on the same team, with two computer-controlled players, against a team of four
computer-controlled opponents. After the users completed two instances of one task,
they moved on to complete two instances of the other.

After each task, each user was given a post-task questionnaire, to subjectively assess the
level of fun experienced by the subject during a particular treatment. In addition,
facilitators took notes on paper, which were later analyzed and aggregated. After all
treatments were completed, the user completed a post-test questionnaire before they
were issued their gift certificates and allowed to leave.




 CSCW to Go                                                                Page 49 of 96
SI 689 Final Report                                                   Nintendo DS Evaluation


Results and Analysis

Overview
To analyze the data, we first performed statistical analysis to see if there is any
statistically significant difference contributed by the factors. After that, we reviewed the
data gathered from the videos of testing sessions and the questionnaires to understand
more about the cause to the effects. Finally, we gave our interpretations of the
quantitative data based on the qualitative data analysis.
In addition to the interpretations we drew from the quantitative data, we tried to identify
all the possible themes of participants’ judgments of fun when they were performing the
tasks from the questionnaires. We reviewed the qualitative data gathered from the
responses and build the conceptual themes of participants when they rated the
experience of fun.


Quantitative Data Analysis
In order to find out the whether and how our factors (independent variables) influenced
the fun and performance of the participants, we performed repeated measure ANOVA
in SPSS. As mentioned before, since each of the 12 participants performed 4 tasks we
designed, we have 12 clusters of repeated measures of responses. Fixed effects include a
between subject effect (familiarity) and two within subject effects (game type and
distance) Participants are included as a random effect, due to the dependency of repeated
observations in this design. .
We first analyzed the main effects caused by the factors separately. Our null
hypothesizes are:
   1. The difference of game type has no effect to our participants’ experience of in the game
   2. The difference of distance has no effect to our participants’ experience of fun in the
      game
   3. The difference in participants’ familiarity has no effect to our participants’ experience of
      fun in the game
   4. The difference of game type has no effect to our participants’ performance in the game
   5. The difference of distance has no effect to our participants’ performance in the game
   6. The difference in participants’ familiarity has no effect to our participants’ performance
      in the game



We found that there was no statistically significance related to game type, distance, and
familiarity on the experience of fun. Therefore, we couldn’t reject the null hypotheses.
However,the interaction between game type and distance showed a trend toward
statistically significance.. With more participants and test sessions, we expect to see
statistical significance in interaction effects.
.




 CSCW to Go                                                                      Page 50 of 96
SI 689 Final Report                                                       Nintendo DS Evaluation


                                                                    a
                                      Type III Tests of Fixed Effects

                                                         Denominator
            Source                      Numerator df         df            F       Sig.
            Intercept                               1           10.672   125.298   .000
            game_type                               1           27.816      .408   .528
            distance                                1           27.102      .496   .487
            familiarity_to_DS                       1           10.672      .047   .833
            game_type * distance                    1           27.051     1.691   .204
            game_type *
                                                    1           27.642      .036   .851
            familiarity_to_DS
            distance * familiarity_
                                                    1           27.102      .461   .503
            to_DS
               a. Dependent Variable: fun.

                        Table 9: The Effects of Independent Variables to Fun

Interaction effects: We didn’t find any statistically significance in interaction effects
among our independent variables. However, as mentioned above, we believed that with
additional sessions (larger sample size), we expect for the interaction effect between
game type and distance to become statistically significant.




                       Figure 15: Interaction between Game Type and Distance

After reviewing the responses of the questionnaire, we interpreted participants’
responses as:
1. Participants had less fun when competing with human in the same space. The major reason
   of this is because in our pairing of the participants, they usually don’t know each other
   before. Therefore, they don’t want to share their feelings with the other participant during
   and after the game. For example, one of our participant said “I can’t express my feeling



 CSCW to Go                                                                         Page 51 of 96
SI 689 Final Report                                                       Nintendo DS Evaluation


   when I am winning or losing because I don’t know her in person” as the reason they felt
   frustrated when competing with the other participant in the same room.
2. Participants had more fun when cooperating with human in the same space. We found that
   the participants were much more willing to share their feeling to each other since they don’t
   need to worry about hurting others’ feelings. Besides, our participants communicate with
   each other much more when cooperating in the same room. A lot of conversation in
   coordinating occurred in the session. One of our participant mentioned “Being in the same
   team is much more fun and it’s good to communicate with the teammate!” as the reason they
   enjoyed when collaborating in the same room with the other participant.




We found there is statistically significant difference in the effects of game type to their
performance. We also noticed the effect of distance was also obvious (not statistically
significant).

                                                                    a
                                      Type III Tests of Fixed Effects

                                                         Denominator
            Source                      Numerator df         df            F       Sig.
            Intercept                               1               36   115.190   .000
            game_type                               1               36    20.704   .000
            distance                                1               36     2.749   .106
            familiarity_to_DS                       1               36      .007   .936
            game_type * distance                    1               36      .444   .510
            game_type *
                                                    1               36      .226   .637
            familiarity_to_DS
            distance * familiarity_
                                                    1               36     1.093   .303
            to_DS
               a. Dependent Variable: performance.

                 Table 10: The Effects of Independent Variables to Performance



Effect of game type:
We found that the participants have better performance in cooperating tasks (p<0.05).
Our interpretation is that people are usually smarter and have better skill level than
computer characters. Moreover, while cooperating in the same room, participants were
able to coordinate strategies and actions. In fact in most of our test sessions, participants
won in the cooperative tasks.

Effect of distance:
We found that the participants have better performance in co-located tasks (marginal
significant, p = 0.106). Since in competitive tasks between participants won’t contribute
to the difference (one must lose when the other wins), we can say the difference were


 CSCW to Go                                                                         Page 52 of 96
SI 689 Final Report                                                 Nintendo DS Evaluation


contributed by the cooperating tasks. Again, our interpretation is that the participants
had better coordination in the game room than separated.


Qualitative Data Analysis

To better understand the data we gathered from the tests, we conducted the qualitative
analysis of the responses gathered from the questionnaires. We tried to identify the
categories and concepts emerged from the responses of the questionnaires. This gave us
a better understanding to the mental models of our participants.

We reviewed the responses from the questionnaires and used inductive method to find
the themes in the text. For each response in different session, we sorted them into the
themes we build from previous responses. If there is no suitable theme, we added a new
theme. Within each theme, we organized the responses into categories.

We focused on participants’ responses when they were asked about the experience of
fun after each task. At the end 6 themes and their sun-categories were identified:




                                                                              Number of
         Themes                              Sub-categories
                                                                              responses
                                 Supporting of communication                       7
1. Communication                 Sharing feeling                                   6
                                 Exchanging strategy                               4
                                 Outcome of the game (win/ lose)                   12
2. Outcome                       Rating of self performance                        4
                                 Comparison of others’ performance                  1
                                 Difficulty to learn                               12
3. Design of the game            Logic of game                                     13
                                 Sense of competing/ cooperating                   14
                                 Difference between AI and human in
4. Sense of “human” than                                                            8
   computer                      the game
                                 Awareness of human in game                         6
5. System                        Display                                            1
6. Learning                      Sense of learning                                  1
                      Table 11: Themes of Participants’ Conceptual Models

Based the number of responses of each theme, we can see that our participant considered
mainly on theme 1~4. For sub-categories, we can see that participants thought about
several particular properties of the gaming session when rating the fun of sessions:




 CSCW to Go                                                                  Page 53 of 96
SI 689 Final Report                                                 Nintendo DS Evaluation


1. Outcome of the game: our participants considered the outcome as one of the criteria when
   they rated their fun experience. Of course, winning is more fun than losing.
2. The difficulty of the game: when the game was easy to learn, our participants were able to
   enjoy the game more. On the other hand, they felt frustrated while they were not able to
   control the characters as the ways they desired.
3. The logic and aesthetic of the game: Many of our participants enjoyed the way that Mario
   Kart DS let them enjoy the racing and attacking with others in the same game. Also, they felt
   frustrated when unexpected events occurred during the tasks. For example, one of the
   participants mentioned that he didn’t like that the game don’t reward players points when
   they successfully attacked their opponents.
4. Sense of competition and cooperation: our participants liked more about interacting with
   other players than playing alone.




 CSCW to Go                                                                    Page 54 of 96
SI 689 Final Report                                                     Nintendo DS Evaluation



Conclusion
Our team learned a great deal about the Nintendo DS during the semester, particularly in
relation to multiplayer games. We also leaned quite a bit about structuring a study like
we did, particularly related to our usability tests. Based on our findings, we have several
recommendations to make about the Nintendo DS.


Recommendations
First, our interaction effects showed that skill-level while co-located was the only
variable we tested that had an effect on performance, and also had an effect on fun. As
previously mentioned, this could have been a result of our small survey size, but our
qualitative results do illustrate the importance of matching users at similar skill levels.
Nintendo and its game developers should take these findings into account when they
design multiplayer games for the DS. Many games already support a limited degree of
matching users over the Wi-Fi by skill level, but in our testing we found this to have a
limited degree of success. By developing a better system of tracking users’ scores and
matching them appropriately, users will likely have a more enjoyable time playing
multiplayer games.

Our second recommendation stems from our finding of the interaction effect between
distance and cooperative gaming. Because cooperative games created a much stronger
level of fun when users were playing in a co-located manner, one would expect the DS
to have a number of games that support cooperative play. We even had support of
cooperative gaming from our interviews. Unfortunately, there are very few multi-player
games that support cooperative play. In fact, one major reason that we chose the games
we did for our heuristic evaluation is that they all supported cooperative play to some
degree, though even in some of those games the level of cooperation was limited.
Nintendo would do well to produce more cooperative, multi-player games, perhaps even
tapping into the current popularity of massively multiplayer online games like World of
Warcraft and Second Life. Indeed, Maple Story, a popular MMO on PC, will be coming
to the DS sometime next year15, though there are still questions about just how much it
will support online, cooperative play16.

Finally, playing the DS in a co-located manner is a very social experience, as
demonstrated strongly by our qualitative testing data and our interviews. Because of the
co-located aspect of most short-range network play, users can easily communicate with
each other. Unfortunately, this ease of communication doesn’t extend to playing at a
distance, and thus the social element is lost. By using the CSCW principles and
technologies we have discussed during the semester, such as synchronous audio and text

15
   “MapleStory DS”. Wikipedia. Accessed December 9, 2007. <
http://en.wikipedia.org/wiki/Maple_Story_DS>
16
   “Maple Story DS lacks the only feature anyone cares about”. Joystiq. Accessed December 9, 2007. <
http://www.dsfanboy.com/2007/11/09/maple-story-ds-lacks-the-only-feature-anyone-cares-about/>


 CSCW to Go                                                                         Page 55 of 96
SI 689 Final Report                                               Nintendo DS Evaluation


communication, Nintendo could better create trust between users who are playing at a
distance.


What We Learned
Looking back at our process, we discovered several interesting (and sometimes painful)
things along the way. The largest issue that we faced dealt with our user testing. No-
show participants are always an issue for any usability test, but the fact that we used
pairs magnified the problem significantly. We had originally scheduled twice as many
usability tests as we actually carried out, and most of those that had to be cancelled were
cancelled due to a single no-show.

In the future, we would have a better policy for dealing with this problem, such as
scheduling a reserve participant in the event that one participant did not show. It may
have worked even better to schedule an extra participant and pay the extra if all three
showed, though that could have gotten expensive.

In general, we were reminded during the test that designing and coordinating a well-
structured usability test is a lot of work, as is analyzing the results. Luckily, we began
carrying out our usability test in early November, and gave ourselves three weeks to
finish, though we ended up having to add extra sessions at the end due to our problems
with no-shows.

One last, interesting result came from our testing questionnaires where we asked how
users had found out about the survey. We had emailed si.all.open, and posted to the UM
network of Facebook marketplace to advertise our test. One participant, however, told
us that they found out about our study on Slickdeals.net, a national website for finding
bargains. Apparently, our study was interesting enough that someone copied it from
either Facebook or our SI email to Slickdeals. This not only illustrated to us the power
of the Internet for advertising, but also reminded us of the variety of outlets we can use
to recruit users.


Final Words
Our team enjoyed working with the Nintendo DS over the past semester. Not only did
we gain an appreciation for the portable gaming system, but we further developed our
evaluation skills and our abilities to create an interesting and compelling study.
Nintendo has obviously succeeded with the DS, as its sales numbers would attest. We
also feel that we have succeeded in discovering useful information about an under-
studied but popular CSCP system.




 CSCW to Go                                                                 Page 56 of 96
SI 689 Final Report                                          Nintendo DS Evaluation


Appendix A: Interview Protocol
Opening Questions:

   •   Do you have a DS?

   •   How many people do you know who own a DS?

   •   Where are you most likely to use your DS?

   •   How often do you use your DS each week?

   •   Do you use any other mobile gaming devices (phone, N-gage, PSP)? If so, how
       do they compare to the DS?

Test related Questions

   1b. Have you ever played a single-player game (by yourself)?
   1c. If so, how often do you play single-player games?
   1d. What games have you played by yourself?
   1e. What single game is your favorite to play single-player? Why?
                                                                        t
   1f. What aspects of single-player play do you enjoy? What aspects don'you enjoy?
   1g. Could you describe the last time you played a game by yourself?
           i. What was fun about the experience? Why?
           ii. What was frustrating about the experience? Why?

   2a. Have you ever played a multi-player game over the short-range wireless network
       (with a nearby person)?
   2b. If so, how often do you play multi-player games over the short-range wireless
       network?
   2c. What games have you played with another person over the short-range wireless
       network?
   2d. What single game is your favorite to play with another person over the short-
       range wireless network? Why?
   2e. What aspects of multi-player play the short-range wireless network do you
                                 t
       enjoy? What aspects don'you enjoy?
   2f. Could you describe the last time you played a game with another person over the
       short-range wireless network?
           i. What was fun about the experience? Why?
           ii. What was frustrating about the experience? Why?

   3a. Have you ever played a multi-player game over the Internet (with someone
       located in a different place)?
   3b. If so, how often do you play multi-player games over the Internet?
   3c. What games have you played with another person over the Internet?



 CSCW to Go                                                            Page 57 of 96
SI 689 Final Report                                            Nintendo DS Evaluation


   3d. What single game is your favorite to play with another person over the Internet?
       Why?
   3e. What aspects of multi-player play the Internet do you enjoy? What aspects don'  t
       you enjoy?
   3f. Could you describe the last time you played a game with another person over the
       Internet?
           i. What was fun about the experience? Why?
           ii. What was frustrating about the experience? Why?

   4. What do you find to be different when playing single-player rather than multi-
      player?

   5. What do you find to be different when playing over the short-range wireless
      network than playing over the Internet?

   6. Do you use your DS for anything outside of gaming?
      • If they don'mention the above ask: Have you used Pictochat/Opera web
                    t
         browser

   7. What games did you find easy to learn to play?

   8. What games did you find difficult to learn to play?




 CSCW to Go                                                             Page 58 of 96
SI 689 Final Report                                          Nintendo DS Evaluation



Appendix B: CSCW To Go Interview Result
Opening Questions:

   •   Do you have a DS?

   U01: Yes

   U02: Yes

   U03: Yes, from June this year.

   U04: Yes, 6 months ago

   U05: Yes, I have 2
   Why do you have 2? – Touch screen scratches easily, especially in EBA. First DS
   touch screen malfunctions now.
   How long did your first DS last? – About 2 years. Bought new one in Jan

   U06: Yes

   U07: Yes, over 30 people have it. Had two lost 1

   •   How many people do you know who own a DS?

   U01: 10-15

   U02: 5 (mostly SI)

   U03: 5, 2 of PHDs and kids of friends’

   U04: 6-7 people. Most are SI. 2 are not

   U05: About 10-12, all friends. – Use WiFi to play Mario Kart with 8 people

   U06: 6-10

   U07: Around 30-40. Four people of whom are in SI

   •   Where are you most likely to use your DS?

   U01: At his house, in the car.

   U02: Commuting on the bus (depending on the controls, touch screen bad on the
   bus), school work breaks, World of Warcraft breaks. Not always to kill time,
   sometimes dedicate an hour or two to achieve a specific goal.



 CSCW to Go                                                            Page 59 of 96
SI 689 Final Report                                               Nintendo DS Evaluation


   U03: Mostly at home

   U04: If I play the games by myself, like on the bus; if I can play multiplayer game,
   the locations are varied, like my apt, my friends’ apt, DIAD. 40% multiplayer, the
   rest is single player.

   U05: On bus, after studying (break). Faster than starting a console like PS3.
   Relaxation

   U06: Waiting for something. Like when I as in the bank. It’s a long process. When I
   ran into people, like john. I will ask them to play. It’s not like I want to play DS.
   Like I have this 10 minute and I’d like to play XXX for 10 minutes

   U07:

   •   How often do you use your DS each week?

   U01: Every day, 15 minutes – 45 minutes

   U02: 3 times a week, approx 1 hour per session

   U03: 1 or 2 sessions, 10 minutes each, per night (brain age)

   U04: I didn’t play much. I play 1 or less than an hour during the regular semester. In
   summer I would play 4-5 hours per week. The playing time are scattered—I never
   play more than half an hour for each section

   U05: 5 hours – sometimes 10-20 (lots on the bus – 30 minutes each way

   U06: Various. 0~ 2 hours. 5~6 MAX

   U07: Originally around two hours each day, now after PSP around half an hour a day

   •   Do you use any other mobile gaming devices (phone, N-gage, PSP)? If so,
       how do they compare to the DS?

   U01: No

   U02: Don’t own any others, have used PSP and Gameboy.
        How do PSP and DS compare?
       PSP sucks – more for solo play and action-oriented games. Most games don’t
       have a timeless quality – for diehard Playstation fans.

   U03: I don’t compare that much since I only have DS. (only some mini games in my
   mobile phone) But he mentioned a bit after when we follow up:
   PSP vs DS: 1) Genre of the games. 2) Multi-purpose gaming device. DS games are
                               s
   easy, low commitment. That' the main reason I bought it. PSP: Higher commitment.



 CSCW to Go                                                                Page 60 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


   Higher resolution. But not extra features, more than games. “If it has more features,
   I’ll prefer that”.

   U04: I used to have a cell phone and play one game on that (tennis). I have also
      played PSP but didn’t play much. My friend tried to sell me one but I didn’t buy
      it. I never thought PSP is a handheld gaming device; on the other hand, DS has
      different kinds of games. I’m looking for different kinds of games and different
      kinds of ways to play the game—the touch screen, the control. I like the DS most
      because the game type

   U05: Have a PSP and a N-gage. Nintendo touch screen integrates human moves
        with the game – very good design. PSP has very good graphics, but is time
        consuming and games are expensive to make. Games are so-so. Need to may
        more attention to interface device.
        Sony games are targeted to Japanese gamers and Xbox targets North America.
        Nintendo does both – game choice and design are good.
        Which mobile gaming device is your favourite? – DS

   U06: Played iPOD nano game. Only once

   U07: Yeah most of them. Game Boy (GB), GBA, PSP, Engage. Like Game Boy
   Advance as favorite

Test related Questions

   1h. Have you ever played a single-player game (by yourself)?
   U01: Yes
   U02: Yes
   U03: Yes
   U04: Yes
   U05: Yes
   U06: Yes
   U07: Yes

   1i. If so, how often do you play single-player games?
   U01: Most of the time.
   U02: More often than group (mostly Zelda). 80% of the time is single player.
   U03: Most of the time
   U04: 60 % of my time for 1 hour or less if no class
   U05: 80% of the time
   U06: 90% of the time
   U07: 90% of the time

   1j. What games have you played by yourself?
   U01: New Super Mario Bros, Animal Crossing, Metroid, Advance Wars
   U02: Zelda, Castlevania (both), Mario Kart, Brain Age
   U03: Brian age, the most; Mario carts , Super Mario bro. Call of duty. Jam sessions.


 CSCW to Go                                                               Page 61 of 96
SI 689 Final Report                                            Nintendo DS Evaluation


   U04: (He couldn’t remember the game name so take out the DS.) Wario Touch.
   Super Mario, Mario Kart, Brain Age
   U05: Nintendogs, EBA (Japanese and USA), Phoenix Wright, Mario Kart, Zelda,
   Mario 3v3, Cooking Mama
   U06: Animal crossing, Phoenix ray, Zelda. Super Mario bro. Brian age.
   For brain age, I compete with Jamie on the machine. (wondered if it’s multi-player)
   U07: most of them

   1k. What single game is your favourite to play single-player? Why?
   U01: New Super Mario Bros, has so much – mini games, side-scroller. Also liked
   Mario Kart

   U02: Zelda, before that would be Castlevania, Dawn of Sorrow.
        Zelda, like the storyline, discover new items and weapons. Zelda interaction is
        unique with the stylus.
        Castlevania – like the character progression, wanted to get 100% completion
        and the best weapons. More achievement oriented than story oriented.

   U03: Mario cart, Brian age 1 & 2.

   U04: Mario kart is the favourite—interactive, more incentive to play. About the
      brain age—it’s good to test yourself how fast you can remember things—kinda
      skill training thing. But sometimes I’m shy to play Mario kart since people
      around would look at me as a kid, so that I don’t feel quite comfortable to play
      Mario kart in public. Brain age can be sort of skill training. I like Elite Beat
      Agents too.

   U05: Elite Beat Agents, music, unique gaming experience, PSP can’t offer it
   U06: Now, Zelda. The most is Sudoko, EBA. I like the action you can do with the
touch screen. There are many neat ways for input that you want to try. Also used the
mic in the games
   U07: Mario kart is the favourite to play individually because it is fun and
competitive


   1l. What aspects of single-player play do you enjoy? What aspects don' you    t
       enjoy?
   U01: Like something I can replay on single player, some games just have a story and
         then all that is left is multiplayer.
         Its just that once you have been in its over. Once you have done everything
        nothing else to do. More than one save file. If you want to go back to the
        beginning you can do it again. Played Mario brothers just yesterday.
   U02: Completing the game to its fullest and getting everything. Need to control a
         cool character. A lame character wouldn’t cut it.
         Hate having to retrace steps in adventure games. Some single-player games
         are too long – nothing seems new, so you want it to be over.


 CSCW to Go                                                              Page 62 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


    U03: I don’t have much time this semester. So I like games with low commitment.
    I like brain age because it helps me in memory. And it keeps tracks about the
    progress. I can save my records and continue playing when I get back.

   U04: One is the space. I can play wherever I want. I’m not constrained by the
   environment. The other one is the level of games. I have free decision to choose
   whatever level I like. I also like that it is free to time constraint.
   When playing the game, I feel like being controlled by the game. Some of the
   games, like Mario Kart, are too easy that you don’t want to play once you finish all
   the levels

   U05: FF3 is an enhanced port – classical games have nostalgia.
   Don’t like that you feel lonely, not as interesting without others. Can use Wi-Fi, but
   it is troublesome

   U06: I don’t like it to eat ups my time. I tend to play in my 10mins break.
        Good: I don’t need people to play with me.
        Bad: But I can’t play it in class though…

   U07: You can play and enjoy games by yourself. Also some RPGs best suited to
   play individually


   1m.        Could you describe the last time you played a game by yourself?
   U01:

   U02: Last week, played Zelda in the beginning stages, discovering travel and new
   characters, advancing the story. Interesting to start out in a single player game
   because everything seems new.

   U03: I like that it requires low commitments while others wants too much. I don’t
   have time for the RPG types of games. 10 minutes sessions (once or 2 times at
   night.) is good to me.

   U04: Super Mario bro. I played in a noisy place. I was playing at a high level of
   game but a friend came and interrupted me so that I couldn’t concentrate. The
   transition of the screen was also hard to catch. Sometimes I made a mistake

   U05: Two weeks ago – feeding dogs in Nintendogs. Was two weeks ago because he
   lent DS to friend

   U06: 20 minutes ago. I had meeting at 2:30PM. I played Zelda.

   U07:

           i. What was fun about the experience? Why?


 CSCW to Go                                                               Page 63 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


           U01: It’s the end, you’ve beaten it. Frustrating thing is there is no midway
                 point. You have to go to the beginning again.
           U02: Discovering new characters and mastering the mechanics of the game.
           U03: It’s partly skill training in my point of view.
           U04:
           U05: Catching girls eyes with dogs – “So cute – I like this!”
           U06: I finally beat the Z dungeon I had not been done before. For games to
           be good. It has to be progressively more difficult. It’s frustrating thing that
           you need to do the same thing again and again. In Z you can save you record
           and continue
           U07: Last game was Tetris vs. the computer. Enjoyed the game because it
           was challenging and fun

           ii. What was frustrating about the experience? Why?
           U01:
           U02: The unknown – not knowing where stuff is. In Zelda, there are mists
           of air that block your path, and you have to get past them before time runs
           out. If time runs out, have to retrace steps.
           U03: It’s hard to go fast since some games require you to use the two
           additional buttons on the top in addition to the direction and action buttons.
           I’m not quite good at playing those games.
           Do you know that you can play DS both through internet and short-range
           Wi-Fi network? Yes
           U04:
           U05: Very time-consuming
           U06:
           U07: Nothing was actually frustrating about that experience



   2g. Have you ever played a multi-player game over the short-range wireless
       network (with a nearby person)?
   U01: Yes. Brother also has it. Doesn’t have to buy other games ?? I play on the wifi
though.
   U02: Yes, with pretty much everyone he knows with a DS.
   U03: Yes, 2~3 times (overall) before. I played that with my friends’ kids. I like that
   you don’t have to have the game to play with others. You can download the game
   within the game session. We often play Mario karts and Super Mario bro.
   U04: Yes, 3 or 4 times. I went to a building with everything about Nintendo in NYC
   and found there are 18 people playing games together. It was really amazing!! But I
   didn’t play there since I wasn’t having mine with me. The games are not limited to 4
   people. I never play the game over the internet although I know that
   U05: Yes
   U06: Yes
   U07: Yes



 CSCW to Go                                                                Page 64 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


   2h. If so, how often do you play multi-player games over the short-range
       wireless network?
   U01:
   U02: Approx 3 times a month.
   U03: 2~3 times (overall) Plays animal crossing, super Mario bros and virchoie
       There was one time when I was in the airplane. I played Mario Kart with a lady.
       It was fun.
   U04: 60% (previous mentioned). I enjoy the multi-player games more than single-
   player ones. It’s more competitive to play with other people. You can’t expect other
   people’s reaction and strategy—the uncertainty attracts me a lot. (He described a
   previous game experience to demonstrate that.)
   U05: Now, not much. Before, about 3-4 hours per week
   U06: 5 or 7 times in total
   U07: Around 60% of the time

   2i. What games have you played with another person over the short-range
       wireless network?
   U01:
   U02: Elite Beat Agents and Mario Kart. Elite Beat Agents is more fun than Mario
   Kart, but have to resort to Mario Kart if people aren’t comfortable with EBA.
   EBA is a simple game, but takes a developed amount of skill to accomplish each
   song. Way to test if you skills are really good, by playing other people rather than
   the machine.
   U03: Super Mario bro, Mario kart
   U04: Elite beat agent, super Mario kart
   U05: Mario Kart
   U06: EBA, Animal crossing, Super Mario Bro
   U07: Lots of them. Mario kart, bleach, Tetris

   2j. What single game is your favourite to play with another person over the
        short-range wireless network? Why?
   U01: Animal crossing. You are not competing you need to buy things from other
   person so there are benefits of playing with other persons not just beating them
   Its really simple to set up. You know who you are playing. If you are playing Wi-Fi
   you don’t know who you are playing. But here you can relate to who you are playing
   with.

   U02: Enjoy seeing the reactions of other people – adds more enjoyment to the
        experience. Can make things quite funny. Also can see people being
        impressed.
        Usually him cursing and the other person laughing, still a fun experience.
   U03: Super Mario because not many games can be played online. (<- this maybe
                   s
        because he' a relatively new to DS)
   U04: Mario Kart. It is competitive. I curse when playing. I think it’s fun to say “I
        BEAT U”. I like competitive games when playing with others
   U05: Mario Kart – easy to play


 CSCW to Go                                                                Page 65 of 96
SI 689 Final Report                                            Nintendo DS Evaluation


   U06: Animal crossing. When you go the someone’s town , you can grab others’
        things
   U07: Tetris is the favourite for short range play because it is more competitive

   2k. What aspects of multi-player play the short-range wireless network do you
                                    t
       enjoy? What aspects don' you enjoy?
   U01:
   U02: Seems more of an interaction with other people.
         Don’t always know how the multiplayer mechanics are different from the
         single player (i.e., in EBA a person can fill starts to shrink things on your
         screen).Also long loading times to connect. Can be problematic to
         communicate in certain environments (i.e. – cursing on the bus is bad). Need
         headphones to play EBA. He’s not really a multi-player sort of guy – doesn’t
         like being ranked (more relevant to Internet play).
   U03: It was fun. Not much I can remember.
   U04:
   U05: Easy – don’t have to set up router and access – just have to turn on and
   connect. Very convenient. Need more short range games
   U06: When we played EBA, it give me some sort of camaraderie?! Dislike: It takes
   some time to connect
   U07: You can play with friends and interact with them. But some games are not
   suited for multilevel play and some are just not as much fun


   2l. Could you describe the last time you played a game with another person
       over the short-range wireless network?

   U01: We played Mario and single cart against my brother.
   U02: Last time he played was with Josh M in a stairwell playing EBA.
   U03:
   U04: I played Elite Beat Agent with John. I didn’t have the game so I downloaded it
   from John’s device. John chose a hard song and I couldn’t handle it. So he chose a
   mid-level song but I still couldn’t make it. And he chose the easy one so we can
   play. But I still lost it.
   U05: We six played Mario Kart together – one person got a star, then another did –
   they beat each other black and blue. One of them spilled a cup of water on the NDS
   and killed it
   U06: with Josh and john. We had sometime to kill. (game?) John and I paired
   together against you. I chose the song I didn’t want.
   U07:

   i. What was fun about the experience? Why?
        U01: You get to see their expression. Get to see their experience when they
        lose! Its fun to see them
        U02: Noticed he wasn’t as good as he thought he was playing difficult songs
        against Josh.


 CSCW to Go                                                              Page 66 of 96
SI 689 Final Report                                            Nintendo DS Evaluation


           U03: I didn’t actually know who I was playing with but the game-playing
           was fun.
           U04:
           U05:
           U06: I’d say it’s good that we can play right away, anywhere
           U07: Played Mario Kart with Ahmad. The best part was beating him! And
           watching his expression

   ii. What was frustrating about the experience? Why?
         U01: If they beat you. If you more around too much, if you point it away
         from the other person.
         U02:
         U03: The control of Super Mario Brother. I need to learn more about the
         control. But I was never a good gamer, even when I was in elementary
         school.
         U04:
         U05: No frustration
         U06: no
         U07: It would have been much more fun with 4-8 people but there were just
         two of them

   3g. Have you ever played a multi-player game over the Internet (with someone
       located in a different place)?

    U01: Yes
    U02: Yes
    U03: Yes, I searched for challengers (strangers) online.
    U04: No. I need to buy the browser (I guess)—I don’t want to buy. Although it’s
interesting, I don’t have time to play too much. I never want to play games over long
range with such a small screen
    U05: Yes
    U06: no
    U07: Yes


   3h. If so, how often do you play multi-player games over the Internet?
   U01: Yes its fun
   U02: Before, DS wasn’t compatible with router, so didn’t play then. Now, play
   infrequently.
   U03: Mario Kart & Super Mario bros
   U04:
   U05:
   U06:
   U07: Around 40% of the time I play multiplayer games




 CSCW to Go                                                              Page 67 of 96
SI 689 Final Report                                               Nintendo DS Evaluation


   3i. What games have you played with another person over the Internet?
   U01: Mario cart, wi fi animal crossing. You need friend codes for it which can be a
   little ….. I traded a bunch when I had Pokemon
   U02: Mario Kart
   U03: Mario Kart
   U04:
   U05: Mario Kart and FFIII to obtain Onion Knight (friend helped him with that)
   U06:
   U07: Mario Kart and Tetris mostly


   3j. What single game is your favourite to play with another person over the
       Internet? Why?
   U01: You can put something on the internet and then just go and leave it and come
   back. The range of players is different that’s good. Metroid Hunters.
   U02: Mario Kart, since that’s all he has. When purchasing games, online play isn’t a
   major interest.
                        s
   U03: Mario Kart. It' easy to play. Brain age is actually a very multi-player ready
       game. But it’s gonna be hard to match. Like the game in Buffalo Wild Wing,
       you’re playing not to the computer but to the people in the restaurant
   U04:
   U05:
   U06:
   U07: Tetris is the favourite because in Mario kart some players are too good!

   3k. What aspects of multi-player play the Internet do you enjoy? What aspects
          t
       don' you enjoy?

   U01: That you can change the access points. If you have the main one at your home.
   If you go to a café or hotel you can change the setting on the go and do it easily. Its
   simple to jump into the game. They just find the opponents for you. Metroid you
   don’t need friend codes. No I don’t have friend codes. I don’t use friend codes very
   much. I just haven’t needed to use them I have more fun playing who I don’t know.

   U02: Good to always have human players to play against. People use interesting
   techniques to gain an advantage. Downside – online play doesn’t have progression,
   and since you can’t see opponent, you don’t have the same experience as when
   playing short range.

   U03: Time-consuming and there is no feedback about the status of connection
       What I’d like to have is that the broadcast functionality that I can tell people that
      I am available. I don’t like to wait for the system. I like to be able to do other
      stuff or another game before the game is ready

   U04:



 CSCW to Go                                                                 Page 68 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


   U05:

   U06:

   U07: Enjoy playing with people all over the world from France to Italy


   3l. Could you describe the last time you played a game with another person
       over the Internet?
   U01:
   U02: Back in September
   U03:
   U04:
   U05: Set up router to connect with friends back home, but friends had trouble setting
   up wireless routers. Only two people succeeded in getting on. The network is a
   problem, particularly with latency. Don’t really play multi-player games over the
   Internet
   U06:
   U07:

   i. What was fun about the experience? Why?

U01: I played Metroid and I just jumped into the game. WE played a bunch of games
and it was evenly matched, It wasn’t just one guy who was very good. Its fun when all
are equal level.
U02:
U03:
U04:
U05:
U06:
U07: Played Tetris over the internet. It was fun because of the challenge and playing wih
someone from another part of the world

   ii. What was frustrating about the experience? Why?

U01: When people leave if they are getting easily beaten. If you are the only one who is
not good. Metroid is this common and not very nice experience.
U02: Losing. That’s about it. Also waiting to play.
U03: The loading time
U04:
U05:
U06:
U07: Waiting time for connecting to the game. Also doesn’t like only one piece of
metadata about competitive player – wins and losses. Will be much better to get a
compete profile of the person doing it. Also important to find someone close to your
skill level


 CSCW to Go                                                               Page 69 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


   9. What do you find to be different when playing single-player rather than
      multi-player?

U01: This is more interactive than single player. In multiplayer when they design it they
want you to do something. You can either go left or right in single player but in
multiplayer you can do other stuff like hide and pick other people up as they go. In the
single players bots don’t have that intelligence that others have.

U02: Human players are more intelligent than AI, so adds more excitement. Winning
means more against another person. Feel that luck plays more of a role against the
computer, skill against humans.

U03:
U04:
U05: DS doesn’t have a lot of multiplayer games
U06: The single player games require more, you need more time. Multi-player, you need
only small chunk of time and the session finishes
U07: Very few really neat features particularly built for multiplayer games. Single
player games usually have more neat features. Hard to find people around who have DS.
Single player games are nice in that way because they can be played anytime

   10. What do you find to be different when playing over the short-range wireless
       network than playing over the Internet?

U01: You know who you are plaing in the short range. In the wifi you don’t know who
you are playing. If they beat you really really badly you hav a choice to never do it
again. But in short range you have to play them again. When its one on one in the close
range it can clearly be that one person is superior. But on wi fi you win some you lose
some depending on who you are paired up against.

U02: Can see reactions of others nearby. Can’t really chat/communicate over the
Internet. Only interacting through your character (make your character jump and down
to show happiness). Can’t communicate your excitement about playing together.
Do you think you learn things from playing with people online with people from other
cultures?
From networking, people you link to are an expression of who you are, so in short range
they tend to be closer to your level. People online may be way better.

U03:


U04:


U05: No opinion, but he hopes it can enhance the experience. Would like to connect to
computer to help enhance it.


 CSCW to Go                                                               Page 70 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


Do you think there is a different between playing with friends or strangers?
Don’t like playing with strangers – don’t really do it. One time in the subway, a stranger
was playing Mario Kart in a subway, so he asked her to play, and played one round with
her before getting off

U06:

U07: In short range you know the person and can interact with him. Over the internet
you mostly never know who you are playing against

   11. Do you use your DS for anything outside of gaming?
   U01:

   U02: Not really, though he wouldn’t mind organizer sort of functionality. No web
   browser or Pictochat.

   U03: I have tried to use it as a music player. I also have calendaring system installed.
      I don’t like the browser. You need extra memory for opera browser. I am looking
        for an alternative solution for it. But others are buggy and hard to see.
        I’ll certainly increase the usage if there is a ready-to-use and not requiring more
        components.
      I don’t think that it will be a replacement of other devices as it is now. The type of
   control and the way of display does not fit the requirements of other purposes. I’d
   like to use the browser for checking emails/news online without opening my laptop
   when necessary

   U04:

   U05:

   U06: Never used it for other purposes

   U07: Yes used Pictochat in college and it was pretty useful. Has tried opera but
   didn’t find any sense in using his browser on the DS.

       •              t
           If they don' mention the above ask: Have you used Pictochat/Opera web
           browser

       U01: I do the picot chat with my brother. Its fun.
       We haven’t had full 16 but I’ve done with other friends before. We have people
       in different parts of the house. They say 30 feet but you can really stretch that
       range. The range is pretty far. If you do it one on one with right next to each
       other is not so much fun. Like the fact that you can write and draw. You can
       make captions drag letters, you can put stuff where you want. You don’t have a
       lot of space but you can really use it to draw and write.
       I don’t use opera. It sounds really cool but I think its only for the Lite


 CSCW to Go                                                                 Page 71 of 96
SI 689 Final Report                                             Nintendo DS Evaluation



       U02:

       U03:

       U04: Some friends showed me the picotchat but I haven’t use it before since no
       one else (I know) use this for chatting. I prefer to concentrate more on the game-
       playing to chatting. Chatting doesn’t make any sense to me—I guess the DS’s
       primary function is game-playing.
       I use DS just for game-playing. I don’t know much about it’s other
       functionalities (asked jack about the memory of DS). I’m hoping it can become
       sth like a PDA, calendar, phone, etc. I would use it if DS has some other
       functions—if it’s easy to use, like easy to connect to wireless network. But I
       would still prefer to use my laptop—I wouldn’t use DS as my primary device

       U05: Know of pirated memory cards that you can store games to. Don’t know of
       Pictochat

       U06:

       U07:



   12. What games did you find easy to learn to play?

U01: New super Mario brothers, animal crossing – those two are the easiest to learn. In
animal crossing there is no plot so you can take as long as you want to figure it out.
NSM brothers there are very few controls that you are going to use.

U02: Stylus games are easier to pick up and learn to play. Most DS games are catered
with more simple controls.

U03: One of the main reasons I bought it is simple to use

U04: Mario kart is easy to learn. Super Mario is easier to have fun with. EBA is easy to
get use to control, too

U05: Every game with Mario is easy to learn, because Nintendo uses him for marketing

U06: All the games are easy to play. Like Z, you get new things, skills progressively.
You have tutorial to learn (and for all the game)

U07: Most DS games are easy (Mario brothers, Brain Age, Soccer etc.)




 CSCW to Go                                                               Page 72 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


   13. What games did you find difficult to learn to play?

U01: Metroid took me a lot of time. I realized that you cold do a lot more if you use the
touch screen rather than the buttons. Since I learnt the touch screen later the controls
came a little bit tougher.

U02: Elite Beat Agents takes a lot of practice. Most rhythm games are hard to play. Not
hard to learn to play, just take a lot of skill.
Any last words about the Nintendo DS?
Don’t know what Pictochat does. Will probably move to the city after this, would like
to automatically see people in the vicinity that are playing a certain game.
Would be nice to know if friends are connected on the Internet to play a game.
Still like it over a PSP – very happy with his investment.
When he showed DS games to girlfriend (who isn’t a gamer) she was much more
interested than into a PSP.

U03: Call of duty might be the hardest game I’ve ever played. You need to use the stylus
and the button , directions at the same time.
  Have you ever used/hear of Piochcat?
I’ve heard a lot of them but didn’t play that. One thing about it is: you need to leave the
game to use that in games. I didn’t use that primarily because of the reason. It’ll be
better if it’s integrated with the games

U04: Sometimes I forgot that only the top screen is not a touch screen and kept tapping
it.
Brain age takes a little bit longer to learn than others but within time it’s still fun.
The reason why I like the short-distance game-playing is it’s fun to see other people’s
interactions, gesture, yielding, etc. I curse when I play=) playing with someone I don’t
know can’t let me have more reactions than that.
Are there any improvements (new features/game design/ hardware) you want to see?
The voice input feature. If I play online, I would like to have conversation with each
other. In terms of games, I prefer some games require me to partner other people as a
group to finish some task—more collaborative portion I need to get involve.
I never play with someone I didn’t know, but I guess I don’t mind playing with
strangers.
The games are competitive—multiplayer games are more fun. For single player game is
just because I want to kill some time.

U05: Some role-playing games. Games not originally designed for the DS are hard..
Can you name a game that frustrates you?
Final Fantasy III was frustrating at first, but fine after an hour or so.
Would you use email or calendaring on the DS if available?
No – don’t want to integrate all devices. DS for gaming, cell phones for calls, etc. PSP
is more of a convergence device, but isn’t as successful. Extra functions increase costs,
so turns people off



 CSCW to Go                                                                Page 73 of 96
SI 689 Final Report                                                Nintendo DS Evaluation


U06: NO.
Other words?
If I have problem, I’ll go to those cheat gaming site to get things. I ‘m really glad that I
had it

U07: Castlevania, Final Fantasy, Mario and Luigi




 CSCW to Go                                                                   Page 74 of 96
SI 689 Final Report                                               Nintendo DS Evaluation


Appendix C: Usability Testing Script and
Questionnaires

Script: Usability Test of the Nintendo DS

Greeting the participant
Meet the participant at the appointed place

Good afternoon. My name is moderator name and I will be working with you
today. This session should take no more than 60 minutes. Before we sit down,
do you need to use the bathroom or get a drink of water?

When the participants are ready, walk them into the Usability Testing Room and sit them
down across from each other, in front of the DSes being used for the study. The
moderator should sit 90 degrees from each participant, so he/she is visible to both
participants.


Pre-test procedures
The following introduction will be read to all participants before starting the test
session.

Thank you for participating in this study.

I’m going to read this script to you now so that I provide you with the exact same
instructions that I will have provided to everybody else, and also to ensure that I
don’t forget anything of importance.

Today, we will be looking at the Nintendo DS video game system. We are
interested in observing how you use the system to play multiplayer games. You
will be given a series of activities to perform with the Nintendo DS system.

Please keep in mind that the purpose of our testing is to evaluate the Nintendo
DS, not you. During today’s session, you will be performing some tasks that may
or may not be familiar to you. Please perform these tasks to the best of your
ability. Again, we are testing the usability of the DS system, and not your
knowledge or personal experience using the DS.

A few things to remember:

   •   While you are working, your actions will be monitored. There is a camera
       here [point to them], and I will be taking notes as you play. The session
       will be videotaped so additional evaluation of the interface can be carried
       out after this session.



 CSCW to Go                                                                 Page 75 of 96
SI 689 Final Report                                          Nintendo DS Evaluation


   •   If you have problems during an activity, you are free to ask me questions,
       and I will answer to the best of my ability, though I may be restricted in
       my answer to maintain the validity of this experiment.

Again, I want to remind you that we are evaluating the DS system, not you. If
you feel uncomfortable, you are free to stop the evaluation session and leave at
any time.

Before we begin, I will need you to read and sign this consent form.

(Hand participant the consent form).

Pre-Test Questionnaire

During the next 45-60 minutes we are going to complete several activities. First,
I will ask you to fill out a brief questionnaire. Your responses will remain
anonymous, and will be used only for the purpose of data analysis.

(Hand participant the pre-test questionnaire).

Next, I will give you several gaming tasks to complete using the Nintendo DS
system. You will be doing several short sessions of Mario Kart DS, which we
will explain before you begin. We will also set up the scenario on the DS, so you
should be ready to participate when you are given the DS.

After each task I will record your score and give you some follow-up questions to
get your opinions of the task. Please hand me the DS so I can set it up for the
next task. At the end of the test I will give you a final questionnaire.

Do you have any questions before we begin?

Task 1: Single Player Warm-up
Open the DS tutorial video on the laptop.

Before we begin, I will show you at short tutorial that will explain what you need
to do to play the game. After the tutorial, you will be doing several rounds of
“Shine Racers”

Play the DS tutorial. While it plays, make sure the DS is set up.
   1. Start Mario Kart DS
   2. Select Single Player
   3. Select Battle
   4. Select Shine Runners
   5. Select Mario (or Luigi, depending on your DS)
   6. Change difficulty to Easy and select OK (settings should be– Easy, Choose, Free,
       and Off)


 CSCW to Go                                                            Page 76 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


   7. Select Palm Shore
   8. When you are ready to begin the test, press OK and pass the DS to the
      participant.

You will begin by playing against 7 computer controlled opponents. You are all
competing to earn the most shines.

Before we begin, do you have any questions?

Hand the player the DS, make sure to press OK to begin.

When the participant either wins or is eliminated, take the DS and Record whether they
won or lost.

Then start the second round:
   1. Select Next Course
   2. Select Palm Shore.
   3. When you are ready to begin the test, press OK and pass the DS to the
       participant.

At the conclusion of the second session, take the DS and Record the Score. Then hand
the player a copy of the post-task questionnaire for Task 1.

Thank you for completing task one. Please fill out this questionnaire.

  Score in Round 1          Score in Round 2        Total score after round 2
    Win/Lose                  Win/Lose



Task 2: Multi-player, Competitive, Distant

In this task, you will be playing with another person, who will be playing Luigi (or
Mario, if you are Luigi). Both of you will be competing to become the winner. There
will only be one elimination, because there are only two players.

Do you have any questions before we begin?




 CSCW to Go                                                              Page 77 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


Set up the game for multi-player over the short range network as follows:
    1. Go back the initial Mario Kart DS screen (reset if necessary).
    2. Select Multiplayer.
    3. Mario: select Create Group and press A. Then select Normal and press A. Then
        wait for Luigi.
    4. Send an IM to the other moderator to let him/her know that you are ready to
        begin task 2
    5. Luigi: wait for the Crescent group to appear, then select it and press A.
    6. Mario: select Cut Off and press A.
    7. Mario: Select Shine Runners and Press A.
    8. Mario: Select Mario and press A.
    9. Luigi: Select Luigi and press A.
    10. All future instructions are for Mario
    11. Default settings (Off, Choose, Free, Off) should be correct. Select OK and press
        A.
    12. Select Palm Shore and press A.
    13. IM Luigi and ask if they are ready.
    14. When they are ready, select OK and press A, and hand the DS to the participant.

When the user is finished, take the DS and record the Score. Then begin the second
round.
Mario: Select Next Course and Press A
       Select Palm Shore and press A
       IM Luigi and ask if they are ready
       When they are ready, select OK and press A

At the end of the second round, take the DS and record the Score.

Thank you for completing task two. Please fill out this questionnaire.

  Score in Round 1          Score in Round 2        Total score after round 2
    Win/Lose                  Win/Lose



Task 3: Multi-player, Cooperative, Co-Distant

In this task, you will be playing with another person, who will be playing Luigi (or
Mario, if you are Luigi). You will also be playing with 6 computer-controlled karts.
You will be on the blue team with Luigi and 2 computer controlled karts. You will be
playing against the red team. You can only target red players with items, and at the end
of the round, the team representing the winning kart will be declared the winner. If you
are eliminated before your other teammates, you will remain as a ghost kart to watch.



 CSCW to Go                                                               Page 78 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


Do you have any questions?
Mario: IM the other person to see if they are ready.
       Select Quit and press A
       Select Shine Runners and press A
       Select Mario and press A
Luigi: Select Luigi and press A
Mario: Select CPU Kart EASY and Team ON
Mario: Press the L or R button on the top of the DS to switch teams so that Mario and
       Luigi are on the same team (should be the blue team).
Mario: Select OK and press A
       Select Palm Shore and press A
       IM Luigi to see if they are ready
       Select OK and press A


When the user is finished, take the DS and record the Score. Then begin the second
round.
Mario: Select Next Course and Press A
       Select Palm Shore and press A
       IM Luigi and ask if they are ready
       When they are ready, select OK and press A


At the end of the second round, take the DS and record the Score.


Thank you for completing task three. Please fill out this questionnaire.


  Score in Round 1          Score in Round 2         Total score after round 2
    Win/Lose                  Win/Lose




 CSCW to Go                                                                Page 79 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


Intermission

Mario: We will now be joined by another player, who will be participating in the next
tasks with you.

Luigi: We will now move to another room, where we will join with another player for
the next tasks.

Second moderator leads test participant to adjoining room, bringing the DS, laptop, and
camera. The second participant should be seated across from the first. At this point,
the first (Mario) moderator will do all future talking.

You may greet each other and communicate any information that you wish during the
next tasks.


Task 4: Multi-player game, cooperative, Co-located

In this task, you will be playing with the person across from you. He/she will be playing
Luigi (point) and he/she will be playing Mario (point). You will also be playing with 6
computer-controlled karts. You will be on the blue team with Luigi and 2 computer-
controlled karts. You will be playing against the red team. You can only target red
players with items, and at the end of the round, the team representing the winning kart
will be declared the winner. If you are eliminated before your other teammates, you will
remain as a ghost kart to watch.

Do you have any questions?


Mario: Select Next Course and Press A
      Select Palm Shore and press A
      IM Luigi and ask if they are ready
      When they are ready, select OK and press A


When the user is finished, take the DS and record the Score. Then begin the second
round.
Mario: Select Next Course and Press A
       Select Palm Shore and press A
       IM Luigi and ask if they are ready
       When they are ready, select OK and press A


At the end of the second round, take the DS and record the Score.


Thank you for completing task four. Please fill out this questionnaire.


 CSCW to Go                                                               Page 80 of 96
SI 689 Final Report                                             Nintendo DS Evaluation




  Score in Round 1           Score in Round 2        Total score after round 2
    Win/Lose                  Win/Lose



Task 5: Multi-player game, cooperative, Co-located

In this task, you will be playing with the person across from you. You will be playing
Mario (points to player) and you will be playing Luigi (points to player). Both of you
will be competing to become the winner. There will only be one elimination, because
there are only two players.

Do you have any questions?

Mario: Select Quit and press A
       Select Shine Runners and press A
       Select Mario and press A
Luigi: Select Luigi and press A
Mario: Default settings should be good (Off, Choose, Free, and Off)
        Select OK and press A
       Select Palm Shore and press A
       Ask both participants if they are ready
       Select OK and press A

When the user is finished, take the DS and record the Score. Then begin the second
round.
Mario: Select Next Course and Press A
       Select Palm Shore and press A
       Ask participants if they are ready
       When they are ready, select OK and press A

At the end of the second round, take the DS and record the Score.


Thank you for completing task five. Please fill out this questionnaire.


  Score in Round 1           Score in Round 2        Total score after round 2
    Win/Lose                  Win/Lose



 CSCW to Go                                                               Page 81 of 96
SI 689 Final Report                                          Nintendo DS Evaluation



Debriefing and Final Questionnaire
(Hand participant the final questionnaire)

The second moderator should walk the second participant back to the original
room.

You have now completed all of the tasks. I have a final set of questions I would
like to ask you.

Read each question from the post-test questionnaire and record the participant’s
answers.

Thank you again for participating in this study. Here is your gift certificate.
Please contact us if you have any future questions.

Hand gift to participant and leave them in privacy to complete the questionnaire.




 CSCW to Go                                                            Page 82 of 96
                    University of Michigan School of Information
               User Input to Nintendo® DS™ Portable Gaming System


Thank you for agreeing to meet about our study of the Nintendo® DS™. We are taking
a course in the School of Information about how to focus on end users in the
development of new digital products designed to support group activity. As part of this
course, we are trying to understand the role of the DS as a tool that fosters collaboration.
We’d like to see how different people interact with it to better understand how it can be
used. Please know, however, that we are testing the DS, not you or your abilities.

The session will take about 45 minutes or less. In this session, we will teach you a little
about the DS and then ask you to do various tasks with it. We are interested in how you
use it so we can design products like it that you and others like you might enjoy. When
you complete each task, as well as when we’re all done, we’ll interview you about what
you experienced and your opinions about this product.

We would like to video record you while you use our product. This videotape merely
records you, your actions, and your voice when you “think out loud” or react to goings-
on with the device. The videotape will be identified only with a number, not your
name. We will report the results of our sessions in the aggregate, not identifying you in
particular in any way except general demographics, though still frames or clips from the
video may appear in our final report, which will be presented to our class, and not
published in any way.

Your participation in this study is strictly voluntary: You may stop at any time.

If you have any concerns with the study, please contact any of the people below.

Study team                                           Professor of the SI Course
Chris Demeniuk, demeniuk@umich.edu                   Professor Gary M. Olson
Cheng-Lun Li, chengli@umich.edu                      gmo@umich.edu, 734-763-5644
Josh Morse, jjmorse@umich.edu
Satyendra Nainwal, nainwal@umich.edu
Xiaomin Jiang, xmjiang@umich.edu

                       University of Michigan IRB Administrator
                       Judith Birk
                       jbirk@umich.edu, 734-936-0933


I consent to participating in this study, under the stipulations outlined above:


Name:____________________________________ Date:_________________

                       One copy to the participant, one to the study team member.
                                      CSCW To Go
                                      ®
                             Nintendo DS™ User Evaluation
                                 Pre-Test Quenstionnaire

                   - For each question, please place a check on ONE line -

1. Do you own a Nintendo DS?

______ Yes

______ No


1b. If you own a DS, how long have you owned a DS system for?

______ 0-6 months

______ 6 months - 12 months

______ 1-2 years

______ More than 2 years


2. How often do you play games on the Nintendo DS?

______ Never

______ Less than 2 times per month

______ Less than 2 times per week

______ More than 2 times per week

______ More than 2 times per day


3. How often do you play any video games on any platform (Playstation, XBox, Computer,
etc.)?

______ Never

______ Less than 2 times per month

______ Less than 2 times per week

______ More than 2 times per week
SI 689 Final Report                                               Nintendo DS Evaluation


______ More than 2 times per day

4. How often do you play any video games on any platform with other people (multiplayer)?

______ Never

______ Less than 2 times per month

______ Less than 2 times per week

______ More than 2 times per week

______ More than 2 times per day



4. How often do you play any video games on the DS with other people (multiplayer)?

______ Never

______ Less than 2 times per month

______ Less than 2 times per week

______ More than 2 times per week

______ More than 2 times per day




5. How many years have you played video games regularly (once per week or more)?

______ 0-1 years

______ 1-3 years

______ 3-5 years

______ 6-10 years

______ More than 10 years




 CSCW to Go                                                                 Page 85 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


6. How often have you played Mario Kart DS?

______ Never

______ 0-5 hours

______ 5-10 hours

______ 10-20 hours

______ More than 20 hours


6. How often have you played any Mario Kart game (Super NES, Nintendo 64, etc)?

______ Never

______ 0-5 hours

______ 5-10 hours

______ 10-20 hours

______ More than 20 hours



7. How did you learn about this study?

______ Email announcement

______ Facebook Marketplace posting

______ Facebook Group message

______ Michi-Poster (School of Information lounge)

______ From another person

______ Other (please specify) _______________________________________________




 CSCW to Go                                                               Page 86 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


Post-Task Questionaire for Task # 1

1. On a scale of 1 to 10, with 1 being not fun at all and 10 being extremely fun, how
would you rate your overall experience the previous gaming session? (circle 1 answer)

(not fun at all) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (extremely fun)


2. On a scale of 1 to 10, with 1 being very difficult and 10 being very easy, how easy
was it to play Mario Kart DS? (circle 1 answer)


 (very difficult) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (very easy)


3. On a scale of 1 to 10, with 1 being very dissatisfied and 10 being very satisfied, how
    do you feel about your performance playing Mario Kart DS? (circle 1 answer)

  (very dissatisfied) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (very
                                  satisfied)


4. Briefly describe anything you found to be particularly enjoyable about the
experience.




5. Briefly describe anything you found to be particularly frustrating about the
experience.




 CSCW to Go                                                               Page 87 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


Post-Task Questionaire for Task #2

1. On a scale of 1 to 10, with 1 being not fun at all and 10 being extremely fun, how
would you rate your overall experience the previous gaming session? (circle 1 answer)


 (not fun at all) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (extremely
                                   fun)


2. On a scale of 1 to 10, with 1 being very bad and 10 being very good, how would you
rate your experience interacting with the other person playing? (circle 1 answer)


   (very bad) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (very good)


3. On a scale of 1 to 10, with 1 being very dissatisfied and 10 being very satisfied, how
    do you feel about your performance playing Mario Kart DS? (circle 1 answer)

  (very dissatisfied) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (very
                                  satisfied)


4. Briefly describe anything you found to be particularly enjoyable about the
experience.




5. Briefly describe anything you found to be particularly frustrating about the
experience.




 CSCW to Go                                                               Page 88 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


Post-Task Questionaire for Task #3

1. On a scale of 1 to 10, with 1 being not fun at all and 10 being extremely fun, how
would you rate your overall experience the previous gaming session? (circle 1 answer)


 (not fun at all) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (extremely
                                   fun)


2. On a scale of 1 to 10, with 1 being very bad and 10 being very good, how would you
rate your experience interacting with the other person playing? (circle 1 answer)


   (very bad) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (very good)


3. On a scale of 1 to 10, with 1 being very dissatisfied and 10 being very satisfied, how
    do you feel about your performance playing Mario Kart DS? (circle 1 answer)

  (very dissatisfied) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (very
                                  satisfied)


4. Briefly describe anything you found to be particularly enjoyable about the
experience.




5. Briefly describe anything you found to be particularly frustrating about the
experience.




 CSCW to Go                                                               Page 89 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


Post-Task Questionaire for Task #4

1. On a scale of 1 to 10, with 1 being not fun at all and 10 being extremely fun, how
would you rate your overall experience the previous gaming session? (circle 1 answer)


 (not fun at all) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (extremely
                                   fun)


2. On a scale of 1 to 10, with 1 being very bad and 10 being very good, how would you
rate your experience interacting with the other person playing? (circle 1 answer)


   (very bad) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (very good)


3. On a scale of 1 to 10, with 1 being very dissatisfied and 10 being very satisfied, how
    do you feel about your performance playing Mario Kart DS? (circle 1 answer)

  (very dissatisfied) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (very
                                  satisfied)


4. Briefly describe anything you found to be particularly enjoyable about the
experience.




5. Briefly describe anything you found to be particularly frustrating about the
experience.




 CSCW to Go                                                               Page 90 of 96
SI 689 Final Report                                             Nintendo DS Evaluation


Post-Task Questionaire for Task #5

1. On a scale of 1 to 10, with 1 being not fun at all and 10 being extremely fun, how
would you rate your overall experience the previous gaming session? (circle 1 answer)


 (not fun at all) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (extremely
                                   fun)


2. On a scale of 1 to 10, with 1 being very bad and 10 being very good, how would you
rate your experience interacting with the other person playing? (circle 1 answer)


   (very bad) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (very good)


3. On a scale of 1 to 10, with 1 being very dissatisfied and 10 being very satisfied, how
    do you feel about your performance playing Mario Kart DS? (circle 1 answer)

  (very dissatisfied) 1 – 2 – 3 – 4 – 5 – 6 – 7 – 8 – 9 – 10 (very
                                  satisfied)


4. Briefly describe anything you found to be particularly enjoyable about the
experience.




5. Briefly describe anything you found to be particularly frustrating about the
experience.




 CSCW to Go                                                               Page 91 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


                           Nintendo® DS™ User Evaluation
                                   Final handout

1. Please rank the five tasks you just completed in order from 1 to 5, with 1 being the
task that was the most fun, and 5 being the task that was the least fun.

______ Task 1: Single player

______ Task 2: Compete with another player in different rooms

______ Task 3: Play on a team with another player in different rooms

______ Task 4: Play on a team with another player in the same room

______ Task 5: Compete with another player in the same room




 CSCW to Go                                                                Page 92 of 96
SI 689 Final Report                                              Nintendo DS Evaluation


                                Post – Test Questionnaire

Part A – Read aloud – moderator will record

Have you played this multi-player game (Mario Kart DS) for the Nintendo DS before?

       Yes -- No
       If yes, can you briefly tell us the difference of the experience between this
       gaming session and your previous experience when playing with others in Mario
       Kart DS?




Do you feel that the other player’s skill level was lower than yours, higher than yours, or
about the same as yours?

       Lower -- Same -- Higher

What caused you to feel that their skill level was (lower, the same, or higher based on
their respose)?




How do you feel that their skill level affected your enjoyment of the game?




 CSCW to Go                                                                Page 93 of 96
SI 689 Final Report                                               Nintendo DS Evaluation


Part B

   1. When you played with another person on the same team, what, if anything, did you like
      about the experience?




   2. What, if anything, did you dislike?




   3. When you competed with each other, what did you like about the experience?




   4.    What, if anything, did you dislike?




 CSCW to Go                                                                 Page 94 of 96
SI 689 Final Report                                               Nintendo DS Evaluation


   5. When you were in different rooms from each other, what, if anything, did you like about
      the experience?




   6. What, if anything, did you dislike?




   7. When you were in the same room with the other player, what, if anything, did you like
      about the experience?




   8. What, if anything, did you dislike?




 CSCW to Go                                                                  Page 95 of 96
SI 689 Final Report                                         Nintendo DS Evaluation


Part C
Was there any point during the experiment where something was vague or confusing to
you?




Is there anything else you would like to tell us today?




                                 [END OF QUESTION]




 CSCW to Go                                                           Page 96 of 96

								
To top