Docstoc

GDC repeatability

Document Sample
GDC repeatability Powered By Docstoc
					     Scalability Tools: Automated
                Testing
                     (30 minutes)
Overview

Hooking up your game
   external tools
   internal game changes




Applications & Gotchas
   engineering, QA, operations
   production & management


Summary & Questions
 Review: controlled tests & actionable
 results useful for many purposes
             (1)
                               Repeatable tests, using N
                               synchronized game clients


             (2)
Test Game
                                 High-level, actionable
                               reports for many audiences




 Programmer Development Director   Executive
 Automated testing accelerates large-scale
 game development & helps predictability



                                                             Better game



                                              earlier            Ship Date
  % Complete
                                                                  Oops

                                       autoTest
                             Time Time
     Project                                            Target
      Start                                             Launch
TSO case study: developer efficiency
           Strong test support
           Weak test support
  Measurable targets & projected trends give
  you actionable progress metrics, early enough
  to react

                                                        Target

                                                         Oops

   Any test
(e.g. # clients)




                                   Time
                   First Passing          Now    Any Time
                        Test                    (e.g. Alpha)
Success stories

   Many game teams work with
    automated testing
       EA, Microsoft, any MMO, …


   Automated testing has many highly
    successful applications outside of
    game development

   Caveat: there are many ways to fail…
How to succeed

   Plan for testing early
       Non-trivial system
       Architectural implications


   Fast, cheap test coverage is a major
    change in production, be willing to
    adapt your processes
       Make sure the entire team is on board
       Deeper integration leads to greater value

   Kearneyism: “make it easier to use than not
    to use”
Automated testing components

                              Any Game




    Startup            Repeatable, Sync’ed                   Collection
       &                                                         &
    Control                 Test I/O                          Analysis

Test Manager           Scriptable Test Client(s)           Report Manager
Test Selection/Setup    Emulated User Play Session(s)      Raw Data Collection
 Control N Clients       Multi-client synchronization   Aggregation / Summarization
     RT probes                                                Alarm Triggers
Input systems for automated
testing
                        scripted
  algorithmic                              recorders




                      Game code


Multiple test applications are required, but each input type
 differs in value per application. Scripting gives the best
                         coverage.
 Hierarchical automated testing
                     subsystem
    unit                                        system




Multiple levels of testing gives you
• Faster ways to work with each level of code
• Incremental testing avoids noise & speeds defect isolation
 Input (Scripted Test Clients)

      Pseudo-code script of users play the game,
       and what the game should do in response
                   createAvatar [sam] Command steps
                   enterLevel 99
                   buyObject knife
                   attack [opponent]

Validation steps   checkAvatar [sam exists]
                                                   …
                   checkLevel 99 [loaded]
                   checkInventory [knife]
                   checkDamage [opponent]
…
Scripted Players: Implementation
                            Or, load
Test Client (Null View)                Game Client
                            both


    Script Engine                      Game GUI


         State                                   State
                           Commands

                     Presentation Layer
                          Game Logic
  Test-specific input & output via a data-
  driven test client gives maximum
  flexibility

 Regression                             Load
                        Reusable
                     Scripts & Data

                  Input
                   API

                    Test Client
                  Output
                   API
                                        Pass/Fail
Key Game States
                   Script-Specific    Responsiveness
                   Logs & Metrics
A Presentation Layer is often
unique to a game
   Some automation scripts should read
    just like QA test scripts for your game

        NullView
    TSO examples          Client
     routeAvatar, useObject
     buyLot, enterLot
     socialInteraction (makeFriends, chat, …)
Input (data sets)

                        Repeatable tests in
  Mock data       development, faster load, edge
                            conditions
                    Unpredictable user element
  Real data           finds different bugs



  Repeatable       Debugging & benchmarking


                      Edge cases & real world
   Random                 performance
Common Gotchas
   Not designing for testability
       Retrofitting is expensive
   Blowing the implementation
     Brittle code
     Addressing perceived needs, not real needs
   Use automated testing incorrectly
     Testing the wrong thing @ the wrong time
     Not integrating with your processes
     Poor testing methodology
    Testing the wrong time at the wrong time
                 Applying detailed testing while the game design is still
                shifting and the code is still incomplete introduces noise
                          and the need to keep re-writing tests

                                   Alpha                              Alpha



       Design                                 Code
       Space                                Completion




                         Time                               Time

     Build Acceptance Tests
                 (BAT)
 Stabilize the critical path for your team
 Keep people working by keeping critical things from
breaking

Final Acceptance Tests (FAT)
   Detailed tests to measure progress against milestones
   “Is the game done yet?” tests need to be phased in
More gotchas: poor testing
methodology & tools
   Case 1: recorders
       Load & regression were needed; not
        understanding maintenance cost
   Case 2: completely invalid test
    procedures
       Distorted view of what really worked (GIGO)
   Case 3: poor implementation planning
       Limited usage (nature of tests led to high
        test cost & programming skill required)
 Case 4: not adapting development
  processes
 Common theme: no senior engineering
  analysis committed to the testing
  problem
    Automated Testing for Online
              Games
Overview
Hooking up your game
   external tools
   internal game changes




Applications
   engineering, QA, operations
   production & management

Summary & Questions
                     Automated testing: strengths
                         Repeat massive numbers of
                          simple, easily measurable
                          tasks
                         Mine the results
                         Do all the above, in
                          parallel, for rapid iteration


“The difference between us and a computer is that
the computer is blindingly stupid, but it is capable of
being stupid many, many millions of times a second.”
   Douglas Adams (1997 SCO Forum)
 Semi-automated testing is best for
 game development
                       Testing
                     Requirements
Automation                          Manual Testing
   Rote work                          Creative bug
    (“does door108                      hunting, visuals
    still open?”)                      Judgment calls,
   Scale                               playability
   Repeatability                      Reacting to change,
   Accuracy                           Evaluating autoTest
   Parallelism                         results




Integrate the two for best impact
Plan your attack with stakeholders
(retire risk early: QA, Production, Management)


   Tough shipping requirements (e.g.)
     Scale, reliability
     Regression costs

   Development risk
     Cost / risk of engineering & debugging
     Impact on content creation

   Management risk
       Schedule predictability & visibility
Automation focus areas (Larry’s “top
5”)

  Performance          Scale is hard to get right

  Critical path        Keep team going forward
     stability
Non-determinism            Gets in the way of
                                 everything
Content regression       Massive, recurring $$

 Compatibility &      Improves life for you & user
     install
Yikes, that all sounds very
expensive!
   Yes, but remember, the alternative costs are
    higher and do not always work
   Costs of QA for a 6 player game – you need
    at least 6 testers at the same time
        Testers
        Consoles, TVs and disks & network
        Non-determinism

   MMO regression costs: yikes2
        10s to 100s of testers
        10 year code life cycle
        Constant release iterations
Stability: keep the team
working!
(TSO use case: critical path analysis)


Test Case: Can an Avatar Sit in a Chair?
 use_object ()
                                     Failures on the Critical
     buy_object ()                    Path block access to
                                       much of the game
          enter_house ()
             buy_house ()
                  create_avatar ()
                     login ()
    Prevent critical path code breaks that
            take down your team

      Candidate code


Development

                               Safe code
                       Sniff
                       Test
        Pass / fail,
        diagnostics
                                     Checki
                                       n
 Stability & non-determinism (monkey
 tests)

      Continual Repetition of Critical Path Unit Tests




Code Repository         Compilers          Reference Servers
AutoTest addresses non-
determinism

   Detection & reproduction of race
    condition defects
       Even low probability errors are exposed
        with sufficient testing (random,
        structured, load, aging)

   Measurability of race condition
    defects
       Occurs x% of the time, over 400x test
        runs
Monkey test: enterLot ()
Monkey test: 3 * enterLot ()
Four different behaviors in thirty
runs!
Content testing (areas)

   Regression
   Error detection
   Balancing / tuning

   This topic is a tutorial in and of itself
       Content regression is a huge cost problem
       Many ways to automate it (algorithmic, scripted &
        combined, …)
       Differs wildly across game genres
Content testing (more
examples)
 Light mapping, shadow detection
 Asset correctness / sameness
 Compatibility testing
 Armor / damage
 Class balances
 Validating against old userData
 … (unique to each game)
    Automated Testing for Online
              Games
            (One Hour)
Overview
Hooking up your game
   external tools
   internal game changes




Applications
   engineering, QA, operations
   production & management

Summary & Questions
Summary: automated
testing
   Start early & make it easy to use
       Strongly impacts your success


   The bigger & more complex your
    game, the more automated testing you
    need

   You need commitment across the
    team
       Engineering, QA, management, content
        creation
Q&A & other resources
   My email: larry.mellon_@_emergent.net
   More material on automated testing for games
       http://www.maggotranch.com/mmp.html
            Last year’s online engineering slides
            This year’s slides
            Talks on automated testing & scaling the development
             process
       www.amazon.com: “Massively Multiplayer Game
        Development II”
            Chapters on automated testing and automated metrics
             systems
       www.gamasutra.com: Dag Frommhold, Fabian Röken
            Lengthy article on applying automated testing in games
       Microsoft: various groups & writings
   From outside the gaming world
       Kent Beck: anything on test-driven development
       http://www.martinfowler.com/articles/continuousIntegrati
        on.html#id108619: Continual integration testing
       Amazon & Google: inside & outside our industry

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:1
posted:3/30/2011
language:English
pages:35