Automated Testing Better_ Cheaper_ Faster_ For Everything

Document Sample
Automated Testing Better_ Cheaper_ Faster_ For Everything Powered By Docstoc
					    Issues and Implementations
       in Multiplayer Game
 Online game development is difficult and riddled with expensive
   risks. Whether you are adding multiplayer support to a single-
 player game or fielding a massively multiplayer persistent virtual
world, the additional complexity of connecting your players over a
network adds substantial development time and testing costs. Get it
  right, and online can bring innovative game play and a sense of
  community to your player base. Get it wrong, and going online
     brings you a Pandora‟s box of missed deadlines, frustrated
               developers, and unhappy customers…
                              Playing with Friends
                              Exciting Competition
  Why Online?                 Community
                                                 Limited Media
                                                 Non-linear time
   Challenges                 Game Play              Persistence
                              Technology             Fairness
                              Our Focus

            Network Distortion     Multi-Player Inputs
Unreliability / Non-Determinism    Multi-Process Execution
                  System Complexity
       Tutorial Takeaway Messages

   Building online games with single-player
    game techniques is painful
   Early changes to your development process &
    system architecture to accommodate online
    play will greatly ease the pain
   “Lessons Learned” from our background will
    provide your project with useful “Rules of
      “A wise man learns from his own mistakes, a
         wiser man learns from someone else’s”
Network Terminology
A measure of the width or capacity of a communications
channel. Greater bandwidth allows communication of more
information in a given period of time.

Important: Bandwidth refers only to the amount of data a
channel can handle. As an analogy, a dump truck has a
higher bandwidth than a sports car. This does not mean that
the dump truck is faster, just more efficient at moving large
amounts of material

 Source: Ubisoft Online Terminology Guide, used with

An exception to a game‟s normal rule set that is included
by the game‟s developers. Usually, cheats are enabled
only in the single-player mode of the game (if it has one) or
were included for use (and forgotten about) by the
developers to aid with testing and/or the media.
This is any networking architecture in which one machine
(the server) is responsible for making final determinations
on what occurs in the game. This can range from game
designs in which the client merely displays information to
the user and returns user inputs to the server, to game
designs in which all machines are running nearly complete
versions of the game, and the server only serves to centralize
communications and adjudicate any disagreements.

Important: This is an architectural distinction. It is entirely
possible that the server is a consumer operated machine.

 Source: Ubisoft Online Terminology Guide, used with

A game in which all of the machines involved are run by
the consumer.

 Source: Ubisoft Online Terminology Guide, used with
        Dedicated Server

A client-server game in which there is a machine which is
serving as a server without having any local player. This
may be hosted by a consumer, or by a game service.

 Source: Ubisoft Online Terminology Guide, used with

A client-server game design in which the client is largely a
dumb input/output device.

  Source: Ubisoft Online Terminology Guide, used with

An unexpected or unintended game result generated by the
actions of the player that modifies the course or outcome the
game in a way not intended by the game‟s designers. Usually
the result of players discovering a bug in the game.

A modification made to a game or its data by an end user.
This can take the form of modifying either code or data, in
memory while the game is running, and/or to the game‟s
files on disk. The game itself or a service that it uses (such
as a driver), or the data it shares with other computers, or
hardware can be modified.
       Hybrid Game

Mix between Client-Server and Peer-to-Peer, where a
server determines the outcome for some events, while each
player‟s computer determine the outcome of some events
(usually local to that player).

A security product that employs a combination of hardware
and software to prevent unauthorized users or traffic from
the Internet from gaining access to a private local area
network. Many problems in online gaming arise because
firewalls block communication between games.

  Source: Ubisoft Online Terminology Guide, used with
       Integrated Server

A client-server game in which the machine which has a user
playing the game is also serving as the server. On some PC
applications, even though there may be a server running on
the same machine as a player, they are separate

 Source: Ubisoft Online Terminology Guide, used with
                 IP Address
This is the identifying number for any machine on the
Internet or for any LAN using the IP protocol. The original,
and most widely used form, is IPv4. Under IPv4, each
address is made up of four octets. If expressed as text, it
will take the form of "" (sometimes called
a dotted IP string). The broader Internet is slowly switching
over to IPv6 to increase the number of available addresses.
As the name implies, IPv6 addresses are made up of six

  Source: Ubisoft Online Terminology Guide, used with
                 IP Address
Additionally, under IPv4, the following rules apply to
network addresses: The local machine is always reachable
by the "loopback" address of, and the following
ranges of addresses are reserved for use on local LANs: - - -

  Source: Ubisoft Online Terminology Guide, used with
A slowing of the execution of a game. Lag can be caused by
two main factors: network issues or computer performance.

        Network lag cause an online game to miss
information bits necessary for the execution of it. For
instance, the last message indicating the position of another
player can be lost or is late, but the game has to display
something on the screen. Depending on how this situation
has been taken into account by the developers, the display of
the other player will freeze and the computer will wait for the
next message to arrive before continuing the execution of the
  Source: Ubisoft Online Terminology Guide, used with

Performance lag is caused by the overload of the main CPU
(too much information to process at a time) or the graphical
CPU (too many polygons or lightning to process).

 Source: Ubisoft Online Terminology Guide, used with

A measure of how long it takes a single bit to propagate
from one end of a link or channel to the other. Latency is
measured strictly in terms of time.

  Source: Ubisoft Online Terminology Guide, used with
            MAC Address

Media Access Control Address, a hardware address that has
been embedded into the network interface card by the
manufacturer to uniquely identify each node or point of
connection of a network.

  Source: Ubisoft Online Terminology Guide, used with
   NAT (Network Address

Enables a local area network to use one set of IP addresses
for internal traffic and a second set of IP addresses for
external traffic.

  Source: Ubisoft Online Terminology Guide, used with

An 8 bit value. Because a byte is the smallest addressable
amount of memory on the machine (6 bit, 7 bit, 9 bit, and
16 bit machines have all been on networks, along with the
now standard 8 bit bytes), network engineers like to use
octet to specify that we really mean 8 bits.

  Source: Ubisoft Online Terminology Guide, used with
In this networking architecture, all games are running full
versions of the game engine. Although one machine may be
used to host the setup of the game, once the game is running,
all machines are effectively equal. Disputes may be resolved
algorithmically (i.e. “claims about your own damage are
always valid”) which lends itself to cheating, or via a voting

Important: This is an architecture. It does not mean “any
game where all the machines involved are consumer

 Source: Ubisoft Online Terminology Guide, used with

A client-server game design in which the clients are running
relatively complete versions of the simulation, and the
server is validating claims, rather than determining all

  Source: Ubisoft Online Terminology Guide, used with
           Service Hosted

A game in which one or more of the machines involved is
provided by a game service (often the game‟s publisher).

  Source: Ubisoft Online Terminology Guide, used with

Transmission Control Protocol. TCP is one of the main
protocols in TCP/IP networks. Whereas the IP protocol
deals only with packets, TCP enables two hosts to establish
a connection and exchange streams of data. TCP guarantees
delivery of data and also guarantees that packets will be
delivered in the same order in which they were sent.

  Source: Ubisoft Online Terminology Guide, used with
A connectionless protocol that, like TCP, runs on top of IP
networks. Unlike TCP/IP, UDP/IP provides no error
recovery services. UDP/IP packets are CRC checked, but if
there is a CRC failure or part of a UDP/IP packet fails to
arrive, the packet is lost entirely. Additionally, where
TCP/IP is stream based, UDP/IP is datagram based, with all
the data in a UDP/IP send arriving at once, if it arrives at
all. It can be used for broadcasting messages over a local
network to find games, and is used by many games as the
primary game data channel.

 Source: Ubisoft Online Terminology Guide, used with
              What is a WAN?

   Wide Area Network (WAN) is a “network of
   Allows different networks to talk to each other
   Size and distance doesn‟t matter
            LAN Environment

   Local Area Network (LAN) shares common
    address space
   Actually independent of media; usually
   Also independent of protocol; usually TCP/IP
   Can “broadcast”
            WAN Environment

   Multiple networks
   Can have overlapping address spaces
   Very media dependent
   Uses edge device to hide the details
   Different types based on connection details
        WAN Connection Types

   Point-to-point – a single leased line between
    two sites
   Circuit Switching – like a phone call,
    connection made on demand
   Packet switched – Virtual circuits are created
                Point to Point

   A leased line from the provider
   Always on
   Fixed bandwidth
   Works like a pipe, whatever goes in at one end
    comes out the other
   Can set up an extended LAN environment
   More expensive
             Circuit Switching

   Similar to a leased line in that there is a
    physical connection between the two sites
   Gets set up and torn down so its only on when
   Similar to making a phone call
   ISDN is an example
                Packet Switched
   Uses provider infrastructure
   No real connection between sites, uses a virtual
   Routes data packets to final destination by whatever
    path is available
   Two types of circuits – Switched Virtual Circuit
    (SVC) and Permanent Virtual Circuit (PVC)
   Examples are ATM, Frame Relay, X.25, SMDS
               Virtual Circuits

   PVC‟s are always on – “permanently
   SVC‟s are brought up and down
     Call Setup
     Idle

     In use

     Tear down
             Quality of Service

   Quality of Service (QOS) requires a queuing
   Differs between media used
   Prioritizes data according to packet
   Line purchased may or may not support -
    Make sure you buy the right thing!
                Edge Device

   Any device that hides the complexity of what
    you are talking to from your LAN
   Does the protocol disassembly and reassembly
   Handles signaling
   Handles media conversion
   Router is an example

   Asynchronous Transfer Mode
   Used by Telco‟s to handle voice
   Breaks everything up into small, fixed size
   Very common, cheap
   T1‟s, T3‟s, OC3‟s etc.
   Need to test for equipment compatibility!
           Typical ATM Usage

   Local Exchange Carrier (LEC) provides
    physical line
   Line is located in the carrier‟s Point of
    Presence (POP) on your facility
   You connect your router to your internal
    Ethernet LAN
   LEC connects line to your carrier
       Typical ATM Usage cont.

   Your carrier (Sprint, ATT, Global Crossing,
    etc.) has a small, fixed number of hops to get
    you onto main trunk
   Main trunk will get your data to destination
    LEC site
   LEC to POP to Router to Destination LAN
   Magic is in how it knows where to go!
 Break until 11:15


Moving from Single Player to
                SINGLE PLAYER GAMES
   Use Case: Steal Ball Being Dribbled By Another Player

                                   Network Distortion

       Player A (San Francisco)                                    Player B (New York)
                                       Ball Position:
                                       State Updates

Local machine always has an accurate                          Remote machine always has
   representation of ball position                          an approximation of ball position
Issue: Kicking a Shared Soccer Ball

   Your opponent has the _exact_ position of the ball
   Your computer has only an _approximate_ position
    of the ball
   When you (the player or the coder) make a decision,
    it may be on incorrect data
   Level of distortion impacts quality of gameplay
   Inaccuracies of data lead to serious coding defects

The more computers involved, the greater the problem
Shared Worldview: Each Node Has A Potentially
   Different Representation Of Every Value
                Client A (Local Memory)
                       Clear Views        Game          Player
                                          Logic         Views
           True Values

                                                           Client B,C
             Damage                                      Distorted Views
 Player      Levels           AI
Position                     State

                                          Shared View
            Latency / Unreliablity
    At any slice in time, fuzzy, loosely coupled views
       produce an inconsistent shared worldview

Machine A, time t                                   Machine B , time t

                     Shared View @ (t) Is
                    ALWAYS DIFFERENT
              Network              C          Network

                         Machine C , time t
           The loss of precision

   Compression saves bandwidth
   What to compress – and how much?
   Possibly compress then decompress before you
    use it in game to minimise divergence
   Examples:
     Car position – vibration
     Car position – aggressive driving
    What is an atomic operation?

   Each message becomes an atomic operation
   Interframe and intraframe timing is not
    preserved across the network
   Troublesome case: Multiple frame actions
    such as shell transitions
   Troublesome case: Multi-step operations in
    which the interim cases are invalid at frame
Event Ordering And Atomicity Across Transactions
  Further Impact GamePlay And Coding Errors

                     Near-Endless (and illogical)
 Machine A             Ordering Are Possible                Machine B
                            A1, B1,B2,A2

             A1,A2          A2,A1,B1,B2
                             Machine C
         Indeterminate timing and
         sequencing of information
   Player A sends a stream of packets describing
    game state
   Timing:
     Player A picks up invulnerability pickup (10s)
     Player A gets shot 9.9s later

     If the second packet is delayed before arriving at
         Indeterminate timing and
         sequencing of information
   Sequencing:
     Player A picks up rocket-launcher
     Player A fires rocket

     If these arrive in the wrong order at B…
      Atomicity Across Transactions
         (Sharing Internal State)
                                      Client Process
           Logical Thread (A)                                 Logical Thread (B)
 1.   Generate House Info Request (11,11)
 2.   Set Local State: waiting for reply on house (11,11)

                                      Local House State (11,11)
                                                1.   Generate House Info Request (22,22)
                                                2.   Set Local State: waiting for reply on house (22,22)

                                      Local House State (22,22)
  3. EnterHouse (LocalHouseState)

Thread (A) acts off of incorrect data
TSO Solution: Encapsulate State &
    Logic in a State Machine
                Regulator A                                                    Regulator B
                                      Client Process
           Logical Thread (A)                                  Logical Thread (B)
 1.   Generate House Info Request (11,11)
 2.   Set Local State: waiting for reply on house (11,11)

                            Local House State (11,11)
                                                1.    Generate House Info Request (22,22)
                                                2.    Set Local State: waiting for reply on house (22,22)

                                                     Local House State (22,22)
  3. EnterHouse (LocalHouseState)

Thread (A) now acts off of correct data
 Latency (and why it isn't constant)

The internet:
 Latency (and why it isn't constant)

Router buffers:
 Latency (and why it isn't constant)

Different routes:
 Latency (and why it isn't constant)

Packet loss:
    API options: State Replication
        Versus Event Driven
           Client A                                             Client B

 State                                                 State
Changes     “Best Available”                          Changes     “Best Available”
          representation of state                               representation of state

     Shared State                                         Shared State
         DB                         Partial Updates           DB

Pros:                                           Cons:
 Intuitive Model                                 API can create new problems
 Hides Network Optimizations                     Hidden network costs
    Support Shared State & Events
            Client A                                              Client B
  State                                                  State
 Changes     “Best Available”                           Changes     “Best Available”
           representation of state                                representation of state

      Shared State                                           Shared State
          DB                         Partial Updates             DB

Shared State:                                      Events:
 Instrument for volume & latency                    Use for stateless transactions,
 Optimize to network budget                         Immedidate actions
       Shared WorldView: Summary
    (Impact On Players & Programmers)
   Player immersion: jerky visuals, occasional
   Fairplay: who killed who?
       Well, who shot first? How can you tell?
       Was Player A actually located where Player B shot him?
   Coding issues
       Race conditions
       Event ordering & loss of precision can change future
        outcomes per machine
       Bugs of this sort are hard to reproduce …
       Shared WorldView: Summary
           (Mitigating Factors)
   High degrees of precision are rarely required
   Predictable motion patterns simplify hiding the
    latency (dead reckoning, predictive contracts, et al)
   Rule of Thumb: design game play to minimize the
    required accuracy (and volume) of shared data
   Useful tactic: handle all inter-process communication
    via state machines that handle each transaction from
    initiation to conclusion (TSO‟s Regulator template
    worked well)
                    Scale and Scaleability

   Scale: the N2 problem
       Data Transmission Optimizations: minimize the network
        cost of sharing the required amount of Shared State
            Interest management, predictive contracts (dead reckoning),
             variable resolution, compression, …
       Application Scalability: minimize the amount of required
        Shared State
       References:
   Bandwidth: these days, who cares?
       Caveat: see “Scale”!
       Best bet: metrics, metrics, metrics
            Set budget per player, optimize to that point, then STOP
       Danger: pre-mature optimization
       Danger: game play changes that break optimizations
          Why “every” and “all” are
   SP: all information is always available
   MP: shared information must be packaged,
    transmitted and unpackaged
       Each step costs CPU & bandwidth, and can happen 10‟s to
        100‟s of times per minute
       May also cause additional overhead (e.g. DB calls)
   Scalability is key: many shared data structures grow
    with the number of players, AI, objects, terrain, …
       Caution: early prototypes may be cheap enough, but as
        game progresses, costs may explode
           Why “every” and “all” are
   Example: TSO‟s Data Service
   Initial design: Transmit entire ReservedLotList to all
    connected clients, every 30 seconds
   Initial fielding: no problem
       Development testing: < 1,000 Lots, < 10 clients
   Complete disaster as clients & DB scaled
       Shipping requirements: 100,000 Lots, 4,000 clients
       LotElementSize * LotListSize * NumClients
       20 Bytes * 100,000 * 4,000
       8,000,000,000 Bytes, TWICE per minute!!
Why “every” and “all” are

                 22,000,000 DS
                 Queries! 7,000
                  next highest
                   Costs of QA

   6 player game – you need at least 6 testers at
    the same time
     Testers
     Consoles, TVs and disks
     Network connections

   End-user grade network connections (ADSL)
   More players - more likely to see hard-to-find
    bugs so they are marked as online bugs
   Automated testing
    Debugging: Trials & Tribulations

   MP games require MP inputs to test
       Often, sufficient machines & QA testers not available
       Developers significantly handicapped
   Non-determinism is a serious issue
       Running the same test twice does NOT necessarily give the same result
   Offline code changes frequently breaks online code
   Run-time debugging of networked games often becomes post-
    mortem debugging
   What helps
       Automated testing
       Architectural support for forcing causality (but $$)
       Strong isolation of online / offline code
     Debugging: Trials & Tribulations

Simple TSO Test:
One avatar holds the
Lot open, while 3
Avatars jump in & out
Debugging: Trials & Tribulations
    Debugging: Trials & Tribulations
   TSO EnterLot: 30 test runs, 4 behaviours
     Successful entry
     Hang or Crash
     Owner evicted, all possessions stolen
   Random results observed in all major features:
    very difficult to track
   What worked
     Massive repetition of tests to establish true
      pass/fail conditions
     Continual repetition of unit tests (monkeys rock!)
     Initial calibration tests: 400x runs per unit test
           7% to 30% failure of any given unit test
     Debugging: Trials & Tribulations

Monkeys: Continual Repetition of Critical Path Unit Tests

Code Repository       Compilers          Reference Servers
     Impact On The Development Team

   Changes to non-networked gamplay often break
    networked play (shared code & implicit assumptions
    & timing changes)
   Changes to online code can inhibit content
    development (“server down” often equals “nobody
   Non-determinisim produces frustrating defects
   Mitigating factors
       Make sure people can work in isolation: no “critical path”
        failures that bring down the team
       Strong testing in the dev team
       Online/offline code sharing via modules, not #ifdef
       Stability Analysis:
  What Brings Down The Team?

Test Case: Can an Avatar Sit in a Chair?
use_object ()                      Failures on the
   buy_object ()                   Critical Path block
                                   access to much of
      enter_house ()               the game.
          buy_house ()             Worse, unreliable
                create_avatar ()
                   login ()

                          Time to Fix
                        Cost of Detection
          Process Shift: Comb Filter Testing

Sniff Test, Monkey Tests   Smoke Test, Server Sniff Full Feature Regression, Full Load Test
 - Fast to run              - Is the game playable? - Do all test suites pass?
 - Catch major errors       - Are the servers stable   - Are the servers stable
 - Keeps coders working       under a light load?        under peak load conditions?
                            - Do all key features work?

                 $                    $$                          $$$
New code ready    Full system build         Promotable to           Promotable to
 For checkin                                 full testing          paying customers

        • Cheap tests to catch gross errors early in the pipeline
        • More expensive tests only run on known functional builds
Lunch Break until 2:00

     2:00pm – 4:00pm

       Case Studies
Break until 4:30

  4:30pm – 5:30pm

  Post Ship Issues
        Cheating: The Big Picture

   A few serious questions every developer
    should ask:
     Why do we care if anyone cheats at our game?
     Is it really a problem?

     What is our relationship with our customers?

     What are we willing to do about it?

     What about our publisher?
                Who is cheating?
   Anyone who can get away with it
       Profiling is valid here – Young males are at
        greatest risk.
   We have a win at all costs culture that does not
    value honesty
   The Anonymous nature of the net allows
    people to step outside their normal behavior
   Single player cheats have not helped
           Why are they cheating?

   To Win
   To cause others not to win
       Crashing others is considered a „win‟
   To exercise control
       Almost pathological need for some
   To show off (attention)
     Matt’s Rules about Cheating
   Rule #1: If you build it, they will come -- to hack and
   Rule #2: hacking attempts increase with the success
    of your game.
   Rule #3: Cheaters actively try to keep developers
    from learning their cheats.
   Rule #4: Your game, along with everything on the
    cheater's computer, is not secure. The files are not
    secure. Memory is not secure. Services and drivers
    are not secure.
                 Rules, cont.

   Rule #5: Obscurity is not security.
   Rule #6: Any communication over an open line
    is vulnerable to interception, analysis, and
   Rule #7: There is no such thing as a harmless
    cheat or exploit. Cheaters are incredibly
    inventive at figuring out how to get the most
    out of any loophole or exploit.
                More Rules…

   Rule #8: Trust in the server is everything in a
    client-server game. Actually its true for all
    game types.
   Rule #9: Honest players would love for a game
    to tip them off to possible cheating. Cheaters
    want the opposite.
   Rule #10: No one likes to have their ass
    handed to them on a platter every time
                  Rules, cont.

   Rule #11: Once a cheat has been perpetrated,
    a patch, sequel, or similar game will suffer
    similar hacking attempts sooner than later.
   Rule #12: There is more than one way to
    perpetrate a cheat.
   Rule #13: It only takes one person to break an
    online community.
              The Last Rule…

   Rule #99: There is no “Silver Bullet” to make
    your game’s vulnerabilities go away. It is an
    ongoing, never ending war. What a developer
    can do is to choose which battles are worth
        Reflecting on the rules…

   What is the purpose of a computer game?
     To entertain & create enjoyable experiences
     To make the developer money

     To make users desire your future products

   Who owns the game experience?
     What exactly did the player buy?
     What are the obligations and expectations?
    General Categories of “Cheats”

   Reflex augmentation
   Authoritative clients
   Information exposure
   Compromised servers
   Bugs and design loopholes (exploits)
   Environmental weaknesses
   Social Cheats
          Reflex Augmentation

   Anything that replaces human skill & response
    with automated input.
   FPS Aiming proxies or bots.
   Situation analyzers or AI players.
   Timing assistants.
   Changes the game from a contest of two
    humans to one of man vs. machine.
           Authoritative Clients

   When a player‟s client is the final arbitrator of
    any aspect of the game
     Less of an issue for server based games
     Obviously someone has to make some definitive
      decisions in a peer to peer game.
     Can all players make the same decision and then
      compare their conclusions?
         Information Exposure

   The revealing of any game information hidden
    from the player
     More of a problem for synchronous simulations
     Can be totally passive

   RTS Map & Fog of War hacks
   Skin & model hacks
   View of other player‟s status
          Compromised Servers

   Related to general PC security issues
   More likely done by the server operator, but
    could be done by someone else
   A perceived problem with clans
   How to detect?
   Ties into matchmaking issues.

   Coding bugs and design loopholes
   Catchall category of problems
   Often discovered by accident
   Question of who makes the call
      Environmental Weaknesses

   When the game‟s environment is subverted
   Corrupted Communication data
   Hacked & Modified Drivers
   OS bugs
   Hacked System Clocks
   Bandwidth and latency calculations
                Social Cheats

   Social Engineering that takes place inside of a
   Online forms of issues and scams we‟ve had in
    person for thousands of years.
   Player Collusion and betrayal.
   Player in the same physical location.
   Player controls pace of game abusively
   Player text/voice chats others abusively
    Other problems: Identity Theft

   The impersonation or hijacking of a player‟s
    online identity and legacy.
   More of problem as we push MMO games,
    persistent games and meta-games.
   Players often have a lot invested in their on-
    line avatars and identities.
        Other: Reverse Engineering

   Reverse engineering of Game Rules and Stats
     This is a design issue
     Non published Statistics

   Random Number Generators
                Some Grey Areas

   Engine & Game Configurations
       Separates the „hardcore‟ player from the causal
   Connection Type
       Not much we can do.
   Equipment Configuration
       Ditto for PC
              Grey Areas cont.

   Cheaters vs. Impossibly Good Players
   What happens if someone can‟t tell the
     The skill curve of your game
     Can be prohibitive to new players
       What the developer can do
   Plan for anti-cheating efforts in the schedule.
   Audit your (final) communication formats.
   Audit your data & program layout.
   Attempt to make cheats using your insider
   Plan for a patch.
   Provide auditing tools to your players.
   Create persistent player identities.
    What developers can do, cont.

   Know in advance the public stance you and
    your publisher will take if a cheat is found.
     Denial can work, but is risky
     Honesty is appreciated by players
            But it could be an admission of liability?
   Communicate with your players
       Active community management.
            Cultivate contacts in the user community
How long do you support a game
        after it ships?
   Problem – no available staff
   Problem – It costs $$$
   One solution – metric based on sales
   Make a decision, be willing to set an EOL date
    for support?
        Dealing with Authoritative
   Run the full simulation on all machines
   Communicate only requests, not actions
     Harder than you think
     Assume nothing regarding game rules

   Verify the full state of the simulation
       Great debugging benefits
         Combating Information
   Add detection processes to client.
   Share information on the engine‟s display
     Good for complex data
     Simple data can still be passively extracted

   Audit previous gameplay for information
   Use Honeypot designs.
        Dealing with Compromised
   Give the users tools to determine the server‟s
    state & configuration.
   Link Identities to operators.
       Allow users to filter operators
   Use third party products like Punkbuster or
            Dealing with Exploits

   Patch, baby, patch….
       Most effective patch case
   Have a consistent policy regarding changes
     And document the changes to your users
     changes in patches not communicated can be seen
      as new cheats or bugs.
    Other things developers can do

   Develop an internal “cheater” team from your
    QA team during the development process.
       Provide them with assistance to crack your own
   Provide Tools to combat Social Cheating
       Recorded games and auditing tools
Programming Recommendations
   Assuming that your game will be hacked for
    the purpose of cheating.
     Make an effort to determine the likely forms that
      the cheat will take, and put code in your game
      whose sole purpose is to detect evidence that a
      cheat has taken place.
     For very likely cheats, place detection code at
      multiple levels in the game if possible.
     Once cheating has been detected, delay responding
      if possible.
         More recommendations

   Single player cheats need to go in their own
    separate game system.
   Have your game validate the integrity of its
    executables, dynamic link libraries, and non-
    variant data files
              Check your output!
   Don‟t leave debug symbols or developer functionality
    compiled into the game‟s executable files.

   Always scan your game‟s executable files for strings and other
    revealing data before releasing the files to anyone outside of
    the development team.

   Make sure that all debugging and developer code is actually
    not compiled in external release versions of your game.
    Don’t forget to check data files
   Make sure that all developer and debugging
    information and functionality is removed from the
    data files that accompany the game‟s executables.

   If your game uses a scripting language or other data
    system that exposes human readable function or
    command names, implement a tokenizer or compiler
    that will change that data to a non-obvious form, and
    don‟t release the uncompiled data files.
    More thoughts about cheating
   Never underestimate the techniques hackers will use
    to rip apart a game.

   Once a code-related hack has been created, fixing it
    with a patch can be a no-win situation.
       Get it right the first time when you patch.
       Consider performing radical changes to the game‟s code to
        fix a hack.
       Previous versions of a game provide a source of difference
        data for a hacker.
     Patching Multiplayer Games
   Patching on consoles ranges from impossible, to very
    difficult, to “only” requiring a certification pass.
   Who pays for the patch to be built?
   How do you get it to your players?
       Network budget
       Impact on the player experience
       Differential copies: game changes to simplify the problem
   What conditions require a patch
       How bad do things have to be to require a bug patch?
       Or do you patch to deliver new content?
       Remember that any patch has a customer service cost.
       Remember that the nature of a multiplayer game requires
        the players to patch if they want to play with others.
       Long-term customer support
   Making support calls not happen must be a priority.
    Especially for games without a recurring income
    stream, any CS call may put you cash negative on the
   Poor customer support can drive customers away
   However, many customers may have unreasonable
    expectations and they will never be satisified,
    regardless of the CS quality.
   Official forums tend to be extremely negative. Happy
    players usually don‟t post, and unhappy players want
    everyone to know they are unhappy, and will post in
    the only forum they know the developer/publisher
          Cross platform support

   Risks: All platforms that are interoperable
    must be able to be patched roughly
   Risks: Small differences in floating point
    processing can give an unintentional advantage
    to some platforms
   Risks: Depending on the differences between
    platforms, input devices may give a significant
                     New Content
   MMP: New content must be added, constantly
       Strong impact on old content
       Gameplay balances, long term additive properties (N2‟ed?
       Code rot, tools, and long term maintainability
   Free vs user created vs commercial expansions
       Legal issues
       Distribution & installation with N prior expansions
       Gameplay imbalances
       Providing hooks for player generated content is not cheap!
Open Q&A

5:30pm - 6:00pm
   References available as part of the final slide

   Final Slides available online @