Testing Tools and Tips by theslasher

VIEWS: 145 PAGES: 73

									Testing Tools and Tips
   Quality is Never an Accident




                make
              better
               games
Your Speaker
•   Evan Birkby
•   QA Manager Totally Games
•   QA Manager Spectrum HoloByte
•   QA Engineer at Dynamix
•   Recent Products
    – X-Wing Alliance
    – Bridge Commander
    – Star Trek Generations

                      make
                    better
                     games
Tools Overview
•   Bug Database
•   Compatibility Database
•   Dialog Database
•   The Test Plan
•   Developer Tools and Testing
•   Test Equipment
•   QA Management


                    make
                  better
                   games
A Bug Database
• Where does the Bug Database begin?
• What does it need to do?
• Who Will Use It?
  – What are the Functions?
  – What lists are needed?
  – What bug counts will be used?
  – How will bugs be classified?
  – How is bug status going to work?


                    make
                  better
                   games
Where does the Database begin?
• Understand the Need.
• Establish responsibility.
• Pick the development tools.
  – Access
  – FileMaker Pro
  – Tracker
  – Programming
  – Using templates


                    make
                  better
                   games
What does it need to do?
•   It should track bugs (features).
•   Bugs should be classified.
•   Entry of bugs should be efficient.
•   Bugs should be categorized.
•   Searching should be made easy.
•   Mistakes should be easy to update.
•   Proper security is needed.
•   It shouldn’t be frustrating.
                    make
                  better
                   games
Who will use it?
•   Testers.
•   Designers.
•   Programmers.
•   Marketing.
•   Management.




                     make
                   better
                    games
What are the functions?
•   Bug Entry
•   Fixer Feedback
•   Bug Status
•   Database Management
•   The White Board
•   Bug Counts
•   Bug Charts


                  make
                better
                 games
Bug Entry
•   Anyone that enters bugs has access.
•   Keep it simple and fast.
•   Automate repetitive tasks.
•   Allow custom editing of value lists.
•   Provide a good help system.




                     make
                   better
                    games
  make
better
 games
Fixing
•   Anyone that fixes bugs has access.
•   QA can enter written feedback.
•   Keep it very simple and fast.
•   Automate tracking tasks.
•   Ask for feedback but be reasonable.
•   Provide a good help system.



                    make
                  better
                   games
Bug Status – QA Verification
•   Testers/QA have access.
•   Tracking maintenance is available.
•   Search functions need to be fast.
•   Search functions need to be effective.




                     make
                   better
                    games
Database Management
•   Track changes to records.
•   Allow fields to be corrected.
•   Provide proper levels of security.
•   Make navigating the database easy.




                    make
                  better
                   games
The White Board
•   List major focus issues.
•   Put up milestone objectives.
•   Inform team of recent changes.
•   Show who is working on what.
•   Provide flowchart information.
•   List cheats and hints.



                    make
                  better
                   games
Bug Counts
•   Count bugs by Priority and Severity.
•   Count bugs by Fixer.
•   Count bugs by Tester.
•   Evaluate Count with Totals.
•   Evaluate Count with Charts.




                     make
                   better
                    games
The Quake Scale
• Damage per bug:
• Magnitude = 1000, 100, 10 for 1, 2, 3
• Magnitude = 1000, 100, 10 for A, B, C

• DPB = Priority x Severity
• Total DPB = the sum of the DPB.
• Quake = log Sum( Mag. x Mag.)


                   make
                 better
                  games
How will bugs be classified?
• Severity: How noticeable is the bug?

• Significance: How many users will
  the bug impact?

• Frequency: How easy is the bug to
  reproduce?


                   make
                 better
                  games
How will bugs be classified?
• Difficulty: How hard will it be to fix
  the bug?

• Risk: How risky is it to fix the bug?

• Priority: How soon must the bug be
  fixed?


                    make
                  better
                   games
How will bugs be classified?
• Milestone requirements?

• Bug Status (Open):
  – Open – New Bug.
  – Reopened. [Feedback]
  – Need more information. [Feedback]
  – On hold, Deferred [Eliminate these].



                    make
                  better
                   games
How will bugs be classified?
• Bug Status (Closed):
  – Closed. [Duplicate, Feature removed]
  – Fixed.
  – Confirm [Test needs to do something]
  – ISV [In Shipped Version, Deferred]
  – [Will not Fix]
  – [Will fix next milestone]




                    make
                  better
                   games
  make
better
 games
  make
better
 games
Compatibility Database
•   It should be relational.
•   Categories should be editable.
•   Performance should be tracked.
•   Issues to be researched.
    – What chip sets will be used?
    – What will be available in the future?
    – Is there a possible software solution?



                       make
                     better
                      games
It can be relational.
• Create a system inventory link.
• Link the user name to the system.
• Link customer support information.
  – Problem reports.
  – Special system configurations.
• Link to staff contact information.




                    make
                  better
                   games
Categories should be editable.
• Suggested Categories.
  – CPU, Clock Speed, Ram, Cache.
  – Video Card, Video Ram, Resolution, Depth.
  – Controllers, Mice, Joysticks, Wheels.
  – Operating system, network drivers.
  – Driver version numbers.




                    make
                  better
                   games
Performance should be tracked.
•   Determine the frame-rate.
•   Check memory usage.
•   Track load times.
•   Recommend configuration settings.




                    make
                  better
                   games
Issues to be researched.
• What chip sets will be used?
  – Identify the chips used in the product.
• What will be available in the future?
  – Use Manufacturer test labs.
  – Test with Beta systems and software.
• Is there a possible software solution?
  – Fix the driver.
  – Bypass hardware bugs.


                     make
                   better
                    games
Quality
• Is it significant?




                     make
                   better
                    games
The Test Plan
• The descriptive plan.
• The checklist.
• The chart.




                   make
                 better
                  games
The descriptive plan.
•   Describe the testing process.
•   List the tools and people involved.
•   Define Terms.
•   Make a schedule.
•   Identify Milestones.
•   Include a testing tutorial.



                     make
                   better
                    games
Define Terms
• Explain acronyms.
  – BLT, ISV, NMI, NAB
• Explain bug classifications.
  – A, B, C, R, S, 1, 2, 3
• Define testing vocabulary.
  – Acid Rain
  – Memory Leak



                        make
                      better
                       games
Make a schedule.
• Assign test team to specific areas.
• Work with the development plan.
• Show staffing levels per project.




                   make
                 better
                  games
Identify Milestones.
• Pre Alpha, Alpha, Pre Beta, Beta.

•   Submitted to test.
•   Testing certified.
•   Testing complete.
•   QA acceptance.

• Partial vs. complete testing.
                     make
                   better
                    games
Include a testing tutorial.
•   Explain memory leaks.
•   Describe game loops and lockups.
•   Explain optimization issues.
•   Explain data packets in multiplayer.
•   Provide historical literature.
•   Explain the testing tools.
•   Talk about the competition.


                     make
                   better
                    games
The Checklist
•   List areas that need to be tested.
•   Put bugs in the database.
•   Make time estimates for sections.
•   Make assignments for sections.
•   Summarize results.




                     make
                   better
                    games
List areas that need to be tested.
• Group the tasks into areas.
  – Keep similar elements together.
  – Separate by QA acceptance.
• Flag items that need to be checked.
• Flag items that have been checked.
• Flag items needing multiple passes.




                    make
                  better
                   games
Put bugs in the database.
•   Bugs can be better described.
•   The database has better tracking.
•   Retesting will be properly covered.
•   The test plan can have closure.
•   Progress can be measured.

• Note: Use multiple pass flag.


                     make
                   better
                    games
Make time estimates for sections.
• Estimate time for best case.
• Estimate time for expected case.
• Estimate time for worst case.

• Estimate = (1 x B)+(3 x E)+(2 x W) / 6




                      make
                    better
                     games
  make
better
 games
  make
better
 games
The Chart
•   List missions, scenarios, maps.
•   Indicate progress by category.
•   Identify problem areas.
•   Define progress points.
•   Indicate what testing will focus on.
•   Carefully identify dependencies.



                     make
                   better
                    games
  make
better
 games
Bug Count Charts
• Chart progress in the Test Plan.
• Chart data from the bug database.
• Calculate the Quake Scale.




                  make
                better
                 games
The Quake Scale
• Magnitude = 1000, 100, 10 for A, B, C
• Magnitude = 1000, 100, 10 for 1, 2, 3

• Quake = log Sum( Mag. x Mag.)




                   make
                 better
                  games
Quality
• Is always the result of intelligent
  effort.




                    make
                  better
                   games
Developer Tools
•   Scripts
•   Editors
•   Source Safe.
•   Programming tools
•   Source text
•   Design Documents



                   make
                 better
                  games
Scripts
• Python, Java, Visual Basic.
• Developer scripting tools.

• Black Box vs. White Box testing.
• Automated Testing Tools.
  – Microsoft Test




                       make
                     better
                      games
Editors
• 3D and 2D editors.
  – Understand the limits of the tools.
  – Black box testing of scripts.
• 3D modeling tools.
  – Model tools can check dimensions.
  – Texture tools can verify placement.
  – Animation tools can find frame jumps.
• Attribute Databases
  – Check values for accuracy and balance.

                     make
                   better
                    games
Attribute Data
•   Check for projectiles that remain.
•   Check for X,Y,Z position errors.
•   Verify charging rates.
•   Verify loading rates.
•   Check math near extreme values.
•   Look for unused attributes.



                     make
                   better
                    games
Source Safe
•   Use the comments field for changes.
•   Show comments with new Builds.
•   Freeze check-ins but limit duration.
•   Make build patches.
•   Display version numbers in build.
•   Avoid frequent fixes to save/load.
•   Let test make the builds.


                     make
                   better
                    games
Programming Tools
•   System Resource Monitors.
•   Memory Eaters.
•   Frame rate counters.
•   Running with a debugger.
    – Copy scripts into the bug record.
    – Find the problem in the code.




                       make
                     better
                      games
Source Text
• Make a dialog database.
• Ask for access to all text in the game.
• Track dialog elements.
  – Give each line a unique id.
  – List the speaker and character.
  – List the effect, locations, and audio.
  – Describe the delivery.
  – Indicate the progress and bugs.
• Indicate text size. Allow 30% more.
                      make
                    better
                     games
Design Documents
•   Build the test plan.
•   Check story for consistency.
•   Anticipate target market needs.
•   Get instructions for the game.




                    make
                  better
                   games
Test Equipment
•   Computers and Dev. Systems.
•   Network environments.
•   Joysticks, controllers, mice.
•   Headphones, microphones.
•   Monitors, TV sets.
•   VCRs and Video Tapes.
•   Console vs. PC.


                    make
                  better
                   games
Computers and Dev. Systems
• Get systems that are high and low.
• Use older systems for bug entry.
• Set up a debugging station.




                   make
                 better
                  games
Network environments.
•   Be sure teams share hubs correctly.
•   Verify network protocols.
•   Check for unusually long cables.
•   Test on mismatched system speeds.
•   Test on dedicated modem lines.




                    make
                  better
                   games
Joysticks, controllers, mice.
•   Test the joystick ports.
•   Get support from manufacturers.
•   Be ready for hardware defects.
•   Look for marketing promotions.
    – Flight decks.
    – New technology.




                     make
                   better
                    games
Headphones, microphones.
•   Get headphones for tester sanity.
•   Test voice control with microphones.
•   Offer to assist with dialog recording.
•   Use hands-free sets for multi-player.




                     make
                   better
                    games
Monitors, TV sets.
• Bigger is better.
• Make a demo machine.
• Set up a debugging system.




                  make
                better
                 games
VCRs and Video Tapes.
•   Record a demo tape.
•   Record bugs that are hard to catch.
•   Keep a library of bug tapes.
•   Use movie tapes for research.




                     make
                   better
                    games
Console vs. PC
•   Hardware testing is more defined.
•   Manufacturer standards are issues.
•   Development kits are restricted.
•   The standards can be different.
•   Patches may not be possible.
•   Game controls are usually different.



                     make
                   better
                    games
Quality Assurance
• Quality Observation.
• Quality Assessment.




                  make
                better
                 games
QA Management
•   Create assignments by project.
•   Create assignments by person.
•   Make a QA personnel database.
•   Give HR interview questions.
•   Give tips on finding bugs.
•   Connect with Marketing.
•   Master sign off forms.


                    make
                  better
                   games
Create assignments by project
• Trek
  – Adam (Lead)
  – Bob
  – Carl
  – Darth
  – Evan
• X-Wing
  – Darth (Lead)
  – Evan
  – Luke

                     make
                   better
                    games
Create assignments by person
• Adam
  – Trek (Lead), Top Gun
• Bob
  – Trek, Falcon
• Evan
  – X-Wing, Trek, Top Gun
• Darth
  – X-Wing (Lead), Trek, Falcon


                     make
                   better
                    games
Give HR interview questions
• How is testing games different from
  playing them?

• What makes a game too hard?

• What is needed in a good bug
  description?


                  make
                better
                 games
Give HR interview questions
• The game crashed when I fired my
  weapon at the monster. How could
  this description be improved?

• How important is historical accuracy
  to a game and why?

• What skills are needed to effectively
  test a game?
                   make
                 better
                  games
Give tips on finding bugs
•   Check for collision detection.
•   Test installation program settings.
•   Fill up all text boxes with “M”
•   Check “L” and “l” and “1”.
•   Use all symbols and characters.
•   Check with screen savers.
•   Maximize numbers.


                     make
                   better
                    games
Give tips on finding bugs
• Adventure Games
  – Get rid of critical inventory items.
  – Give items in the wrong order.
  – Advance time incorrectly.
  – Go to the 13th floor.
  – Try the gray buttons.




                      make
                    better
                     games
Give tips on finding bugs
• 3D Simulators
  – Cross check technical documents.
  – Check for level of detail changes.
  – Look for sorting errors.
  – Check all boundaries and limits.
  – Load up on textures.




                     make
                   better
                    games
Connect with Marketing
•   Provide product demonstrations.
•   Conduct Focus Group tests.
•   Give bug report summaries.
•   Help with conventions.
•   Provide great screen shots.




                    make
                  better
                   games
Master sign off forms
•   Establish procedures.
•   Indicate final product status.
•   Give information to support.
•   Show compatibility report.
•   Give Marketing screen shots.
•   Provide localization information.



                     make
                   better
                    games
Make better games
Keep records of good bug discoveries.
Be aware of the available testing tools.
Get all the dialog into a text database.
Push for cheats and special options.




                   make
                 better
                  games

								
To top