Data Massage Therapy:
Pipelines, SCons and 'SOCOM: Confrontation‘ November 18th 2008
Introduction
Who Are We? Who are you?
Introduction
Who Are We? Dean Giberson: • Started out in 1999 as tool developer for 4 years working on several StarWars titles • Currently Sr. Tools & Pipeline Engineer at Slant Six Games • Specialist areas; data build management and pipelines, … • After a short period teaching and contracting he returned to professional game development working on Sports titles for 3 years • Credits:
– – – – – – – NBA Live 08 (2007) Xbox 360/PS3 NBA Live 07 (2006) Xbox 360/PS22/XBox/PC NBA Live 06 (2005) PS2/XBox/Gamecube/PC Midway Arcade Treasures 2 (2004) Gamecube Rogue Squadron 3: Rebel Strike (2003) Gamecube Rogue Squadron 2: Rogue Leader (2001) Gamecube Battle for Naboo (2000) N64
Paul Martin: • Started out in the industry 1996 graphics programmer for PlayStation • Currently a Technical Director and one of the principals of Slant Six games • Specialist areas are in graphics rendering, special effects and data build pipelines. • Credits:
– – – – – – – SOCOM: Tactical Strike (2007) PSP, Slant Six Games Syphon Filter: Dark Mirror (2006) PSP, Sony Computer Entertainment America, Inc. Syphon Filter: The Omega Strain (2004) PS2, SCEA Sled Storm (2002) PS2, Electronic Arts, Inc. SSX (2000) PS2, Electronic Arts, Inc. Need for Speed III: Hot Pursuit (1998) PS1, Electronic Arts, Inc. [bunch of unreleased stuff prior to this…]
Introduction
Who Are We?
Slant Six Games – Intro:
• Slant Six Games is one of the largest video game development studios within the Vancouver game development hub. We were established in February 2005 and specialize in developing games for the PlayStation 3 and PlayStation Portable. • We currently have 110 full time staff and occupy 20,000 square feet in our studio located at the mouth of Granville Island in Vancouver, British Columbia.
Introduction
Who Are We? Slant Six Games – Our Games: • • First project was the creation of the graphics rendering engine for Sony Computer Entertainment America’s (SCEA) Syphon Filter: Dark Mirror (developed by Sony Bend, Oregon). Inaugural title, SOCOM: Tactical Strike, published by SCEA, in October 2007. Was subsequently named the Best PSP Multiplayer Experience at E3 2007 and in February 2008 it was awarded the Elan Award for the Best Handheld Game of the Year. To date it has sold over 400,000 copies worldwide. Also in February 2008 we were awarded the prestigious Elan Award for Best New Video Game Company.
•
Just completed development of our largest and most significant project to date, SOCOM: Confrontation, for the PS3, in late summer of 2008. SOCOM: Confrontation has been in production for just under two years and was one of the more highly anticipated titles of this fall.
Currently working on
• • • • Shipping SOCOM: Confrontation in other worldwide regions Unannounced PSP title for SCEA which is scheduled for completion in June of 2009 Expansion packs and other new downloadable content for SOCOM: Confrontation on PS3 Another unannounced new concept game for PS3
•
Introduction
Who Are You?
• Programmers/Technical Artists/Bunch of other people, hopefully interested in how to build an efficient data build pipeline for current/next generation game content • In the wrong room?
Introduction
First, a movie…
SOCOM: Confrontation - Urban Wasteland
Data Build System Requirements
• • • • • Fast! Correct Extensible Debuggable Asset management integration
Our Data Builder System
Features: • • • • Dependency Tracking Shared Build Cache Tight integration with Perforce Partial Builds (build only: Textures, Collision, World Descriptors, etc)
Built on top of Scons…
SCons
What is it?
• From SCons.org:
SCons is an Open Source software construction tool—that is, a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.
Where did it come from?
• Also from SCons.org:
SCons began life as the ScCons build tool design which won the Software Carpentry SC Build competition in August 2000. That design was in turn based on the Cons software construction utility. This project has been renamed SCons to reflect that it is no longer directly connected with Software Carpentry (well, that, and to make it slightly easier to type...).
SCons
Why are we using it? …back to Data Build System Requirements • • • • • Fast! Correct Extensible Debuggable Asset management integration
SCons
Pros: • • • • • Robust Reliable Rich Scripting Interface (Python!) Shared Data Caching Multi-processor support
SCons
Cons: • Phase 1 dependency & job building can be slow • Core built on non-trivial Python code
• Can make non-standard extensions to the system tricky to implement (e.g. dependency insertion, dependency pruning)
SConstruct
What is it?
• Equivalent of a Makefile • The input file that SCons uses to control the build • A Python script. With a one main difference
• Definition order of SCons builder functions, does not dictate execution order • Execution order of non-builder functions is as expected
• Two-step processing
• Regular Python (non-SCons Builder) functions first • Then SCons builders invoked as deemed necessary by the dependency engine
SConstruct
Some usage examples: • Texture processing • SSG (wad/pak/zip/rx2/etc) processing • Tool Target Scanner (examples)
Pretty It Up – Use a GUI!
Nature abhors a vacuum
Pretty It Up – Use a GUI!
Artists and Designers abhor a command-line interface
Pretty It Up – Use a GUI!
Undesirable Desirable
Pretty It Up – Use a GUI!
SCons Data Build Demo
• Small Level Build Example
• Small Level Incremental Build Example
Scons – Live Demo
Scons Builder Demo…
Not all roses!
• Staged building (the graph problem) • Maya • Large trees have large dependency graphs
Not all roses! Staged building
• You can’t know what Maya files to build until you have scanned your level files. • You can’t know what textures to process until you have scanned your Maya files. • You don’t know what assets to place into you ZIP until you have everything else first.
Solution. Staged building
• We manually scanned the level files using Python XML • Texture build can still be a problem • ZIPs get rebuilt if there is any change • We have alpha level code for staged building for automatic LOD generation, but it was not exercised enough before we changed LOD methods
Not all roses! Maya
• We use MBs as our source file format for 3D art. • You can only have one instance of Maya running (and keep it reliable). • Maya can take 30 seconds to launch. • Bottleneck!
Solution. Maya
• Going to use gzip MAs, and write a text scanner for texture links, or write a simple MB scanner for textures • SN-DBS MayaBatch • Stripping Maya’s default launch scripts can get ~3 second launches • Still bottleneck!
Not all roses! Big tree
• Building the dependency tree for the whole game could take 1.5 minutes for all assets • MD5 sum of many files can be slow • Artists that only want to change one texture (a 2 second problem) should not have to wait on this • But it works great for build servers • Balance?
Solution. Big tree
• Internally trim the tree for some assets based on type • ResourceBuilder tells SCons the sub portion of the tree the artist is interested in and we only build that graph • New versions of SCons have the `-interactive’ flags to scan the tree once and let you run multiple commands on it • Balance
Not all roses! Other platforms
• Make \ XNA \ Ant \ Cmake \ YABS • All have similar problems • SCons’ MD5 checking gives best results so far • …and Python script, not XML
The rebuild story
SOCOM: Confrontation
SOCOM: Confrontation
Some Stats: • Source data total size: 200 Gb • Target (PS3) data total size: 3 Gb • World models for 1 world:49 • Unique prop models for 1 world: 167 • Prop instances for 1 world: 1901 • Unique textures for 1 world: 1209 • Lightmaps for 1 world: 1581 (x2 – day+night)
SOCOM: Confrontation
More Movies: • SOCOM: Confrontation – Crossroads • SOCOM: Confrontation - Fallen
What Next?
• • • • Debugger Staged building Link to SN-DBS Code?
References
• SCons: http://www.scons.org/ • Slant Six Games: http://slantsixgames.com • SOCOM: Confrontation:
http://www.us.playstation.com/PS3/Games/SOCOM_U_S_Navy_SEALs_Confrontation
Talk to Us!
Paul Martin: paul@slantsixgames.com Dean Giberson: dgiberson@slantsixgames.com
Slant Six Games 3rd Floor, 1523 West 3rd Avenue, Vancouver, BC V6J 1J8
http://www.slantsixgames.com
Bye For Now!