Building an entire game’s worth of source assets (from Maya, Photoshop, etc) into engine usable assets is a very time consuming process. It is a process ripe for automation.
During the development of PubCrawl we lacked the time and infrastructure to create a complete system for building assets, we were relegated to a manual build process. Tony would copy source assets from a network share on Denrei’s computer and build a game asset from each source asset using our command line tools. The game assets would then be copied into PubCrawl’s game file system. The entire process was horrendously error prone; it was not uncommon to spend precious time trying to figure out why something wasn’t showing up in the game only to discover that the wrong game asset had been copied into the game path.
A major factor preventing us from creating an asset build system was that our source assets were completely unmanaged. The source assets that Denrei created existed only on his computer. We tried checking source assets into our Subversion repository, a process which took nearly an hour any time they were checked in. Then we tried storing the source assets in a seperate repository from the source code. Tony tried valiantly to integrate it into our existing systems, but it was just not to be.
Then we discovered Perforce and all our problems evaporated. Now both code and assets are stored in one repository on our Linux server and the repository is easily accessible for scripting. The stage was set for an automated asset build system.
What I created was SNAX: the Stolen Notebook Automated eXporter. It is written in Python and designed to fit into our Buildbot/Perforce build system. It is also designed to be very extensible. I’ll get into how SNAX works, but first some background is in order.
Building Dependantly
Build systems generally work by explicitly defining dependencies between files and executing commands to resolve those dependencies. For example:
This graph is traversed from the ‘project1’ node down to find what dependencies are not met and what steps are necessary to resolve those dependencies. In this example if data.o were missing a command is executed to recreate it from data.c and data.h. The effect of this system is that only nodes that fail their dependency check are updated. If only one file has been modified only that file and those depending on it are processed.
However, a dependency based system requires that the dependency relationships for all files be completely defined. This is great for source code because a lot of files result in one executable. Assets, on the other hand, have a one-to-one or one-to-many relationship between game asset and source asset. Two source assets never result in one game asset.
Further complicating things, source assets can create arbitrarily named intermediate files. It is common in to define general dependency rules to simplify dependency definitions. A common rule for source code is that .o files depend on .h and .c files of the same name. This concisely defines a number of dependencies and also automatically handles new .h and .c files with out modifying the dependency definition. However, when the names of files are arbitrary it is impossible to use this rule. For example a Maya file named ‘level01.mb’ might produce materials named ‘wood.material’, ‘grass.material’, ‘rock.material’, etc. There is no simple way to define these dependencies, they must all be specified. It is as if the above dependency graph were reversed, with .c and .h files being produced from .o files and the names of files didn’t correspond. Each dependency has to be explicitly defined before any building could begin.
Creating and maintaining a dependency graph definition of source assets complicated by the explosion of game assets generated from source assets seemed like more trouble than it was worth… so I decided against dependency based building.
Forwards is Backwards
SNAX works in a forward direction as opposed to the dependency systems reverse traversal. Each node takes in a list of files, does a build operation, and outputs a list of built files which is passed to it’s child nodes, and so on.
These nodes are linked together in a graph that is traversed forward. For example, to build materials and scenes from source assets a Maya binary is built into a collada file, materials and scenes are then exported from the collada file. It looks like this:
The advantage of this is the simplicity of defining the graph. It isn’t necessary to specify what files build to what files, only what file types build to what other types. The nodes themselves handle passing around the specific files. The definition for building scenes and materials from Maya binaries is simple:
<File filename="*.mb" builder="maya_to_collada"> <File filename="*.dae" builder="collada_to_material"/> <File filename="*.dae" builder="collada_to_scene"/> </File>
There are a couple cool features here. The builder names in the above XML correspond directly to python functions making it very easy to add new builder operations. Also, building only specific assets is easily handled by passing only the source files to be considered to the root of the forward builder. Tony used this to make the build server rebuild only the modified source assets after each check in to Perforce. Full rebuilds can also be forced from the buildbot web interface.
Conclusions
Building in a forward manner simplifies build definitions and reduces user error. In a dependency graph based system it would be easy to forget to update a dependency definition because they must be defined for each game asset file. Dependency based builds err towards doing too little whereas forward building may do too much. I prefer to do too much work to ensure that the latest game assets are always built.
For now forward building is the simplest method. We don’t have a huge number of source assets for our current project so even rebuilding all game asset takes under 3 minutes. For a larger project the limitations of forward building could become more problematic. In a project with many source assets a dependency based build system would allow very explicit definitions of what should be processed and what is output. The dependency build system can filter out unused source assets by not including them in the dependency graph. This could be emulated in the forward build system by explicitly maintaining a list of source assets and passing into the forward build system. But at that point it may be easier and more robust to simply use a dependency build system.