Tony’s Blog

I proved an important theory today. Having a conversation at 150 yards can indeed be just as awkward as a conversation from 3 feet away. I often look down from my apartment building and spot people I know. I find it amusing to call them on their cell phones and notify them that I am judging them from an elevated position. Then I tell them exactly how and why I am judging them and hang up. Sometimes I keep it short and sweet. If you know me and are walking past my building it is not uncommon to receive a phone call and hear “Nice pants loser. Ya’ll just got judged!… CLICK”. If you quickly turn around and look up at my floor you might catch me ducking behind a sofa. Anyway, today one of my friends managed to pull me into an awkward conversation about network cards before I was able to judge them. The conversation was fine but it was made awkward by the fact that the person I called stopped and looked at me while having the conversation from 150 yards away. It’s very odd to make extreme gestures like you are yelling to someone far away but then be talking in a normal voice. The entire experience was terrifying. I’d like to try it again someday. Finally, incase you were wondering, I actually yell the work “CLICK!” before I hang up a phone.

One part of our next milestone consists of extending our automated build system. We already use Buildbot to automate most of our builds and testing. Every time source code is checked into the repository it is compiled and all unit tests are run. If the compile or unit tests fail the entire team is notified via email.

Our next milestone requires us to take our automated build system one step further. Source code and game source assets will be automatically built and distributed for all to use when checked into the repository. Any changes to the source code or game assets will result in all developers automatically using the updated version. If bad source code is checked in (it doesn’t compile or fails a unit test) then the build will fail, the team is notified and the broken version is not automatically deployed. There are many other cool extras we get for free when using this system. For example the ability to automatically deploy any version of the repository. If something went horribly wrong with the latest checked in version we can instantly build and deploy any prior version of the repository and all developers will automatically be using that version. Developers can deploy and work on different versions of the repository at the same time. This can also extend to branches and tags. We can have one part of our team working on an unstable branch of the engine while the other part is working on the main trunk of the repository which is stable. Each team works on their own branch and any changes they make are automatically deployed to the others using that same branch. When the unstable branch becomes stable and is merged into the main trunk then the new changes are instantly deployed to the group working with the main trunk.

This will also be very useful for game assets. The artist never has to compile or generate game assets from their source assets and then check the generated game assets into the repository. They just check in their source assets which are then automatically built into game assets and deployed to all developers. To test the new automated build system we decided to make another small game. Gavin and I are going to do most of the modeling so we can become acquainted with Maya. The game is a simple target range where you can shoot at various objects. As a result I felt compelled to name our current milestone “Hogan’s Alley”.

We finally got internet at our new place today. The release of our Pub Crawl milestone should happen shortly. We also replaced our old 100 Mbit networking hardware with new 1 Gbit switches. I was worried about the cost being too much for the extra speed. I was also worried that we wouldn’t see a big difference in our transfer rates. After hooking up the new switches my worries were quickly put to rest. The new transfer speeds over the network are wonderful and it was worth every penny. If you are deciding whether or not to switch to Gigabit Ethernet I highly recommend you do. If your computers don’t have Gigabit Ethernet cards you can pick them up pretty cheap and the money saved by using 100 Mbit hardware will not justify the slower transfer rates.

We have been hard at work on our fourth milestone titled ‘Pub Crawl’. All of our major goals were finished by our deadline on Sunday. This included creating ScriptNodes, updating snTools, and creating a small game to test them. I went into the milestone with the idea of creating a hacky throw away game system which sole purpose was to test the ScriptNodes. The system turned out better then I thought and we began to gain some insight into how our actual game system might operate. Denrei has been working endlessly on modeling the pub and its characters. His apartment filled with water during the flood and I think he has been living off of Zebra Cakes and Chipotle as it’s all I have seen him eat. Our milestone most likely won’t be publicly available for a little bit because of our big move to the new apartment. A lot of our time this week will be spent packing and unpacking. Not to leave you hanging here is a development screenshot of Pub Crawl.

Crossblast was the result of our third milestone “Lord of Collisions”. The main goal of the milestone was to finish the engine’s collision system which included the maya exporter with bounding volume nodes. The original end result was supposed to be a simple level where the user walks around and collides with objects. The objects would then glow to show the collision. After the first week or so we had a simple sphere colliding demo. You could move around a sphere and bump other spheres. The collisions were being processed and passed to the python code. We realized that a much better test of the engine and tools was to create a simple game in 24 hours. We didn’t want to spend more time on it then we had to as it was not our original goal for the milestone. After designing a simple game Denrei realized we had thought of the game once before. A while back we were creating a simple test for our last engine. It was named “Crossblast: Struggle of a Lifetime!” It might have had more exclamation points but I can’t remember. That game never came to be but it did generate some artwork which was never used. Denrei quickly found the old artwork and it was reborn into the new Crossblast.

Creating the game in 24 hours seemed like a challenge. In actuality it wasn’t very hard to accomplish due to the stable state of the engine. There were no show stopping bugs to hinder progress on the game. Python is a great language for quickly prototyping things and then implementing them. This allowed me to quickly prototype an idea for the game and if it turned out to be good the work was mostly done. The same game code written in C or C++ would have taken at least twice as long to develop. The more I use python the more it becomes apparent how much quicker and easier software can be developed with it.

After finishing the game we had a double elimination tournament. The game was displayed on a large wall with a nice projector. I didn’t even place in the top 3 proving that I am the worst Crossblast player. After the dust had settled Denrei was the victor. I should have programmed in some cheats or something to give me an advantage. I think I may just retire and become a Crossblast coach. He’s good, but with my help, he could be the best.

One really hard task was testing Crossblast on a wide range of computers and operating systems. We don’t have a ton of spare computers lying around so we couldn’t do a thorough test on different video cards and systems. This means that crossblast was only tested on high end machines. It will most likely crash on other systems for very simple reasons which I can’t reproduce without more knowledge about the computer it crashed on. For the next release Stolen Notebook will be more prepared for this. We will be setting up a testing system which PXE boots. Upon PXE Boot it will load a menu from a TFTP server which has a selection of operating systems to install. This will quickly install a fresh ghost image of the selected OS. I will most likely write a more thorough blog on how to setup this configuration.

Crossblast isn’t a finished game. It’s only a milestone test. There were some memory leaks which I found after we released the milestone. I may create another version with the fixes but for all purposes the Crossblast milestone is done and we are moving on to the next milestone titled “Pub Crawl”.

While making our collision system we kept in mind the idea of someday implementing it with a collision detection library other then our own. As a result we designed the collision system to be easily implemented with any collision library. The changes to the implementation are invisible to the rest of the engine. I wrote an implementation for the collision system which included basic intersections between oriented bounding boxes, spheres, and rays. After finishing the initial implementation we decided to use the Open Dynamics Engine. ODE is a nice open-source physics library for simulating rigid body dynamics. Under the rigid body physics system it has a very simple and easy to use collision detection system. I was saddend to see my collision implementation go but its tragic death was not in vain. It allowed us to create a nice collision system which was not tied to or influenced by any existing collision library’s API. This made implementing the collision system with ODE a very straightforward task.

This is an in progress tutorial on how to run buildbot as a Windows XP service. Most of the information comes from these three articles by Grig Gheorghiu:

continuous-integration-with-buildbot
running-buildbot-on-various-platforms
running-python-script-as-windows

At Stolen Notebook we use Buildbot for our automated testing. Buildbot is a system that automates project builds and tests. It helps our team quickly identify errors in our code before they become problems. Buildbot runs after every repository check-in and compiles the code with tests. If the build or tests fail our team receives an email indicating who checked in code last and why the build failed. This aids in catching small problems such as forgetting to add a new file to the repository. A missing file is a very simple problem which can hold up other developers in a big way. Buildbot can also immediately catch larger issues such as tests failing due to new code changes. If you are new to buildbot there is a nice article by Grig Gheorghiu on setting up buildbot at continuous-integration-with-buildbot. If you are setting up buildslaves on multiple platforms he also has a nice article at running-buildbot-on-various-platforms.

Once you get your buildbot farm going you might want to run your Windows NT/XP/2003 buildslaves as windows services. Our WindowsXP buildslave serves multiple purposes. Different users log in and out of it on a regular basis. Running buildbot from the command line doesn’t work very well in this situation. The easiest solution is to run buildbot as a Windows Service. This allows buildbot to start running when Windows begins. Buildbot then runs in the background despite which users log in or out of the machine. Some small issues may arise when turning buildbot into a service so it is best to make sure buildbot is working from the command line first.

Here is an article about running a python script as a windows service running-python-script-as-windows

I created a buildslave user and got my slave working on the command line under that user’s account. I then created a service called BuildSlave and set it to run under the buildslave user. You can do this by going to Control Panel-> Administrative Tools->Services. Right click on the Buildslave service and select Properites. Go to the “Log On” tab and under “Log on as:” select “This account:” Now type in the buildslave username (I named mine buildslave) and password. The service will now run as the buildslave user. Some environmental variables might not get set. See the Problem Solving section below for a more detailed description.

There are different methods for launching the python script. The one mentioned in the article uses a batch file.

UPDATE: A comment by jdpipe mentions a better way to launch buildbot by using a python script instead of a windows batch file. Check out his article at http://ascendwiki.cheme.cmu.edu/BuildBot.

For buildbot you would make a batch file with the contents:

C:\Python24\python C:\buildbot\bin\buildbot start c:\buildslave\game

Make sure to substitute the correct paths for python, buildbot, and the buildslave directory. I named my batch file startslave.bat.

It should be mentioned that if the batch file method is used then stopping the service might not kill the python.exe process. You will have to kill it manually.

Problem Solving:

Buildslave stays in ACTIVE state:
A problem I had was when a buildslave would do a build and then never return to the IDLE state. On the next build the buildmaster would go through the buildslaves looking for IDLE ones but the buildslave would still be in the ACTIVE state and get skipped. Since it would never return to the IDLE state it would never do another build. I believe this problem was caused by not calling a superclass’s __init__ function from a subclass in my master.cfg file. If you have a configuration where you are subclassing a buildbot class(Such as subclassing step.ShellCommand to change the name of a build step) make sure you call the superclass’s __init__ function from the subclass’s __init__ function. If you don’t the effects could be small and unnoticeable at first but cause the buildslaves or buildmaster to behave in very strange ways because certain classes were not initialized properly.

Buildslave hangs or doesn’t start a build when running as a service:
When buildbot is started as a service the windows environment may be different even if you run the service as the same user you ran buildbot as from the command line. Certain environment variables which are set and valid when you run your slave from the command line might be invalid or not even exists when the slave is run as a service. Two big variables are HOMEPATH and HOMEDRIVE. If your buildbot setup requires some programs which rely on config files in the buildslave user’s home directory they might use those environment variables to find where those files reside. If the variables don’t exist those programs might use default options which cause them to just sit there and do nothing. When buildbot receives a command it dumps the environment just before running the command. A good idea is you run the buildslave from the command line and take a look at the environment. Then run the buildslave as a service and compare the two environments. You can find all logged messages in the twistd.log file which resides in your buildslave’s configuration directory. After comparing environments I found the service environment was missing both the HOMEPATH and HOMEDRIVE variables. My ssh setup required config files in those directories and was unable to find them. I then set those variables in my startslave.bat(which is launched by the BuildSlave service) prior to launching buildbot. It now looks like this:

set HOMEDRIVE = "C:"
set HOMEPATH = "\Documents and Settings\buildslave"
C:\Python24\python C:\buildbot\bin\buildbot start c:\buildslave\game

I finished most of the engine’s collision detection for the next milestone. It’s coming along nicely and fits well into the scene graph. My rusty math skills have come back to haunt me by slowing my progress some. I have mainly been doing network programming for the last couple of years so I needed a little review. I tried to obtain the knowledge by means of voodoo but I ended up burying Bill Pullman alive in my back yard. According to Hollywood that’s pretty much how a voodoo ceremony should go down. Yet, in the end, I had obtained no additional math skills. Finally, I realized there was no quick fix. I would have to buckle down and do it the old fashioned way. I hid behind a car outside the math building at my local university. When people walked out of the building I jumped out and yelled a quick question about trigonometry at them. This usually resulted in either them or me running away. It’s a sick world we live in.