Dev Notebook

For a while now I’ve been trying to find a general human readable data format for C2. I’ve created some simple languages in the past, such as NASL [PDF 69Kb], but maintaining a custom language is too time consuming for me to consider anymore. So, I’ve been shopping around for a new language to use for data storage and wanted to share some of my findings. For my examples I am using the geometry data of a textured square.

Requirements
A good data language satisfies a few requirements.

1) Has a C/C++ parsing library
2) Has a python library, making writing tools easy
3) The language is simple, without features that complicate it’s use as a data language
4) The language is easily human readable and writable

XML
First up is XML, everyone’s goto human readable data format.

<geometry>
  <vertexbuffers>
    <Position>
      (-1.0 -1.0 0.0 1.0)
      (-1.0  1.0 0.0 1.0)
      ( 1.0 -1.0 0.0 1.0)
      ( 1.0  1.0 1.0 1.0)
    </Position>
    <TexCoord0>
      (0.0 0.0)
      (0.0 1.0)
      (1.0 0.0)
      (1.0 1.0)
    </TexCoord0>
  </vertexbuffers>
  <indexbuffers>
    <mesh0 mode="triangle_strip">
      0
      1
      2
      3
    </mesh>
  </indexbuffers>
</geometry>

XML (24 lines, 429 chars)

It seems that everyone uses XML for everything, so why shouldn’t I? XML libraries exist for all programming languages. As a data language, however, I find it to be much too complicated. There are too many ways to specify data (as attributes, as text between tags, etc). The syntax also leaves something to be desired and dealing with the DOM to get at data is no treat either.

JSON
Next is JSON a data serialization language that originally comes from Javascript.

{
  "vertexbuffers": {
    "Position": [
      [-1.0, -1.0, 0.0, 1.0],
      [-1.0,  1.0, 0.0, 1.0],
      [ 1.0, -1.0, 0.0, 1.0],
      [ 1.0,  1.0, 1.0, 1.0]
    ],
    "TexCoord0": [
      [0.0, 0.0],
      [0.0, 1.0],
      [1.0, 0.0],
      [1.0, 1.0]
    ]
  },
  "indexbuffers": {
    "mesh0": {
      "mode": "triangle_strip",
      "data": [
        0,
        1,
        2,
        3
      ]
    }
  }
}

JSON (27 lines, 441 chars)

Many JSON parsing libraries exist for both C/C++ and python. The JSON python library conveniently parses directly into a python dictionary making the data easily accessible. JSON is also much more human readable than XML. While it’s definitely a step up from XML it isn’t perfect. When writing JSON files I found that the double quoted value names and commas between value statements made writing JSON files error prone. Escaping string characters is also necessary and troublesome. If I didn’t need to write data files by hand JSON would be a good solution but as is it doesn’t quite fit my needs.

DS Object Template
The next language is inspired by object templates used in Dungeon Siege created by Gas Powered Games.

[geometry]
{
  [vertexbuffers] {
    Position = [
      [-1.0, -1.0, 0.0, 1.0],
      [-1.0,  1.0, 0.0, 1.0],
      [ 1.0, -1.0, 0.0, 1.0],
      [ 1.0,  1.0, 1.0, 1.0]
    ];
    TexCoord0 = [
      [0.0, 0.0],
      [0.0, 1.0],
      [1.0, 0.0],
      [1.0, 1.0]
    ];
  }
  [indexbuffers] {
    [mesh0] {
      mode = "triangle_strip";
      data = [
        0,
        1,
        2,
        3
      ];
    }
  }
}

Object Template (28 lines, 447 chars)

This language solves many of the problems I have with JSON: value names aren’t quoted; semi-colons end every value statement; and separators aren’t required after subsections. An extra feature is that sections are separated out using brackets making it a bit easier to parse visually. The major con is that this is not a standard format. There are no libraries that will parse these data files for C/C++ or python. I created a parser using Boost Spirit to see if it would be worth developing further but it didn’t seem to be.

YAML
The last language I looked at was YAML. Billed as “…a human friendly data serialization standard for all programming languages,” it did not disappoint.

vertexbuffers:
  Position: [
    (-1.0 -1.0 0.0 1.0),
    (-1.0  1.0 0.0 1.0),
    ( 1.0 -1.0 0.0 1.0),
    ( 1.0  1.0 1.0 1.0)
  ]
  TexCoord0: [
    (0.0 0.0),
    (0.0 1.0),
    (1.0 0.0),
    (1.0 1.0)
  ]
indexbuffers:
  mesh0:
    mode: triangle_strip
    data: [
      0,
      1,
      2,
      3
    ]

YAML (22 lines, 333 chars)

YAML has libraries for both C and python. The YAML python library even parses data directly to a python dictionary just like JSON. But by far the biggest benefit of YAML is it’s syntax. Indentation is used to define blocks so all those spurrious braces are gone. String quoting is not required so escaping characters is irrelevant. I could write XML, JSON, Nvidia Cg, or just about anything inside a YAML block and not have to worry about escaping it. JSON and XML can’t touch these features. All of this makes writing a YAML file by hand a breeze making YAML the only language that satisfies all of my requirements.

Conclusion
If you’re using XML or another human readable format for data specification YAML is definitely worth a look. It’s easy to read and write, easy to parse and has just enough features to define any human readable data you need.

My most recent side project has been developing a simple community system for online games. It’s like a small subset of Xbox Live. Players signup for an account on a website then their highscores are tracked and displayed on the website for all to see. I’m keeping the features to a bare minimum and using this opportunity to try some programming styles I don’t generally use. The website uses django, has a restful api with data representation in JSON. It was originally an XML api until Tony told me about his experience with JSON, which beats XML hands down at simple data serialization. I’ve even gone back to c2 and added a JSON data loader using TinyJSON.

I’ve also been working with XNA 2.0 and C#. I like C# a lot, it fixes many issues I have with C++. One that has always bothered me in C++ is that enumerations don’t create a new namespace. This causes all kinds of name collision problems if you don’t add a prefix to each enumerant. In C# enumerations do create a namespace so you don’t have to add stupid prefixes at all. A lot of fixes like this make me feel good about C# being a better C++. The C# standard library is also great and includes tons of useful functionality that you’d have to scrounge around for in C++.

In the near future I’m planning to combine these two projects into a simple game written in C# with an online community component. Maybe some other changes to this site are in order as well…

Finally, here’s a shot of my new workspace setup only because it’s ridiculous.

An in-engine console is an indispensable asset when developing games. It can display a log of engine messages, allow simple querying of engine values and even provide a full programming environment for the engine. I had initially thought of implementing a console like Quake: toggled with ~ and rendered in-game at the top of the screen. I quickly decided against this route and chose to create a console using wxPython.

A wxPython console can provide a native OS interface with scrollbars, selectable text, copy and paste, etc all for free. These were features I’d never have time to implement in a Quake style console. Besides, I wanted to integrate wxPython into c2 anyway so I would have a way to create simple tools for the engine.

The console I ended up with is a simple dialog box that provides an interactive python prompt. It doesn’t have to interact with the engine directly, it just runs commands in a python environment which happens to have the engine interface imported. So the console can be used in any wxPython application. You can download it here. The only requirement is that you have wxPython.

If you’re interested in using the Xbox 360 controller there are two things you need to know: XInputGetState and XInputSetState. There are a few more functions if you want to interface with the headset attached to the controller, but for input that’s it. Once I understood these two simple functions I had Xbox 360 controller support in C2 in 10 minutes (literally). DirectInput in comparison is a nightmare, acquiring devices, checking if the device has been lost, etc. Why bother?

Reference:
http://msdn2.microsoft.com/en-us/library/bb173048.aspx

My goal for this weekend was to get a collision and physics library into C2.

We used ODE for PubCrawl and it didn’t leave me with much confidence. This time I chose to use the Bullet Physics Library because I’ve heard good things about it. Bullet is geared more towards game engines and is much simpler than ODE. I was able to get a quick test program up using Bullet in about a day. After another day I had integrated Bullet physics into the engine itself very nicely.

In C2 physics can be added to any SceneNodes by creating and attaching a PhysicsObject. A PhysicsObject does two things:

1) provides a description of an object for simulating physics
2) updates the transformation of the SceneNode after simulation each frame

For example a PhysicsObject can be created with a mass of 10 and a sphere collision shape of radius 2. The simulation is run: objects are moved, collisions are found and resolved. Then the PhysicsObject copies it’s simulated transformation to the SceneNode’s transformation. Then the scene is rendered.

The PhysicsObject class is actually a very simple wrapper around Bullet rigid bodies. It is mainly meant to keep the engine from using Bullet directly. This hides library and header dependencies. Once compiled into a library only the C2 engine headers are necessary to use the engine. If this were not the case using the engine would be a nightmare of dependencies.

I was able to add physics information to the scene file format as well. Any SceneNode with physics definitions has a PhysicsObject created and attached to itself on load. It’s very gratifying to write a quick scene file and load it up and watch objects fall and collide. It is surprising that good physics can be had at such a low investment of time. I highly recommend the Bullet Physics Library to anyone interested in adding physics to their game engine.

The next step in the physics system is to add constraints. Constraints would allow large static objects made up of multiple PhysicsObjects that act as one. Dynamic constraints would allow objects to move relative to each other in interesting ways. There also needs to be a way to fire scripts events on collision.

Portal was awesome and the ending is fantastic.
“The cake is a lie…”