Dev Notebook

I’ve never really been satisified with any existing data languages (see my last post).  They never do all the things I want or are too general, complicated or cumbersome to use.  So, I decided to come up with my own language.  It is a middle ground between Dungeon Siege skrit and id’s decl language.  The language is basically a list of sections and keyvals (like decls) and schema definitions (like skrit) to allow type checking and data driven editors.

The grammar is pretty simple:

Section         ::= "[" <name> <name> "] "{" <SectionContents> "}"
SectionContents ::= <SectionContent> | <SectionContent> <SectionContents>
SectionContent  ::= <KeyVal> | <Section>
KeyVal          ::= <name> <Values> <EOL>
Values          ::= <Value> | <Value> <Values>
Value           ::= <string> | <name>

Put simply, each definition is a section which contains a list of keyvals or other sections.  Keyvals are allowed to have an arbitrary number of values.  The key and section names are not required to be unique.

Here’s a simple example:

[schema material]
{
  Type        string metal "material type for audio and effects"
  Translucent bool   false "if the material can be seen through"

  [keyval Texture]
  {
    Type string Diffuse "type of texture (diffuse, normal, etc)"
    File string textures/default "Texture file"
  }

  [keyval Uniform]
  {
    Name  string Uniform   "the name of the uniform"
    Value float4 "1 1 1 1" "the uniform value"
  }
}

[material test]
{
  Type        concrete
  Translucent false

  Texture Diffuse   textures/concrete_diffuse
  Texture NormalMap textures/concrete_normal
  Uniform Color     "1 1 1 1"
  Uniform Specular  "1 0 0 1"
}

In this example, the data is defined in the material definition while the schema is used to typecheck the data in the material definition.  In schema, keyvals are defined as follows:

<name> <type> <default> <docstring>  // single value keyval

[keyval <name>] // multi-value keyval
{
  <name> <type> <default> <docstring> // value 1
  <name> <type> <default> <docstring> // value 2
  <name> <type> <default> <docstring> // value 3
}

The Type and Translucent keyvals from the material definition are examples of single value keyvals while Texture and Uniform are examples of multi-value keyvals.

The information defined in schema not only allows typechecking and verification of data, but it can also be used to data drive a gui editor. The user chooses the schema of the type of object to create and the editor creates the default structure and data for the object using the schema.  The editor just displays a property sheet view of the data and can provide special controls for editing each piece of data based on it’s datatype.  It can even display the doc string as a tooltip.

The most powerful feature of this system is that entirely new objects can be created simply by writing a new schema.  Once the new schema is defined the gui editor can be used to create new objects of that type.  All without having to edit any code.

References:

Scott Bilas: “A Data-Driven Game Object System”
http://www.drizzle.com/~scottb/gdc/game-objects.ppt

For a while now I’ve been trying to find a general human readable data format for C2. I’ve created some simple languages in the past, such as NASL [PDF 69Kb], but maintaining a custom language is too time consuming for me to consider anymore. So, I’ve been shopping around for a new language to use for data storage and wanted to share some of my findings. For my examples I am using the geometry data of a textured square.

Requirements
A good data language satisfies a few requirements.

1) Has a C/C++ parsing library
2) Has a python library, making writing tools easy
3) The language is simple, without features that complicate it’s use as a data language
4) The language is easily human readable and writable

XML
First up is XML, everyone’s goto human readable data format.

<geometry>
  <vertexbuffers>
    <Position>
      (-1.0 -1.0 0.0 1.0)
      (-1.0  1.0 0.0 1.0)
      ( 1.0 -1.0 0.0 1.0)
      ( 1.0  1.0 1.0 1.0)
    </Position>
    <TexCoord0>
      (0.0 0.0)
      (0.0 1.0)
      (1.0 0.0)
      (1.0 1.0)
    </TexCoord0>
  </vertexbuffers>
  <indexbuffers>
    <mesh0 mode="triangle_strip">
      0
      1
      2
      3
    </mesh>
  </indexbuffers>
</geometry>

XML (24 lines, 429 chars)

It seems that everyone uses XML for everything, so why shouldn’t I? XML libraries exist for all programming languages. As a data language, however, I find it to be much too complicated. There are too many ways to specify data (as attributes, as text between tags, etc). The syntax also leaves something to be desired and dealing with the DOM to get at data is no treat either.

JSON
Next is JSON a data serialization language that originally comes from Javascript.

{
  "vertexbuffers": {
    "Position": [
      [-1.0, -1.0, 0.0, 1.0],
      [-1.0,  1.0, 0.0, 1.0],
      [ 1.0, -1.0, 0.0, 1.0],
      [ 1.0,  1.0, 1.0, 1.0]
    ],
    "TexCoord0": [
      [0.0, 0.0],
      [0.0, 1.0],
      [1.0, 0.0],
      [1.0, 1.0]
    ]
  },
  "indexbuffers": {
    "mesh0": {
      "mode": "triangle_strip",
      "data": [
        0,
        1,
        2,
        3
      ]
    }
  }
}

JSON (27 lines, 441 chars)

Many JSON parsing libraries exist for both C/C++ and python. The JSON python library conveniently parses directly into a python dictionary making the data easily accessible. JSON is also much more human readable than XML. While it’s definitely a step up from XML it isn’t perfect. When writing JSON files I found that the double quoted value names and commas between value statements made writing JSON files error prone. Escaping string characters is also necessary and troublesome. If I didn’t need to write data files by hand JSON would be a good solution but as is it doesn’t quite fit my needs.

DS Object Template
The next language is inspired by object templates used in Dungeon Siege created by Gas Powered Games.

[geometry]
{
  [vertexbuffers] {
    Position = [
      [-1.0, -1.0, 0.0, 1.0],
      [-1.0,  1.0, 0.0, 1.0],
      [ 1.0, -1.0, 0.0, 1.0],
      [ 1.0,  1.0, 1.0, 1.0]
    ];
    TexCoord0 = [
      [0.0, 0.0],
      [0.0, 1.0],
      [1.0, 0.0],
      [1.0, 1.0]
    ];
  }
  [indexbuffers] {
    [mesh0] {
      mode = "triangle_strip";
      data = [
        0,
        1,
        2,
        3
      ];
    }
  }
}

Object Template (28 lines, 447 chars)

This language solves many of the problems I have with JSON: value names aren’t quoted; semi-colons end every value statement; and separators aren’t required after subsections. An extra feature is that sections are separated out using brackets making it a bit easier to parse visually. The major con is that this is not a standard format. There are no libraries that will parse these data files for C/C++ or python. I created a parser using Boost Spirit to see if it would be worth developing further but it didn’t seem to be.

YAML
The last language I looked at was YAML. Billed as “…a human friendly data serialization standard for all programming languages,” it did not disappoint.

vertexbuffers:
  Position: [
    (-1.0 -1.0 0.0 1.0),
    (-1.0  1.0 0.0 1.0),
    ( 1.0 -1.0 0.0 1.0),
    ( 1.0  1.0 1.0 1.0)
  ]
  TexCoord0: [
    (0.0 0.0),
    (0.0 1.0),
    (1.0 0.0),
    (1.0 1.0)
  ]
indexbuffers:
  mesh0:
    mode: triangle_strip
    data: [
      0,
      1,
      2,
      3
    ]

YAML (22 lines, 333 chars)

YAML has libraries for both C and python. The YAML python library even parses data directly to a python dictionary just like JSON. But by far the biggest benefit of YAML is it’s syntax. Indentation is used to define blocks so all those spurrious braces are gone. String quoting is not required so escaping characters is irrelevant. I could write XML, JSON, Nvidia Cg, or just about anything inside a YAML block and not have to worry about escaping it. JSON and XML can’t touch these features. All of this makes writing a YAML file by hand a breeze making YAML the only language that satisfies all of my requirements.

Conclusion
If you’re using XML or another human readable format for data specification YAML is definitely worth a look. It’s easy to read and write, easy to parse and has just enough features to define any human readable data you need.

My most recent side project has been developing a simple community system for online games. It’s like a small subset of Xbox Live. Players signup for an account on a website then their highscores are tracked and displayed on the website for all to see. I’m keeping the features to a bare minimum and using this opportunity to try some programming styles I don’t generally use. The website uses django, has a restful api with data representation in JSON. It was originally an XML api until Tony told me about his experience with JSON, which beats XML hands down at simple data serialization. I’ve even gone back to c2 and added a JSON data loader using TinyJSON.

I’ve also been working with XNA 2.0 and C#. I like C# a lot, it fixes many issues I have with C++. One that has always bothered me in C++ is that enumerations don’t create a new namespace. This causes all kinds of name collision problems if you don’t add a prefix to each enumerant. In C# enumerations do create a namespace so you don’t have to add stupid prefixes at all. A lot of fixes like this make me feel good about C# being a better C++. The C# standard library is also great and includes tons of useful functionality that you’d have to scrounge around for in C++.

In the near future I’m planning to combine these two projects into a simple game written in C# with an online community component. Maybe some other changes to this site are in order as well…

Finally, here’s a shot of my new workspace setup only because it’s ridiculous.

An in-engine console is an indispensable asset when developing games. It can display a log of engine messages, allow simple querying of engine values and even provide a full programming environment for the engine. I had initially thought of implementing a console like Quake: toggled with ~ and rendered in-game at the top of the screen. I quickly decided against this route and chose to create a console using wxPython.

A wxPython console can provide a native OS interface with scrollbars, selectable text, copy and paste, etc all for free. These were features I’d never have time to implement in a Quake style console. Besides, I wanted to integrate wxPython into c2 anyway so I would have a way to create simple tools for the engine.

The console I ended up with is a simple dialog box that provides an interactive python prompt. It doesn’t have to interact with the engine directly, it just runs commands in a python environment which happens to have the engine interface imported. So the console can be used in any wxPython application. You can download it here. The only requirement is that you have wxPython.

If you’re interested in using the Xbox 360 controller there are two things you need to know: XInputGetState and XInputSetState. There are a few more functions if you want to interface with the headset attached to the controller, but for input that’s it. Once I understood these two simple functions I had Xbox 360 controller support in C2 in 10 minutes (literally). DirectInput in comparison is a nightmare, acquiring devices, checking if the device has been lost, etc. Why bother?

Reference:
http://msdn2.microsoft.com/en-us/library/bb173048.aspx