I got back into OpenGL this past week and spent some time getting caught up on all the new features of OpenGL3 and the sweet bindless graphics extensions nVidia released in their 185 drivers.  There’s a lot to like, but also some minor aggravations.  

As a test I wanted to get a simple tonemapping function working with framebuffer objects and floating point rendertargets.  The geometry I was rendering had color values in the range [0, 4] and the tonemapping shader would map [1, 3] -> [0, 1] for display.  Simple enough, except the result was entirely black.  After searching around a bit I took a peek at the OpenGL color_buffer_float extension spec and found that I have disable [0,1] clamping using glClampColor.  

Now everything is nice and pretty.

I happened upon two interesting data langauges today:

• OGDL (http://ogdl.sourceforge.net/)
• Harpoon Data Language (http://harpoon.sourceforge.net/)

OGDL is a very simple indentation based language, it’s like UDL without the braces.  I really really (really!) like the simplicity of this language.  There is very little syntax, none of which is really required (not counting quotes).

Harpoon, on the other hand, is like a heavily evolved UDL.  It has extra syntax that allows for a much more descriptive language.  It also has an experimental type system that could be used just like schema.  It even has the idea of key expansion.

I’m thinking about experimenting with certain features of both for UDL…

The holidays gave me some time to make some significant updates to UDL. The core language hasn’t changed (in fact, I haven’t touched it since I first wrote it two months ago!) but I did make a number of fixes and improvements to the schema system that make the language much more elegant.

UDL-DOM
The UDL event-emitting parser is fine as a low-level interface to the UDL core language, but using it in practice is tedious and cumbersome. I found that in order to do a lot of useful analysis I was manually building C++ structures from the event-emitting parser in each of my tools. So I created the UDL-DOM, a standard structure that any UDL object can be parsed into. Tools use the UDL-DOM to processes objects (analyze them, compile them to binary, etc) without having to interact with the low-level event-emitting parser. This has drastically reduced the amount of code in my tools and made what is left much cleaner and easier to understand.  

Another benefit of the UDL-DOM is that objects can be analyzed without having to know the specifics of their type. The schema checker can type check each object and even find and validate references to other objects. High-level operations even become possible like: What objects are referenced by this object? What Shader objects are used by Materials that start with “high_”? What are the all the values of a given key in a given type of object? etc. This will come in handy for a future content compiler I have had on the back burner for a while.

Key Expansion
Previously a multi-value key could be defined which had values with heterogeneous types:

[schema Object]
{
  [key MutliValueKey]
  {
    FloatKey  float  required "float key"
    StringKey string required "string key"
  }
}

[Object Test]
{
  MultiValueKey 1.0 "some value"
}

The implications of multi-valued keys led the internals of the checker to be very complicated. Worse was trying to fit complex keys (with value range limits and enumerations) into a key definitions.

I eventually realized that multi-valued keys are really just sections of single valued keys. For instance, the following two representations contain the same data:

[MultiVal]
{
  FloatKey  1.0
  StringKey "some value"
}

MultiVal 1.0 "some value"

All keys now have exactly one value (though that value may be a list), multi-value keys no longer exist. However, the syntax remains as a way to conveniently specify data for an entire section. This is called key expansion. A section’s data can be specified in order on a single line as a key.

[schema Object]
{
  [section MultiVal]
  {
    FloatKey  float  required "float key"
    StringKey string required "string key"
  }
}

[Object Test]
{
  MultiVal 1.0 "some value"
}

During schema checking, this ‘key’ will get expanded into a section using the schema, resulting in:

[Object Test]
{
  [MultiVal]
  {
    FloatKey  1.0
    StringKey "some value"
  }
}

Key expansion simplifies schema by removing the special definitions for multi-valued keys. Unlike multi-valued keys, key expansion does not interfere with other features like complex keys.

Complex Keys
The type system I started in my last post was doomed due to the immaturity of UDL at the time. The event-emitting nature of the parser made it very difficult to cleanly add type options, such as value ranges and possible values, to key schema. Originally, I had decided that I wanted to be able to define types like schema (IE: as an entirely separate object). This would complicate the language too much, so I chose to make them part of the schema again. I also considered introducing special syntax so type ranges and enumerations could be specified as a single value. Neither of these ideas used any of the features inherent to the structured nature of UDL, instead they attempted to create new syntax, complicating the language.

With UDL-DOM and key expansion it became obvious how to fit ideas like range limited values and enumeration neatly into schema definitions in a way that is very understandable. I term any key that defines a value range or enumeration list a complex key. A complex key must be defined in a schema section because it can take a number of different parameters. The definition of a complex key in schema looks like this:

[schema Object]
{
  [key Range]
  {
    type float
    min -1
    max  1
  }

  [key Enum]
  {
    type string
    options cat hat bat
  }
}

In this definition the key Range can have a float value between -1 and 1, while the key Enum can have a string value of cat, hat or bat. Any other values will produce a checking error. A valid data definition is:

[Object Test]
{
  Range 0.2
  Enum  bat
}

The most important thing about complex keys is that it is completely defined using UDL structure and has no special syntax, a major benefit over the previous type system.

UDL has gathered together a great new set of features:  

• UDL-DOM provides a higher-level view of objects allowing both high-level object analysis and simplified tool creation.
• Key expansion removes the complexity of multi-valued keys while maintaining it’s concise syntax.
• Complex keys add value range and enumerant type checking in a way that integrates into UDL and doesn’t require new syntax.

Hopefully, in the near future I can clean this up and provide a simple UDL toolset.

So I went in and added the most basic data-driven type system and all was good…until I realized there is no elegant way to define value limits. For instance if you wanted the valid value of a key to be a floating point between 0 and 1, you’re out of luck. It can only say it’s a float (by matching a regular expression). It doesn’t even know it’s a numeric value.

So I have to add numeric types to the type definitions, at least. But, only allowing limits in type definitions would be cumbersome. Imagine having to use types like float_0_1. Actually, that’s not completely terrible, but I would desperately enjoy a shorthand range syntax. Then I don’t have to go around creating tonnes of type definitions with slightly different value limits.

Type definitions still have their uses, though. Like, I don’t know, if you wanted to make sure all your integer values could fit into 16-bits, you could define a custom type called ‘myint’ (or maybe I’ll allow the built-in type ‘int’ to be redefined!) and give it the appropriate limits (−32768,32767) and use it throughout your data definitions. Then one day you decide 32-bits is where it’s at, and you can just change the limits on ‘myint’ and you’re done!

So, there will be built-in primitive types after all and shorthand value range syntax. Not so bad I guess. I suppose the types will end up being something like: string, int, float, float2, float3, float4 with some kind of range syntax. Then I can go around saying things like: float(0,1). Hmm, I’m still undecided…

After finishing up the core UDL parser I found it was extremely simple to use the core language to define schema for type checking. Right now UDL can parse a config and roughly check it against a schema (it only checks that names are correct and keys have the correct number of values). It’s functional but not very robust. No error messages, etc.

While I was finishing up the schema stuff I started thinking about how I wanted to handle the actual type checking of values. I originally planned to have primitive types basically hard coded into UDL itself, so you could say a given value was an int, float or whatever. This isn’t very flexible since you have to write code if you want to add a new type. My plan now is to provide a very simple API for adding new type checkers. Once that’s in it should be a simple matter to drive it by some simple UDL definitions! Allowing new types without new code will be really great! I hope it works out.

There were some features missing from the original UDL core language that I wanted to address. Some of the new features include keys with no values and quoted names for key and section names. However, the most important new feature is an new syntax for specifying key values over multiple lines.

Previously, if you wanted a key with 100 values you had to put them all on one line. This is both inconvenient to write out and hard to read. The simple and elegant solution is the addition of a multi-line key syntax. In this example both definitions of the key ‘data’ are equivalent.

[example "some name"]
{
  UseSystemMemory
  "Key String" "some string"
  data 1 2 3 4 5 6 7 8 9 0 11 12 13 14 15
  data
  {
     1  2  3  4  5
     6  7  8  9 10
    11 12 13 14 15
  }
}

Originally, I had been considering introducing a new type beyond keys and sections for specifying large amounts of data. This would have required a completely new data type, complicating the language and resulting API. Multi-line key syntax only requires a change to the parser. In addition any key can be specified on multiple lines using this syntax if it’s convenient for the user.

With these updates the grammar of UDL is now:

file        : sections
            | empty

sections    : section
            | section sections

section     : "[" value value "]" NEWLINE "{" statements "}" NEWLINE

statements  : statement
            | statement statements

statement   : section
            | keys

key         : value NEWLINE
            | value values NEWLINE
            | value NEWLINE "{" value_lines "}" NEWLINE

values      : value
            | value values

value_lines : values NEWLINE
            | values NEWLINE value_lines

value       : NAME
            | STRING

I finished the last few features of the UDL core language: comments and semicolons. Comments affect the lexer while semicolons affect the parser.

Comments are simple C-style single and multi-line comments. The lexer just skips right over them and they don’t make it to the parser.

Semicolons are used to break up keys on a single line allowing multiple keys per line. It also makes it convenient to specify an entire section on one line if necessary.

/*
Example object (
*/
[object example]
{
  // multiple keys on a line
  Texture textures/default.tga; Size "512 512"

  // single line section
  [Uniforms] { Color "1 1 0 1"; Offset "0.5 0.5" }
}

While neither of these features is particularly earth shattering they do fill out the usefulness of the language. Now it’s time to focus on converting my old python UDL parser into a simple UDL C Api.

I’ve never really been satisified with any existing data languages (see my last post).  They never do all the things I want or are too general, complicated or cumbersome to use.  So, I decided to come up with my own language.  It is a middle ground between Dungeon Siege skrit and id’s decl language.  The language is basically a list of sections and keyvals (like decls) and schema definitions (like skrit) to allow type checking and data driven editors.

The grammar is pretty simple:

Section         ::= "[" <name> <name> "] "{" <SectionContents> "}"
SectionContents ::= <SectionContent> | <SectionContent> <SectionContents>
SectionContent  ::= <KeyVal> | <Section>
KeyVal          ::= <name> <Values> <EOL>
Values          ::= <Value> | <Value> <Values>
Value           ::= <string> | <name>

Put simply, each definition is a section which contains a list of keyvals or other sections.  Keyvals are allowed to have an arbitrary number of values.  The key and section names are not required to be unique.

Here’s a simple example:

[schema material]
{
  Type        string metal "material type for audio and effects"
  Translucent bool   false "if the material can be seen through"

  [keyval Texture]
  {
    Type string Diffuse "type of texture (diffuse, normal, etc)"
    File string textures/default "Texture file"
  }

  [keyval Uniform]
  {
    Name  string Uniform   "the name of the uniform"
    Value float4 "1 1 1 1" "the uniform value"
  }
}

[material test]
{
  Type        concrete
  Translucent false

  Texture Diffuse   textures/concrete_diffuse
  Texture NormalMap textures/concrete_normal
  Uniform Color     "1 1 1 1"
  Uniform Specular  "1 0 0 1"
}

In this example, the data is defined in the material definition while the schema is used to typecheck the data in the material definition.  In schema, keyvals are defined as follows:

<name> <type> <default> <docstring>  // single value keyval

[keyval <name>] // multi-value keyval
{
  <name> <type> <default> <docstring> // value 1
  <name> <type> <default> <docstring> // value 2
  <name> <type> <default> <docstring> // value 3
}

The Type and Translucent keyvals from the material definition are examples of single value keyvals while Texture and Uniform are examples of multi-value keyvals.

The information defined in schema not only allows typechecking and verification of data, but it can also be used to data drive a gui editor. The user chooses the schema of the type of object to create and the editor creates the default structure and data for the object using the schema.  The editor just displays a property sheet view of the data and can provide special controls for editing each piece of data based on it’s datatype.  It can even display the doc string as a tooltip.

The most powerful feature of this system is that entirely new objects can be created simply by writing a new schema.  Once the new schema is defined the gui editor can be used to create new objects of that type.  All without having to edit any code.

References:

Scott Bilas: “A Data-Driven Game Object System”
http://www.drizzle.com/~scottb/gdc/game-objects.ppt

For a while now I’ve been trying to find a general human readable data format for C2. I’ve created some simple languages in the past, such as NASL [PDF 69Kb], but maintaining a custom language is too time consuming for me to consider anymore. So, I’ve been shopping around for a new language to use for data storage and wanted to share some of my findings. For my examples I am using the geometry data of a textured square.

Requirements
A good data language satisfies a few requirements.

1) Has a C/C++ parsing library
2) Has a python library, making writing tools easy
3) The language is simple, without features that complicate it’s use as a data language
4) The language is easily human readable and writable

XML
First up is XML, everyone’s goto human readable data format.

<geometry>
  <vertexbuffers>
    <Position>
      (-1.0 -1.0 0.0 1.0)
      (-1.0  1.0 0.0 1.0)
      ( 1.0 -1.0 0.0 1.0)
      ( 1.0  1.0 1.0 1.0)
    </Position>
    <TexCoord0>
      (0.0 0.0)
      (0.0 1.0)
      (1.0 0.0)
      (1.0 1.0)
    </TexCoord0>
  </vertexbuffers>
  <indexbuffers>
    <mesh0 mode="triangle_strip">
      0
      1
      2
      3
    </mesh>
  </indexbuffers>
</geometry>

XML (24 lines, 429 chars)

It seems that everyone uses XML for everything, so why shouldn’t I? XML libraries exist for all programming languages. As a data language, however, I find it to be much too complicated. There are too many ways to specify data (as attributes, as text between tags, etc). The syntax also leaves something to be desired and dealing with the DOM to get at data is no treat either.

JSON
Next is JSON a data serialization language that originally comes from Javascript.

{
  "vertexbuffers": {
    "Position": [
      [-1.0, -1.0, 0.0, 1.0],
      [-1.0,  1.0, 0.0, 1.0],
      [ 1.0, -1.0, 0.0, 1.0],
      [ 1.0,  1.0, 1.0, 1.0]
    ],
    "TexCoord0": [
      [0.0, 0.0],
      [0.0, 1.0],
      [1.0, 0.0],
      [1.0, 1.0]
    ]
  },
  "indexbuffers": {
    "mesh0": {
      "mode": "triangle_strip",
      "data": [
        0,
        1,
        2,
        3
      ]
    }
  }
}

JSON (27 lines, 441 chars)

Many JSON parsing libraries exist for both C/C++ and python. The JSON python library conveniently parses directly into a python dictionary making the data easily accessible. JSON is also much more human readable than XML. While it’s definitely a step up from XML it isn’t perfect. When writing JSON files I found that the double quoted value names and commas between value statements made writing JSON files error prone. Escaping string characters is also necessary and troublesome. If I didn’t need to write data files by hand JSON would be a good solution but as is it doesn’t quite fit my needs.

DS Object Template
The next language is inspired by object templates used in Dungeon Siege created by Gas Powered Games.

[geometry]
{
  [vertexbuffers] {
    Position = [
      [-1.0, -1.0, 0.0, 1.0],
      [-1.0,  1.0, 0.0, 1.0],
      [ 1.0, -1.0, 0.0, 1.0],
      [ 1.0,  1.0, 1.0, 1.0]
    ];
    TexCoord0 = [
      [0.0, 0.0],
      [0.0, 1.0],
      [1.0, 0.0],
      [1.0, 1.0]
    ];
  }
  [indexbuffers] {
    [mesh0] {
      mode = "triangle_strip";
      data = [
        0,
        1,
        2,
        3
      ];
    }
  }
}

Object Template (28 lines, 447 chars)

This language solves many of the problems I have with JSON: value names aren’t quoted; semi-colons end every value statement; and separators aren’t required after subsections. An extra feature is that sections are separated out using brackets making it a bit easier to parse visually. The major con is that this is not a standard format. There are no libraries that will parse these data files for C/C++ or python. I created a parser using Boost Spirit to see if it would be worth developing further but it didn’t seem to be.

YAML
The last language I looked at was YAML. Billed as “…a human friendly data serialization standard for all programming languages,” it did not disappoint.

vertexbuffers:
  Position: [
    (-1.0 -1.0 0.0 1.0),
    (-1.0  1.0 0.0 1.0),
    ( 1.0 -1.0 0.0 1.0),
    ( 1.0  1.0 1.0 1.0)
  ]
  TexCoord0: [
    (0.0 0.0),
    (0.0 1.0),
    (1.0 0.0),
    (1.0 1.0)
  ]
indexbuffers:
  mesh0:
    mode: triangle_strip
    data: [
      0,
      1,
      2,
      3
    ]

YAML (22 lines, 333 chars)

YAML has libraries for both C and python. The YAML python library even parses data directly to a python dictionary just like JSON. But by far the biggest benefit of YAML is it’s syntax. Indentation is used to define blocks so all those spurrious braces are gone. String quoting is not required so escaping characters is irrelevant. I could write XML, JSON, Nvidia Cg, or just about anything inside a YAML block and not have to worry about escaping it. JSON and XML can’t touch these features. All of this makes writing a YAML file by hand a breeze making YAML the only language that satisfies all of my requirements.

Conclusion
If you’re using XML or another human readable format for data specification YAML is definitely worth a look. It’s easy to read and write, easy to parse and has just enough features to define any human readable data you need.

My most recent side project has been developing a simple community system for online games. It’s like a small subset of Xbox Live. Players signup for an account on a website then their highscores are tracked and displayed on the website for all to see. I’m keeping the features to a bare minimum and using this opportunity to try some programming styles I don’t generally use. The website uses django, has a restful api with data representation in JSON. It was originally an XML api until Tony told me about his experience with JSON, which beats XML hands down at simple data serialization. I’ve even gone back to c2 and added a JSON data loader using TinyJSON.

I’ve also been working with XNA 2.0 and C#. I like C# a lot, it fixes many issues I have with C++. One that has always bothered me in C++ is that enumerations don’t create a new namespace. This causes all kinds of name collision problems if you don’t add a prefix to each enumerant. In C# enumerations do create a namespace so you don’t have to add stupid prefixes at all. A lot of fixes like this make me feel good about C# being a better C++. The C# standard library is also great and includes tons of useful functionality that you’d have to scrounge around for in C++.

In the near future I’m planning to combine these two projects into a simple game written in C# with an online community component. Maybe some other changes to this site are in order as well…

Finally, here’s a shot of my new workspace setup only because it’s ridiculous.