FlatBuffers Explained: Schema

FlatBuffers , Protocol
,

I. What Is FlatBuffers?

FlatBuffers is an open-source serialization library that implements a serialization format similar to Protocol Buffers, Thrift, Apache Avro, SBE, and Cap’n Proto. It was primarily written by Wouter van Oortmerssen and open-sourced by Google. Oortmerssen originally developed FlatBuffers for Android games and performance-sensitive applications. Today, it has ports for C++, C#, C, Go, Java, PHP, Python, and JavaScript.

A FlatBuffer is a binary buffer that uses offsets to organize nested objects (structs, tables, vectors, etc.), allowing data to be accessed in place, much like any pointer-based data structure. However, unlike most in-memory data structures, FlatBuffers uses strict alignment rules and byte ordering to ensure that the buffer is cross-platform. In addition, for table objects, FlatBuffers provides forward/backward compatibility and optional fields to support the evolution of most formats.

The primary goal of FlatBuffers is to avoid deserialization. It achieves this by defining a binary data protocol: a well-defined way to convert data into binary form. The binary structure created by this protocol can be sent over the wire and read without any further processing. By comparison, when transmitting JSON, we need to convert data into strings, send it over the wire, parse the strings, and convert them into local objects. FlatBuffers does not require these operations. You load data into binary form, send the same binary data, and read directly from the binary data.

Although FlatBuffers has its own interface definition language for defining the data to serialize, it also supports the .proto format used by Protocol Buffers.

Object types are defined in a schema, and can then be compiled into various mainstream languages such as C++ or Java to enable zero-overhead reads and writes. FlatBuffers also supports dynamically parsing JSON data into a buffer.

Beyond parsing efficiency, the binary format brings another advantage: the binary representation of data is usually more efficient. For example, we can use a 4-byte UInt instead of 10 characters to store a 10-digit integer.

II. Why Was FlatBuffers Invented?

JSON is a language-independent data format, but parsing the data and converting it into objects such as Java objects consumes both time and memory resources. Parsing a 20 KB JSON stream on the client takes roughly 35 ms, while a single UI refresh budget is 16.6 ms. In highly real-time games, any stutter or latency is unacceptable, so a new data format is needed. On the server side, parsing JSON can sometimes create a very large number of small objects. For servers that need to process JSON data from millions of players per second, this increases pressure. If each JSON parse produces many small objects, the massive number of small objects generated by a massive number of players may cause GC-related issues during memory reclamation. To solve performance problems in games, Google employee Wouter van Oortmerssen developed FlatBuffers. (Note: Protocol buffers was created by Google, while FlatBuffers was created at Google.)

A few years ago, Facebook announced that its Android app had achieved a major performance improvement in data processing. Across almost the entire app, they abandoned JSON and replaced it with FlatBuffers.

Like FlatBuffers (9490 stars), Cap’n Proto (5527 stars), and simple-binary-encoding (1351 stars), it supports “zero-copy” deserialization. During serialization, no temporary objects are created and no extra memory allocations are made. Accessing serialized data also does not require first copying it into a separate region of memory. This makes accessing data in these formats much faster than in formats that require such processing, such as JSON, CSV, and protobuf.

FlatBuffers is indeed quite similar to Protocol Buffers. The main difference is that FlatBuffers does not need to be parsed/unpacked before data can be accessed. The amount of code for the two is also in the same order of magnitude. However, Protocol Buffers has neither optional text import/export functionality nor the union language feature, both of which FlatBuffers provides.

FlatBuffers focuses on mobile hardware (where memory capacity and memory bandwidth are more constrained than on desktop hardware), as well as applications with the highest performance requirements: games.

III. FlatBuffers Adoption

After all that, readers may wonder: how widely is FlatBuffers used? Google’s official page mentions three well-known apps and one framework that use it.

BobbleApp, India’s leading sticker app. After BobbleApp adopted FlatBuffers, the app’s performance improved significantly.

Facebook uses FlatBuffers in its Android app for client-server communication. They wrote an article, “Improving Facebook’s performance on Android with FlatBuffers”, describing how FlatBuffers accelerates content loading.

Google’s Fun Propulsion Labs uses FlatBuffers extensively across all of its libraries and games.

Cocos2d-X, the leading open-source mobile game engine, uses FlatBuffers to serialize all game data.

As you can see, FlatBuffers is widely used in game-oriented apps.

IV. Defining a .fbs Schema File

Writing a schema file allows you to define the data structures you want to serialize. Fields can have scalar types (integers/floats of all sizes), or they can be strings, arrays of any type, references to another object, or a set of possible objects (Union). Fields can be optional or have default values, so they do not need to be present in every object instance.

For example:

// example IDL file

namespace MyGame;

attribute "priority";

enum Color : byte { Red = 1, Green, Blue }

union Any { Monster, Weapon, Pickup }

struct Vec3 {
  x:float;
  y:float;
  z:float;
}

table Monster {
  pos:Vec3;
  mana:short = 150;
  hp:short = 100;
  name:string;
  friendly:bool = false (deprecated, priority: 1);
  inventory:[ubyte];
  color:Color = Blue;
  test:Any;
}

root_type Monster;

The above is the syntax of the schema language. Schema is also known as IDL (Interface Definition Language). Its code looks very similar to C-family languages.

In a FlatBuffers schema file, there are two very important concepts: struct and table.

1. Table

A Table is the primary way to define objects in FlatBuffers. It consists of a name (Monster here) and a list of fields. Each field has a name, a type, and an optional default value (if omitted, it defaults to 0 / NULL).

Every field in a Table is optional: it does not have to appear in the wire representation, and each field of an individual object can be omitted independently. As a result, you can add fields flexibly without worrying about data bloat. This design is also FlatBuffers’ forward- and backward-compatibility mechanism.

Suppose the current schema is as follows:

table { a:int; b:int; }

Now suppose you want to modify this schema.

There are a few things to keep in mind:

Adding fields

New fields can only be added at the end of a table definition. Old data will still be read correctly, and default values will be provided for you when reading it. Old code will simply ignore the new fields. If you want the flexibility to use fields in the schema in any order, you can manually assign ids (much like Protocol Buffers); see the id attribute below.

Example:

table { a:int; b:int; c:int; }

This works. The old schema will ignore the presence of the new field c when reading the new data structure. The new schema, when reading old data, will get the default value for c (in this case, 0, because it was not specified).

table { c:int a:int; b:int; }

Adding new fields at the beginning is not allowed, because it makes the old and new versions of the schema incompatible. When old code reads new data, what it reads as the new field c is actually the old field a. When new code reads old data, what it reads as the old field a is actually the old field b.

table { c:int (id: 2); a:int (id: 0); b:int (id: 1); }

This is feasible. If your intent is to order/group semantics in a meaningful way, you can do so by assigning explicit identifiers. Once ids are introduced, the order of fields in the table no longer matters. The new and old schemas are fully compatible, as long as we preserve the id sequence.

Removing Fields

You cannot delete fields that are no longer used from the schema, but you can simply stop writing them into the data, which has almost the same effect as writing and then deleting the fields. In addition, you can mark them as deprecated, as shown in the example above. Fields marked this way will no longer generate C++ accessors, thereby forcing the field to no longer be used. (Be careful: this may break code!)

table { b:int; }

This approach to deleting fields is not workable. We can only remove a field by deprecating it, regardless of whether an explicit ID identifier is used.

table { a:int (deprecated); b:int; }

The approach above is also valid. The old schema will get the default value of a when reading the new data structure, because it does not exist. The new schema code can neither read nor write a (existing code that attempts to do so will result in a compile error), but it can still read old data (it will ignore that field).

Changing Fields

Field names and table names can be changed; if your code continues to work correctly, then you can change them as well.

table { a:uint; b:uint; }

Directly changing the field type may work in some cases, but not in others. It is only feasible when the type change keeps the same size. If the existing data contains no negative numbers, this is safe; if it does contain negative numbers, this change will cause problems.

table { a:int = 1; b:int = 2; }

This change is not feasible. Any legacy data written with a numeric value of 0 will no longer be written to the buffer, and will instead rely on the recreated default value. Those values will now show up as 1 and 2. It may not cause issues in some cases, but you need to be careful.

table { aa:int; bb:int; }

This modification approach may cause issues after changing the original variable name. Because the field has been renamed, it will break all code (and JSON files) that use this version of the schema, which is incompatible with the actual binary buffer.

Tables are the cornerstone of FlatBuffers, because changes to data structures are essential for most applications that need serialization. In general, handling changes to data structures can be done transparently during the parsing process in most serialization solutions. However, a FlatBuffer is not parsed before it is accessed.

To address data-structure changes, tables access fields indirectly through a vtable. Each table carries a vtable (which can be shared among multiple tables with the same layout) and contains information about the fields stored in this particular vtable instance type. The vtable may also indicate that a field is absent (because this FlatBuffer was written by an older version of the software, simply because the information is not required for this instance, or because it is considered deprecated), in which case the default value is returned.

Tables have very little memory overhead (because vtables are small and shared) and low access cost (indirection), while providing a great deal of flexibility. A table may even use less memory than an equivalent struct, because fields do not need to be stored in the buffer when they are equal to their default values.

2. Structs

Structs are very similar to tables, except that no fields in a struct are optional (so there are no default values), and fields may not be added or deprecated. A struct may contain only scalars or other structs. If you are certain that no future changes will be made (as is very clear in the Vec3 example), use it for simple objects. Structs use less memory than tables and are faster to access (they are always stored inline in their parent object and do not use virtual tables).

Structs do not provide forward/backward compatibility, but they have a smaller memory footprint. Storing very small objects that are unlikely to change (such as coordinate pairs or RGBA colors) as structs is very useful.

3. Types

The scalar types supported by FlatBuffers are as follows:

  • 8 bit: byte (int8), ubyte (uint8), bool
  • 16 bit: short (int16), ushort (uint16)
  • 32 bit: int (int32), uint (uint32), float (float32)
  • 64 bit: long (int64), ulong (uint64), double (float64)

The names in parentheses are the corresponding type aliases.

The non-scalar types supported by FlatBuffers are as follows:

  • Arrays of any type. However, nested arrays are not supported; you can replace nested arrays by defining arrays inside a table.
  • UTF-8 and 7-bit ASCII strings. Encoded strings in other formats or binary data should be represented using [byte] or [ubyte].
  • tables, structs, enums, unions

Fields of scalar types have default values. Fields of non-scalar types (string/vector/table) default to NULL if they have no value.

Once a type has been declared, try not to change its type; once you do, errors are very likely to occur. As mentioned above, if you change int to uint, and the data contains negative numbers, errors will occur.

4. Enums

An enum defines a series of named constants. Each named constant can be assigned a fixed value, or by default it increments by one from the previous value. The default first value is 0. As seen in the enum declaration in the example above, use : (byte in the example above) to specify the enum’s underlying integer type, and then determine the type of each field declared with this enum type.

In general, you should only add enum values and not delete enum values (there is no notion of deprecating enum values). This requires developer code to handle forward compatibility by dealing with unknown enum values itself.

5. Unions

This is a type that Protocol Buffers does not yet support.

A union is a concept from the C language. A union can hold multiple types, all sharing the same memory region.

In FlatBuffers, however, Unions can share many properties with Enums, but instead of new names for constants, they use table names. You can declare a Unions field that can contain a reference to any one of these types, meaning that this memory region can be used by only one of those types. In addition, a hidden field with the suffix _type is generated; this field contains the corresponding enum value, so at runtime you can know which type it should be converted to.

A union is similar to an enum, but a union contains tables, while an enum contains scalars or structs.

Unions are a good way to send multiple message types in a single FlatBuffer. Note that because a union field is actually two fields (there is a hidden field), it must always be part of a table; it cannot itself be the root of a FlatBuffer.

If you need to distinguish different FlatBuffers in a more open-ended way, such as files, see the file identification feature below.

Finally, there is an experimental feature supported only in the C++ implementation: as shown in the example above, [Any] (array of unions) is added as a type to the Monster table definition.

6. Root type

This declares what you consider to be the root table (or struct) of the serialized data. This is especially important for parsing JSON data that does not contain object type information.

7. File identification and extension

In general, a FlatBuffer binary buffer is not self-describing; that is, you need to know its schema in order to parse the data correctly. But if you want to use a FlatBuffer as a file format, it is convenient to have a “magic number” there, as most file formats do, so you can perform a full check to see whether you are reading the file type you expect.

Although FlatBuffer allows developers to prepend their own file header to a FlatBuffer, FlatBuffers provides a built-in method that lets the identifier take up minimal space while still keeping FlatBuffers with such identifiers mutually compatible with FlatBuffers that do not have such identifiers.

The way to declare a file format is similar to root_type:

file_identifier "MYFI";

Identifiers must be exactly 4 characters. These 4 characters will be used as bytes [4,7] at the end of the buffer.

For any schema with such an identifier, flatc will automatically add the identifier to any binary file it generates (with -b), and generated calls such as FinishMonsterBuffer will also add the identifier. If you have specified an identifier and want to generate a buffer without one, you can do so by explicitly calling FlatBufferBuilder::Finish directly.

After loading buffer data, you can use a call such as MonsterBufferHasIdentifier to check whether the identifier is present.

Adding an identifier to a file is best practice. If you simply want to send one of a set of possible messages over the network, it is better to use a Union.

By default, flatc outputs binary files as .bin. This declaration in the schema changes it to whatever you want:

file_extension "ext";

8. RPC Interface Declarations

RPC declares a set of functions that take a FlatBuffer as the input parameter (request) and return a FlatBuffer as the response (both must be of the table type):

rpc_service MonsterStorage {
  Store(Monster):StoreResponse;
  Retrieve(MonsterId):Monster;
}

These generated code artifacts and their usage depend on the language and RPC system in use. By adding the --grpc compiler option, the code generator provides preliminary support for GRPC.

9. Attributes

Attributes can be attached to field declarations, either after the field or after the name of a table/struct/enum/union. These attributes may or may not have values.

Some Attributes are recognized only by the compiler, such as deprecated. Users can also define their own Attributes, but they must be declared in advance. Once declared, they can be queried when parsing the schema at runtime. This is very useful when developing your own code compiler/generator, or when you want to add special information, such as help text, to your own FlatBuffers tooling.

The latest version currently recognizes 11 types of Attributes.

  • id:n (on a table field)
    id sets the identifier of a field to n. Once this id identifier is enabled, all fields must use id identifiers, and the ids must be contiguous numbers starting from 0. Special attention is required for Union: because a Union consists of two fields, and the hidden field is ordered before the union field. (Assume that the id of the field before the union has reached 6; then the union will occupy ids 7 and 8: 7 is the hidden field, and 8 is the union field.) After adding id identifiers, the relative order of fields inside the schema no longer matters. The id used by a new field must be the next available id immediately following the previous one (ids cannot be skipped; they must be contiguous).
  • deprecated (on a field)
    deprecated means accessors are no longer generated for this field, and code should stop using this data. Old data may still contain this field, but new code can no longer access it. Note that if you deprecate a field that was previously required, old code may fail to verify new data when using the optional verifier.
  • required (on a non-scalar table field)
    required means this field cannot be omitted. By default, all fields are optional and can be omitted. This is desirable because it helps with forward/backward compatibility and the flexibility of data structures. It also places a burden on reading code, because for non-scalar fields, it requires you to check for NULL and take appropriate action. By specifying a required field, the code that builds FlatBuffers can be forced to ensure this field has been initialized, so the reading code can access it directly without checking for NULL. If the construction code does not initialize this field, it will get an assertion indicating that a required field is missing. Note that if you add this attribute to an existing field, it is only safe when existing data always contains this field and existing code always writes this field.
  • force_align: size (on a struct)
    force_align forces this struct to have a higher alignment than its natural alignment. If the buffer is created with a force_align declaration, then all structs inside it will be forced to align accordingly. (For buffers accessed directly in FlatBufferBuilder, this is not necessarily guaranteed.)
  • bit_flags (on an enum)
    bit_flags means the values of this field represent bits. This means that any value N specified in the schema will ultimately represent 1 « N, or, if no value is specified by default, the sequence 1,2,4,8, … will be used.
  • nested_flatbuffer: "table_name" (on a field)
    nested_flatbuffer means this field, which must be an array of ubyte, contains nested flatbuffer data whose root type is given by table_name. The generated code will generate a convenient accessor for the nested FlatBuffer.
  • flexbuffer (on a field)
    flexbuffer means this field, which must be an array of ubyte, contains flexbuffer data. The generated code will create a convenient accessor for the FlexBuffer root.
  • key (on a field)
    The key field is used in the current table as the key when sorting arrays of its containing type. It can be used for in-place lookup via binary search.
  • hash (on a field)
    This is an unsigned 32/64-bit integer field whose value is allowed to be a string during JSON parsing, and is then stored as its hash. The value of the attribute is the hashing algorithm to use, namely one of fnv1_32, fnv1_64, fnv1a_32, or fnv1a_64.
  • original_order (on a table)
    Because elements in a table do not need to be stored in any particular order, they are usually sorted by size to optimize space. original_order prevents this from happening. There should generally be no reason to use this flag.
  • ‘native_*’
    Several attributes have been added to support the C++ object-based API, all prefixed with “native_”. For details, click the link to view the supported documentation: native_inline, native_default, native_custom_alloc, native_type, native_include: "path".

10. Design Recommendations

FlatBuffers is an efficient data format, but to achieve that efficiency, you need an efficient schema. There are usually multiple ways to represent data with very different size characteristics.

Because of FlatBuffers’ flexibility and extensibility, representing any type of data as a dictionary, as in JSON, is a very common practice. Although this can be emulated in FlatBuffers, as an array of tables with keys and values, doing so is inefficient for a strongly typed system like FlatBuffers and results in relatively large binary files. In most systems, FlatBuffer tables are more flexible than classes/structs, because tables remain very efficient when there are a large number of fields but only a small subset of those fields is actually used. Therefore, data should be organized as tables as much as possible.

Similarly, when possible, use enums instead of strings.

FlatBuffers has no concept of inheritance, so the way to represent a group of related data structures is a union. However, unions do have a cost. Another efficient approach is to create a table. If these data structures have many similar or shareable fields, a single table is recommended and is very efficient. This table can simply contain all fields from all data structures. The reason this is efficient is that optional fields are very cheap and have low overhead.

FlatBuffers can store all integers by default, so choose the smallest size you need instead of defaulting to int/long.

You can consider using a string or table in the buffer to share common data, which improves efficiency. Therefore, splitting repeated data into shared data structures + private data structures is well worth doing.

V. FlatBuffers JSON Parsing

FlatBuffers supports parsing JSON into its own format. That is, the parser that parses the schema can also parse JSON objects that conform to the schema rules. Therefore, unlike other JSON parsers, this parser is strongly typed, and the parsing result is also just FlatBuffers. For details, refer to the flatc documentation and the corresponding C++ FlatBuffers documentation to see how to parse JSON into FlatBuffers at runtime.

To parse JSON, in addition to defining a schema, the FlatBuffers parser has the following changes:

  • It accepts field names both with and without quotes, just as many JSON parsers already do. It can also output them without quotes, but can output them with quotes by using the strict_json flag.
  • If a field has an enum type, the parser recognizes enum symbolic values, with or without quotes, instead of numbers, for example field: EnumVal. If a field is an integer type, you can still use symbolic names, but these values need to be prefixed with their type and enclosed in quotes. field: “Enum.EnumVal”. For enums that represent flags, spaces can be inserted between multiple strings, or dot syntax can be used. For example, field: “EnumVal1 EnumVal2” or field: “Enum.EnumVal1 Enum.EnumVal2”.
  • For unions, these need to be specified using two fields, just as when serializing from code. For example, for field foo, you must add a foo_type：FooOne before the foo field, where FooOne is the table that can be used outside the union.
  • If the value of a field is null, for example, field: null, it means this field has the default value, which has the same effect as not specifying the field at all.
  • The parser has several built-in conversion functions, so you can use the rad(180) function instead of writing 3.14159. The following functions are currently supported: rad, deg, cos, sin, tan, acos, asin, atan.

When parsing JSON, the parser recognizes the following escape codes in strings:

\n - Newline.
\t - Tab.
\r - Carriage return.
\b - Backspace.
\f - Form feed.
\“ - Double quote.
\\ - Backslash.
\/ - Forward slash.
\uXXXX - 16-bit Unicode, converted to the equivalent UTF-8 representation.
\xXX - 8-bit binary hexadecimal number XX. This is the only place that is not part of the JSON specification (see http://json.org/), but it is needed to be able to encode arbitrary binary data in strings as text and back without losing information. For example, byte 0xFF cannot be represented as standard JSON.

When generating JSON back from the binary representation, it will generate these escape codes again.

VI. FlatBuffers Naming Conventions

Identifiers in a schema are intended to be translated into many different programming languages, so changing the schema’s coding style to match the style used by the current project language is the wrong approach. The schema’s coding style should be more general.

  • Table, struct, enum and rpc names (types) use UpperCamelCase.
  • Table and struct field names use snake_case. This allows the generated code methods to use lowerCamelCase automatically.
  • Enum values use UpperCamelCase.
  • namespaces use UpperCamelCase.

There are also 2 recommendations for formatting:

  • Braces: place them on the same line as the start of the declaration.
  • Spacing: indent with 2 spaces. No spaces around :, and one space on each side of =.

VII. Some FlatBuffers “Pitfalls”

In FlatBuffers, however, this applies to everything except scalar values. By default, FlatBuffers does not write fields whose values are equal to their defaults (for scalars), which saves a significant amount of space. However, this also means that testing whether a field “exists” is somewhat meaningless, because it does not tell you whether the field was set by calling the add_field method—unless you care about information represented by non-default values. Default values are not written into the buffer.

The mutable FlatBufferBuilder implementation provides a method named force_defaults, which can disable this behavior: fields are written even if they are equal to their default values. You can then use IsFieldPresent to query whether a field exists in the buffer.

Another approach is to wrap the scalar field in a struct. This way, if it does not exist, it returns null. The nice thing about this approach is that structs do not take up more space than the scalars they represent.

8. Conclusion

After reading this article on the encoding principles of FlatBuffers, readers should understand the following points:

Compared with protocol buffers, FlatBuffers schema definition files provide the following functional “improvements”:

  • Deprecated fields do not require manually assigned field IDs. In .proto, when extending an object, you need to find an unused slot among the numbers (because protocol buffers use a more compact representation, smaller numbers must be chosen). Besides this inconvenience, it also makes deleting fields problematic: if you keep them, it is not semantically obvious that the field can no longer be read or written, and accessors will still be generated. If you delete them, there is a risk of serious bugs, because if someone reuses those IDs, old data may be read, causing data corruption.
  • FlatBuffers distinguishes between table and struct. All table fields are optional, and all struct fields are required.
  • FlatBuffers has a native array type instead of repeated. This gives you a length without needing to collect all items, provides a more compact representation in the case of scalars, and ensures adjacency.
  • FlatBuffers has a union type, which protocol buffers do not have. A union can replace many optional fields, which also saves the time required to check each field one by one.
  • FlatBuffers can define default values for all scalars, without needing to handle their optionality on every access. Since default values do not exist in the buffer, there is no need to worry about space overhead.
  • A parser can handle both schemas and data definitions uniformly (and is compatible with JSON). protocol buffers are not compatible with JSON. The flatc compiler for FlatBuffers also supports more powerful parameters; for the complete list of supported parameters, see this document
  • Schemas extend some Attributes that protocol buffers do not have.

Apart from functional differences, there are also some subtle differences in schema syntax:

  • To define an object, protocol buffers use message, while FlatBuffers use table
  • For IDs, protocol buffers start numbering from 1 by default, while FlatBuffers start from 0 by default.

For the complete syntax of schemas, refer to this document

The next article will cover the principles and source-code analysis related to FlatBuffers encoding and decoding performance.

Reference:

FlatBuffers official documentation
Improving Facebook’s performance on Android with FlatBuffers

GitHub Repo: Halfrost-Field

Follow: halfrost · GitHub

Source: https://halfrost.com/flatbuffers_schema/