Chapter 4 Encoding and Evolution

4.1 Avro

4.2 Thrift

4.3 Protocol Buffers

Protocol Buffers Internals


Protocol buffers messages always use little-endian encoding. Implementations running on big-endian architectures should be doing the conversions automatically.

4.4 Flatbuffer


Posted on Thursday, August 17, 2017 by Wouter van Oortmerssen The recent release of Flatbuffers version 1.7 introduced truly zero-copy support for gRPC out of the box.

Flatbuffers is a serialization library that allows you to access serialized data without first unpacking it or allocating any additional data structures. It was originally designed for games and other resource constrained applications, but is now finding more general use, both by teams within Google and in other companies such as Netflix and Facebook.

Flatbuffers enables maximum throughput by directly using gRPC’s slice buffers with zero-copy for common use cases. An incoming rpc can be processed directly from gRPCs internal buffers, and constructing a new message will write directly to these buffers without intermediate steps.

This is currently, fully supported in the C++ implementation of FlatBuffers, with more languages to come. There is also an implementation in Go, which is not entirely zero copy, but still very low on allocation cost (see below).

Each scalar is also always represented in little-endian format, as this corresponds to all commonly used CPUs today. FlatBuffers will also work on big-endian machines, but will be slightly slower because of additional byte-swap intrinsics.