Skip to content

Serialization Policy

Jump to bottom
Steven Schveighoffer edited this page Aug 9, 2025 · 8 revisions

Overview

Serialization and Deserialization in iopipe is done via the serialize module. This module contains the entry points serialize and deserialize. The library offers multiple mechanisms to customize how serialization should be performed, with reasonable defaults.

Serialization

Serialization is done by converting data (structs/classes, etc) into valid json or json5 to be sent to an output pipe.

Currently, all serialization is determined by the item itself, along with default serialization functions.

jsoniopipe handles the following types:

D Type Examples Description Relevant UDAs
null type typeof(null) Serialized as null.
numbers int, float Serialized as a JSON number as appropriate.
booleans bool Serialized as true or false
strings string, char[], const(wchar)[] Serialized to a JSON compliant string, inserting escapes where needed.
arrays int[], string[], float[5] Serialized via a JSON array, items serialized recursively
input range Chain!(int[], int[]) Serialized as JSON array, items serialized recursively
enums enum X { a, b} Serialized as the name of the enum. enumBaseType
AAs with string-like key int[string] Serialized as a JSON object, with the key translated to a string field name, and the value serialized recursively
Nullable!T Nullable!int, Nullable!string Serialized as null or a value recursively.
JSONValue!Char JSONValue!Char Direct representation of JSON, serialized as specified. extras
struct not covered above struct Point { int x, y; } Serialized as a JSON object, with field names mapped and values serialized recursively. Can be customized with a toJSON member function. IgnoredMembers, serializeAs, extras, ignore, alternateName
class or interface class Point { int x, y; } If null, serialized as null. Otherwise, serialized as a JSON object, with field names mapped and values serialized recursively. Can be customized with a toJSON member function. IgnoredMembers, serializeAs, extras, optional, ignore, alternateName

Serialization with policy

A policy can be provided to customize the behavior of serialization.

Details TBD.

Deserialization

Deserialization is the population or generation of a D type given an input JSON or JSON5 stream. First the stream is parsed into JSON tokens. The stream of tokens is then passed to the deserializer.

The process is not driven by the incoming data but by the type being deserialized. The input stream may determine how types get serialized, but the ultimate determination is by the type. In other words, the type defines how it is deserialized, and the input stream is expected to conform to this design.

By default, deserialization happens according to the following rules:

D Type Examples Description Relevant UDAs
null type typeof(null) Deserializes a null token.
numbers int, float Deserializes a number token.
booleans bool Deserializes a true or false token.
strings string, char[], const(wchar)[] Deserializes a JSON string token. Makes a copy if the input stream window differs in type
arrays int[], string[], float[5] Deserializes a JSON array recursively.
enums enum X { a, b} Deserializes a string converted to the enum. enumBaseType
AAs with string-like key int[string] Deserializes a JSON object recursively, with the key translated from a JSON string field name
Nullable!T Nullable!int, Nullable!string Deserializes null as Nullable!T.init, or a value recursively.
JSONValue!Char JSONValue!Char Deserialized from any valid JSON token stream. extras
struct not covered above struct Point { int x, y; } Deserialized from a JSON object, with field names mapped and values serialized recursively. Can be customized with a fromJSON static member function. IgnoredMembers, serializeAs, extras, ignore, alternateName
class or interface class Point { int x, y; } If token is null, set to null. Otherwise, derialized from a JSON object, with field names mapped and values serialized recursively. Can be customized with a fromJSON static member function. IgnoredMembers, serializeAs, extras, optional, ignore, alternateName

Deserialization with Policy

A deserialization policy can be used to customize all aspects of deserialization. A user can specify how to deserialize types, including types they do not control (and therefore cannot add special toJSON or fromJSON hooks).

The entry point for deserializing a type is deserializeImpl.

The hook is called like this:

policy.deserializeImpl(tokenizer, item);

IMPORTANT: a single deserializeImpl member function in a policy will override all deserializeImpl calls. Therefore, you must handle all types you want to be deserialized through the policy, even if you want to forward to a default implementation. The reason it is done this way instead of using a dispatch function is because D's only tool to check for a match is __traits(compiles). This means hooks that have a bug in them that causes them not to compile would fail to match, and confusingly the library would pick the default implementation without any indication of what is wrong.

The recommended mechanism to hook calls is to define a single function template, and use static if to distinguish implementations, with an ultimate call to the default deserializeImpl if desired.

All deserializeImpl default implementations are public for this reason.

Deserializing JSON Object with callbacks

Any type being deserialized from a JSON object can use the callback mechanism to deserialize. This mechanism uses the entry point deserializeObject(policy, tokenizer, item). This function validates the object structure from JSON and calls the following three policy callback functions:

Context onObjectBegin(ref JT tokenizer, ref T item);
void onField(ref JT tokenizer, ref T item, string fieldname, ref Context ctx);
void onObjectEnd(ref JT tokenizer, ref T item, ref Context ctx);

The Context type can be defined by the policy. This is a piece of information that is initialized and stored on the stack during deserialization per object. If you don't need any context, you still need to return something other than void. You can return ubyte[0] if you don't care.

If you do not provide these callbacks, then a default callback is used.

The default deserializeImpl for structs and classes as above will use the deserializeObject entry point function.

Deserializing JSON Array with callbacks

Arrays or array-like structures can use a callback system to deserialize. An entry point for this can be called via deserializeArray(policy, tokenizer, item). The three callback functions are:

Context onArrayBegin(ref JT tokenizer, ref T item);
void onArrayElement(ref JT tokenizer, ref T item, size_t idx, ref Context ctx);
void onArrayEnd(ref JT tokenizer, ref T item, size_t length, ref Context ctx);

Note that the onArrayElement function accepts the array type as T, not the element type. This allows the array callbacks to determine how the element is initialized.

By default, static arrays use the deserializeArray callback mechanism. Dynamic arrays by default use the deserializeArray mechanism into a std.array.Appender, and then assign the resulting array data to the item.

Examples for using a policy

An example for using a policy to override deserialization of a DateTime type from a JSON string.

struct DTStringPolicy {
    void deserializeImpl(JT, T)(ref JT tokenizer, ref T item) {
        static if (is(T == DateTime))
        {
            // Deserialize DateTime from a string
            auto jsonItem = tokenizer.nextSignificant
                .jsonExpect(JSONToken.String, "Parsing DateTime");
            item = DateTime.fromSimpleString(extractString!string(jsonItem, tokenizer.chain));
        }
        else {
            // default to module behavior
            .deserializeImpl(this, tokenizer, item);
        };
    }
}

This only overrides the deserialization of DateTime and not other struct types. All others go to the standard deserializer function.

Clone this wiki locally