Stdlib proposal: "Overridable methods" should be namespaced

[lipfangmoe]

I had previously made a Ziggit post about this, and I wish I would've gotten a bit more feedback before making a more formal proposal, but for now this will have to do!

There are a few places in the Zig Standard Library where a pattern of an "overridable method" is used. The ones I personally use most often are format, and the json* (jsonParse/jsonParseFromValue/jsonStringify)

However, I do have one big gripe still. Namely, it can often be extremely frustrating to write reusable (de)serializers. One use-case I have for reusable serializations is Packed Flag structs, which should be parsed and stringified as integers. So I'll need to implement the format, jsonStringify, jsonParse, and jsonParseFromValue methods for each case:

pub const MyBitFlags = packed struct {
    foo: bool = false,
    bar: bool = false,
    baz: bool = false,

    const Mixin = PackedFlagsMixin(@This());
    pub const format = Mixin.format;
    pub const jsonStringify = Mixin.jsonStringify;
    pub const jsonParse = Mixin.jsonParse;
    pub const jsonParseFromValue = Mixin.jsonParseFromValue;
}

However, this gets extremely verbose to write over and over again, especially considering the old solution was a single line: pub usingnamespace PackedFlagsMixin(@This());

Namespacing is good, actually

This is of course the reasoning behind removing usingnamespace in the first place! And it's a good reason. However, because the standard library doesn't namespace overridable serialization methods, so there was no great alternative to mixins for those cases (besides writing a line for each specific method you want to override).

A good solution could be to put serialization methods in a namespace. For instance:

pub const MyBitFlags = packed struct {
    foo: bool = false,
    bar: bool = false,
    baz: bool = false,

    pub const Fmt = PackedFlagsFmt(@This());
    pub const Json = PackedFlagsJson(@This());
}

And then fmt would look for T.Fmt.format instead of T.format, and json would look for T.Json.parse instead of T.jsonParse.

Another related option would be a "standard" namespace (i.e. Mixins), so that a single mixin could be used to define all overridable behaviors, similar to what we had before with usingnamespace:

pub const MyBitFlags = packed struct {
    foo: bool = false,
    bar: bool = false,
    baz: bool = false,

    pub const Mixins = PackedFlagsMixin(@This());
}

In this case, fmt would look for T.Mixins.format, and json would look for T.Mixins.jsonParse.

Feedback welcome!

[mlugg]

FWIW, I would say that we're generally moving away from having overrideable methods at all. This comment talks about this a little. There are no plans to remove format, but if you have a bunch of types with similar formatting logic (AIUI that's your use case), rather than using a mixin-style API, it's probably easier to just use a custom formatter instead (that's just a separate type with its own format function; see std.fmt.alt/std.fmt.Alt for some related helpers). JSON (etc) is trickier because of the fact that you kind of want to be able to override any type at any "depth" into the root thing you're parsing, but there are still some possible solutions, like this one proposed by InKryption. One of those could lead to jsonStringify/jsonParse/etc being removed entirely.

[edqx]

Not entirely relevant to the OP, but IMO it would be nice if Zig had something akin to JS' Symbols, such that format could be referenced with an opaque identifier rather than a name.

Using JS syntax:

pub const MyBitFlags = packed struct {
    foo: bool = false,
    bar: bool = false,
    baz: bool = false,

    pub fn [std.fmt.format](self: MyBitFlags, writer: *std.Io.Writer) !void {
        ...
    }
}

I think it's a shame to rely on convention to make format essentially a reserved keyword, along with jsonStringify, jsonParse, etc. and any other libraries that use the pattern.

[lipfangmoe]

@mlugg (and cc @MasonRemaley)

My concern with this property is that it's harder to understand code that has it, and naming the method incorrectly leads to code that compiles but behaves differently.

I haven't yet been able to think of a neat way to offer custom parsers per type without violating this property which is why std.parse doesn't have this feature yet, but I'm open to ideas.

In terms of the way that other languages solve this:

Many languages use traits/interfaces to solve this issue, along with some kind of annotation or keyword for methods to ensure that if you misspell the method name, a compiler error is generated. Zig isn't very well-equipped for a solution like this, as there aren't concepts of "interfaces" or "overriding". But now that I think of it, the Fmt and Json decls are kind-of similar to a trait, in that you're declaring that you want to override the behavior of "Fmt"/"Json". It doesn't change the root issue though - the presence of an arbitrary method name entirely changes the behavior of how the struct is serialized.

JavaScript I think actually has one of the best solutions to this problem. JS has a symbol type, with the only way to get access to the symbol to be to get it from the identifier that it's assigned to. They're roughly similar to Zig enum values, but if you were required to access them through where they were defined (i.e., enum literals would not be legal in this equivalency). Anyway, since symbols are valid object keys, they're a great way to define overridable behaviors - for instance, to make an object "iterable", you can define a Symbol.iterator key for your object like so:

const myIterableObject = {
    [Symbol.iterator]: function*() {
        yield 1;
        yield 2;
        yield 3;
    },
}

Possible Solution? Symbol identifiers:

A solution could be a combination of the previous two solutions: some way to strongly bind a decl's value with an identifier, possibly from a different file:

// lib.zig
pub const foo = @symbol();

pub fn callFoo(a: anytype) void {
    a.[foo]();
}

// main.zig
const lib = @import("./lib.zig");

pub const Bar = struct {
    pub fn [lib.foo]() { ... }
    // alternatively
    pub const [lib.foo] = undefined; // replace undefined with some random function
};

pub fn main() void {
    const bar = Bar{};
    lib.callFoo(bar);
}

This could be used with Writer.print like so:

// Writer.zig

pub const formatFn = @symbol();

pub fn printValue(...) {
   // ...
   switch (fmt[0]) {
       'f' => value.[formatFn](w),
    }
    // ...
}

and then people could "implement" the format function like this:

const std = @import("std");

pub const MyFormat = packed struct {
    // the "fully namespaced way"
    pub fn [std.Io.Writer.formatFn](self: MyFormat, writer: *std.Io.Writer) std.Io.Writer.Error!void {
        try writer.writeAll("Hello, world!");
    }

    // also legal
    const format = std.Io.Writer.formatFn; // could be defined at top-level
    pub fn [format](...) ... { ... }

    // also legal
    pub const [std.Io.Writer.formatFn] = undefined;

    // also legal
    const format = std.Io.Writer.formatFn; // could be defined at top-level
    pub const [format] = undefined; 
}

This would then solve the problem of misspellings continuing to compile, and it also solves the JSON problem which was also mentioned in the linked comment:

I believe that the JSON parser is the only remaining place in the standard library that has the property that the presence or absence of a method can silently change the behavior of the API

Because the symbol is strongly linked to the std.json package, I'd say the behavior is explicitly opt-in, and no longer silently changing the behavior of the API.

This could also be a better alternative for root-level declarations like pub const std_options: std.Options (which could now be something like pub const [std.options]: std.Options)

The huge downside to this of course is that this would be a language change.

Another possible solution? Reconsidering @Type allowing declarations

This could solve the issue by allowing something like:

pub const MyType = std.json.Serializable(formatter, struct {
    some_field: i32,
});

Where std.json.Serializable could dynamically generate a semi-random identifier name (ie jsonParse_ah3hh34jb6o with each build, and formatter would contain the parse/parseFromValue/stringify functions. This is awful though, as it makes everything and @Type allowing declarations is a pandora's box which will have all sorts of other implications, so I'd rather not open it.

[lipfangmoe]

Not entirely relevant to the OP, but IMO it would be nice if Zig had something akin to JS' Symbols, such that format could be referenced with an opaque identifier rather than a name.

Using JS syntax:

pub const MyBitFlags = packed struct {
foo: bool = false,
bar: bool = false,
baz: bool = false,

pub fn [std.fmt.format](self: MyBitFlags, writer: *std.Io.Writer) !void {
    ...
}

}

I think it's a shame to rely on convention to make format essentially a reserved keyword, along with jsonStringify, jsonParse, etc. and any other libraries that use the pattern.

lol, I like how we came up with the exact same solution

[lipfangmoe]

Actually, when I further consider @InKryption's solution, I like it a lot more than I initially did. I had initially mistook that there was no way for an arbitrary struct to have ownership over how it's parsed, and that I'd need to entirely restructure my entire library (which makes extremely heavy use of JSON parsing).

Implementing the old behavior for std.json would actually be quite easy!

pub const ParseCtx = struct {
    pub fn parse(ctx: *@This(), comptime T: type, arena: std.mem.Allocator, source: anytype, options: std.json.ParseOptions) !T {
        if (std.meta.hasFn(T, "jsonParse")) {
            return try T.jsonParse(ctx, arena, source, options);
        }
        return try std.json.innerParse(ctx, T, arena, source, options);
    }
};

var ctx: ParseCtx = .{};
const result = try std.json.parseFromSlice(T, gpa, "{}", .{ .parse_ctx = &ctx });

(I didn't quite understand the purpose of canParse, so I decided to omit it. Although it'd make sense if the Zon parser doesn't have an equivalent for innerParse)

This would also remove the need for this namespacing proposal altogether, as then the target struct could have something like a marker decl which the individual contexts could read:

// example flags struct
pub const Flags = packed struct {
    foo: bool = false,
    bar: bool = false,
    baz: bool = false,

    pub const packed_flags_marker = void{};
};

// example usages
pub const ParseCtx = struct {
    pub fn parse(ctx: *@This(), comptime T: type, arena: std.mem.Allocator, source: anytype, options: std.json.ParseOptions) !T {
        if (@hasDecl(T, "packed_flags_marker")) {
            // deserialize packed flags
        }
        return try std.json.innerParse(ctx, T, arena, source, options);
    }
};

pub const ParseFromValueCtx = struct {
    pub fn parseFromValue(ctx: *@This(), comptime T: type, arena: std.mem.Allocator, source: std.json.Value, options: std.json.ParseOptions) !T {
        if (@hasDecl(T, "packed_flags_marker")) {
            // deserialize packed flags
        }
        return try std.json.innerParseFromValue(ctx, T, arena, source, options);
    }
};

pub const StringifyCtx = struct {
    pub fn stringify(ctx: *@This(), value: anytype, json_writer: anytype) !void {
        if (@hasDecl(@TypeOf(value), "packed_flags_marker")) {
            // serialize packed flags
        }
        return try json_writer.write(ctx, value);
    }
};

var parse_ctx: ParseCtx = .{};
const from_slice = try std.json.parseFromSlice(T, gpa, "{}", .{ .ctx = &parse_ctx });

var parse_from_value_ctx: ParseFromValueCtx = .{};
const from_value = try std.json.parseFromValue(T, gpa, someJsonValue, .{ .ctx = &parse_from_value_ctx });

var stringify_ctx: StringifyCtx = .{};
const flags: Flags = .{ .bar = true };
try std.json.Stringify.value(flags, .{ .ctx = &stringify_ctx }, &some_writer.interface);

Love this solution, and it doesn't even require any language changes. It also allows for people to implement some of their own solutions for things like diagnostics! (Maybe I shouldn't have added .ctx to the parse/stringify options, and should've instead had a separate method for adding a context. Idk. Someone better at design can figure that out for me :) )

[InKryption]

(I didn't quite understand the purpose of canParse, so I decided to omit it. Although it'd make sense if the Zon parser doesn't have an equivalent for innerParse)

The purpose behind canParse is principally the ability to tell the caller whether the given context supports parsing the given type, ie in the event that you would like to cede to fallback behavior. That minimizes the responsibilities your context has to taken on. Can additionally be used for composition between contexts to do the same thing: check if some parameterized context wants to take charge of a given type, and if not, apply your own fallback logic.

[lipfangmoe]

closing since it sounds like there's a better solution in the works