[New Codable API] CodingField, et al

[kperryua]

In this issue we will discuss and iterate on the CodingField, StaticStringCodingField, and JSONOptimizedCodingField APIs, with the goal of preparing them for an evolution proposal.

[kperryua]

Overview

In the new Codable system, struct/object decoding and encoding require matching field names from the serialized data against the fields of a Swift type. Today's Codable system uses CodingKey for this — a protocol backed by String and optional Int values, requiring dynamic dispatch and heap allocation. The new system replaces CodingKey with a family of field protocols designed for static dispatch, zero allocation, and compile-time-optimizable key matching.

The field protocol hierarchy has three layers:

1. Base layer (DecodingField, EncodingField, CodingField) — format-agnostic protocols for field name matching during decoding and encoding. All they require is the ability to convert between field enum cases and UTF-8 representations.

2. StaticString layer (StaticStringDecodingField, StaticStringEncodingField, StaticStringCodingField) — refinements for fields backed by StaticString literals. These enable the compiler to work with guaranteed constant pointer data, avoiding any heap allocation. This layer enables raw memcmp-based key matching where Unicode canonical equivalence checks are not required.

3. JSON-optimized layer (JSONOptimizedDecodingField, JSONOptimizedEncodingField, JSONOptimizedCodingField) — These are the fastest path for JSON decoding and decoding, but they're only appropriate for field types where all keys are guaranteed not to require canonical equivalent, nor require escapes when encoded, nor contain characters like double-quotes. With these guarantees, the decoder's optimistic ordered key decoding implementation can utilize a simple memcmp without pre-scanning the string and the encoder can simply memcpy these strings into the output buffer.

The most straightforward conformer for these protocols is an enum that carries no associated values with each encoded field name mapping to one case.

The base protocols

DecodingField

DecodingField is the core protocol for field name matching during decoding:

public protocol DecodingField: ~Escapable {
    associatedtype UTF8SpanComparator: DecodingFieldUTF8SpanComparator & ~Escapable
        = DecodingFieldUTF8SpanCanonicalEquivalenceComparator

    static func field(for key: String) throws(CodingError.Decoding) -> Self
    static func field(for key: UTF8Span) throws(CodingError.Decoding) -> Self
    func matches(_ key: UTF8Span) -> Bool
    var utf8Span: UTF8Span { yielding borrow }
}

The key design elements:

• field(for:) converts an incoming key (as UTF8Span or String) into the corresponding field enum case. If the key is unexpected, the implementation can return an .unknown case or throw.

• matches(_:) checks whether a field case matches a given key. The default implementation delegates to the UTF8SpanComparator associated type.

• utf8Span provides access to the field's UTF-8 representation as a UTF8Span. The implementation uses a yielding borrow accessor so that in the rare case that a backing StaticString has a non-pointer representation it can temporarily allocate memory for the UTF8Span. NOTE: yielding borrow accessor is proposed and presumed for the final design, but not yet implemented. The current prototype has withUTF8Span().

• UTF8SpanComparator associated type. This controls how field names are compared. The default is DecodingFieldUTF8SpanCanonicalEquivalenceComparator (Unicode-correct), but this can be replaced with a custom comparator or DecodingFieldUTF8SpanRawByteEquivalanceComparator for alternative behaviors or increased performance.

EncodingField

EncodingField is the encoding counterpart:

public protocol EncodingField: ~Escapable {
    var utf8Span: UTF8Span { yielding borrow }
}

It's simpler than DecodingField because encoding doesn't need to match incoming keys — it only needs to emit the field's name as UTF-8 bytes. Like DecodingField, the utf8Span property uses a yielding borrow accessor.

CodingField

In typical fashion, field types that are used in both encoding and decoding can conform to the combined CodingField protocol:

public protocol CodingField: EncodingField, DecodingField, ~Escapable { }

In practice, almost every field enum conforms to CodingField (or its JSON-optimized refinement JSONOptimizedCodingField), since most types are both encodable and decodable.

The comparator system

When the decoder reads a key from the input and needs to match it against a known field, the comparison strategy determines the cost per match.

DecodingFieldUTF8SpanComparator

public protocol DecodingFieldUTF8SpanComparator: ~Escapable {
    init(_ span: UTF8Span)
    func matchesSpan(_ span: UTF8Span) -> Bool
    static func ~= (lhs: StaticString, rhs: Self) -> Bool
}

The ~= operator is what makes the field matching work inside switch statements — it allows StaticString literals to be used as case patterns against a comparator value:

static func field(for key: UTF8Span) throws(CodingError.Decoding) -> Field {
    switch UTF8SpanComparator(key) {   // wraps the incoming key
    case "metadata": .metadata         // StaticString ~= comparator
    case "created_at": .created_at
    case "id": .id
    // ...
    default: .unknown
    }
}

The default implementation of ~= is implemented in terms of matchesSpan().

Two concrete comparators

DecodingFieldUTF8SpanCanonicalEquivalenceComparator — the default. Compares field names using UTF8Span.isCanonicallyEquivalent(to:). This is Unicode-correct: two different UTF-8 byte sequences that represent the same characters (e.g. precomposed vs. decomposed forms) will match. This is the safe default for arbitrary serialization formats where the encoder might normalize Unicode differently than the decoder.

DecodingFieldUTF8SpanRawByteEquivalanceComparator — the fast path. Compares field names using memcmp on the raw bytes. Two UTF-8 sequences match if and only if they are byte-for-byte identical. This is safe for JSON because JSON keys in practice use ASCII field names, and even when they don't, the JSON specification requires that keys be compared as raw Unicode code points (not canonical equivalents).

When the compiler sees a switch statement over the raw-byte equivalance comparator, it can optimize this into a length-bucketed binary search: first switch on the byte count, then memcmp only against literals of the matching length. This is the code pattern that makes field matching faster than a traditional switch statement.

StaticString-backed field protocols

The StaticString* protocols provide default implementations for fields whose names are StaticString literals:

public protocol StaticStringDecodingField: DecodingField & ~Escapable {
    var staticString: StaticString { get }
}

public protocol StaticStringEncodingField: EncodingField, ~Escapable {
    var staticString: StaticString { get }
}

public protocol StaticStringCodingField: StaticStringDecodingField, StaticStringEncodingField, ~Escapable { }

When a DecodingField conforms to StaticStringDecodingField, it gets a default matches(_:) implementation that creates a comparator from the incoming key and uses ~= against self.staticString, and a default utf8Span accessor that yields a UTF8Span from the StaticString. The same for EncodingField, StaticStringEncodingField and utf8Span.

JSON-optimized field protocols

The JSON layer adds three more protocols that combine the StaticString and raw-byte-comparison traits with JSON-specific optimizations:

public protocol JSONOptimizedDecodingField: StaticStringDecodingField, ~Escapable
    where UTF8SpanComparator == DecodingFieldUTF8SpanRawByteEquivalanceComparator {
    var safeStringKey: JSONSafeStringKey { get }
}

public protocol JSONOptimizedEncodingField: StaticStringEncodingField, ~Escapable { }

public protocol JSONOptimizedCodingField: JSONOptimizedDecodingField, JSONOptimizedEncodingField, ~Escapable { }

JSONSafeStringKey

@frozen
public struct JSONSafeStringKey {
    public let string: StaticString
}

JSONSafeStringKey wraps a StaticString that is known to the user (or macro) to be safe to use directly in JSON output — meaning it doesn't contain characters that would need to be escaped (no unescaped quotes, backslashes, control characters, etc.). When the encoder sees a JSONSafeStringKey, it can write "field_name": directly into the output buffer without needing to check for characters needing escapes at runtime, which is a significant performance chokepoint.

There is a TODO in the code noting that the original design intended for a compile-time validation check (containsUnescapedQuote) to run inside the JSONSafeStringKey initializer, ensuring safety at build time. However, the constant-folding optimization stopped working for strings longer than 9 characters, so the check is currently disabled to avoid impacting performance. This is a correctness gap: if a field name contains a quote character, the JSON output would be malformed. Without this safety net, the work of validating the field names' safety is up the author of the field type (whether that be the macro or the developer of a manual implementation).

How it all fits together: a complete example

Here's how the Status type from the Twitter benchmark defines its field enum:

struct Status: JSONDecodable {
    // ... stored properties ...

    enum Field: JSONOptimizedCodingField {
        case metadata
        case created_at
        case id
        case id_str
        case text
        case source
        // ... 15 more fields ...
        case lang
        case unknown

        @_transparent
        var staticString: StaticString {
            switch self {
            case .metadata: "metadata"
            case .created_at: "created_at"
            case .id: "id"
            // ...
            case .lang: "lang"
            case .unknown: fatalError()
            }
        }

        static func field(for key: UTF8Span) throws(CodingError.Decoding) -> Field {
            switch UTF8SpanComparator(key) {
            case "metadata": .metadata
            case "created_at": .created_at
            case "id": .id
            // ...
            case "lang": .lang
            default: .unknown
            }
        }
    }
}

Decoding usage

The field enum is used in the decodeEachField iteration pattern:

static func decode<D: JSONDecoderProtocol & ~Escapable>(
    from decoder: inout D
) throws(CodingError.Decoding) -> Status {
    try decoder.decodeStruct { structDecoder in
        var metadata: Metadata?
        var created_at: String?
        // ...

        var field: Field = .unknown
        try structDecoder.decodeEachField { keyDecoder in
            field = try keyDecoder.decode(Field.self)
        } andValue: { valueDecoder in
            switch field {
            case .metadata: metadata = try valueDecoder.decode(Metadata.self)
            case .created_at: created_at = try valueDecoder.decode(String.self)
            // ...
            default: break
            }
        }

        guard let metadata, let created_at, /* ... */ else {
            throw CodingError.dataCorrupted(debugDescription: "Missing required fields")
        }
        return Status(metadata: metadata, created_at: created_at, /* ... */)
    }
}

The keyDecoder.decode(Field.self) call is where the FieldDecoder invokes Field.field(for:) with the raw key bytes — triggering the optimized switch/memcmp matching. A JSONOptimizedDecodingField can also used be used to optimize optimistic ordered key decoding via decodeExpectedOrderField().

Encoding usage

On the encoding side, the same Field enum is used to emit keys:

func encode(to encoder: inout JSONDirectEncoder) throws(CodingError.Encoding) {
    try encoder.encodeDictionary { dictEncoder in
        try dictEncoder.encode(field: Field.metadata, value: metadata)
        try dictEncoder.encode(field: Field.created_at, value: created_at)
        try dictEncoder.encode(field: Field.id, value: id)
        // ...
    }
}

The encode(field:value:) method accesses field.utf8Span to get the key's UTF-8 bytes, then writes them into the output. For JSONOptimizedEncodingField conformers, the encoder can additionally use field.safeStringKey to skip escape-checking entirely.

Macro notes

The macros may end up supporting options like the following:

• @CommonCodable(unknownKeyBehavior: .skip) vs @CommonCodable(unknownKeyBehavior: .throw), which controls whether the generated field(for:) has an .unknown field or rejects unknown keys by throwing.
• @JSONCodable(rawFieldComparisonOptimization: true) to opt in to JSONOptimizedCodingField. The macro could also validate that the keys are good condidates for this optimization, generating an error if a key is considered "unsafe".

Open questions / discussion topics

1. Code size tradeoff.

The field-matching switch statement is a tradeoff that seeks to achieve high throughput while also reducing runtime allocations. The main cost of this approach is higher code size compared to the main alternative: a traditional hash table. A hash table also has a higher (amortized) performance ceiling than the switch statement, but requires numerous allocations and setup overhead. The construction of the table could be performed once on first use and memoized, but this would leave these allocations resident in memory immortally. This strategy also has a higher tendency of increasing heap fragmentation, since it intermixes these permanent allocations with temporary ones. This could be another configuration option for code-size-sensitive or higher-throughput-demand clients that are willing to pay the memory costs.

2. Integer-based field keys for protobuf and similar formats.

The entire field protocol hierarchy is currently string-oriented. This works well for JSON, Property List, MessagePack, and other formats where fields are identified by string keys. But formats like Protocol Buffers, FlatBuffers, and ASN.1 identify fields on the wire by integer tag numbers, not strings. If possible, it would be good for CommonCodable to enable limited support for these formats.

A critical constraint here is integer tags should be explicitly assigned, never automatically derived. Integer field tags become permanent serialization "ABI" — changing them breaks wire compatibility with all existing encoded data. A system that automatically assigns integers (like RawRepresentable enum synthesis) would be a footgun: adding, removing, or reordering properties could silently change tag numbers and break deserialization. The design must require authors to explicitly declare each tag number, most commonly via a macro, which ultimately would be used to implement a new integer-focused CodingField conformance.

I expect most CommonCodable adopters will likely not anticipate their type being used with integer-tagged formats, and will therefore not appreciate being required to think about the complexities of integer tags and forward compatibility. Thus, the need for explicit opt-in and the desire for format agnosticism in CommonCodable are in conflict.

I'm unsure how best to navigate this conflict. I can think of a couple of options.

1. Runtime enforcement: We could add integer tag provisions to the CodingField protocols with default implementations that trap or throw errors. This would prevent clients from using non-integer-tag-aware CommonCodable types with a hypothetical Protobuf encoder, but the limitation would only be caught at runtime.
2. Separate protocols: We might be able to add a new set of "Common" protocols focused on integer tags that refine the string-based ones. A hypothetical Protobuf encoder would accept only types that conform to ProtobufEncodable OR CommonIntegerEncodable. It's unclear how much duplication or complication this adds to the existing API surface.

Another open question is in regards to tag integer width and type. Protobuf tags always fit in a UInt32. FlatBuffers uses UInt16. Some formats allow negative integers. Some formas allow arbitrary sized integers. Is there a scheme that lets encoders/decoders declare their integer tag type (mabye via associatedtype)? Or do we have to compromise with a single canonical integer type? If the latter, what would that type be? (We probably want to avoid any usage of Int or UInt here given that its definition changes depending on device architecture.)

To complicate matters more, it might also be a semantic mistake to require a Common(Integer)Codable type to unify integer values anyway. There might be legitimate need for a Protobuf and Flatbuffer tag value to be different for the same field.

Again, kotlinx.serialization has an attactive-looking solution to this problem, though it unfortunately doesn't translate well to Swift. In that environment, you can declare the following:

@Serializable
data class User(
    @ProtoNumber(1) @CborLabel(10) val name: String,
    @ProtoNumber(3) @CborLabel(20) val id: Int,
    @ProtoNumber(4) @CborLabel(30) val email: String
)

These @ProtoNumber and @CborLabel constructs are represent annotations, not attribute macros. They get constructed as class instances and the Serializable macro adds them to a type-erased list of annotations per field. At encode time, the Protobuf or CBOR encoder filters through that list looking for the annotation of the desired type at runtime. In Swift this would require type erasing boxes and dynamic casting, which are much more expensive in Swift than Kotlin, and is also fundamentally incompatible with Embedded Swift.

Until a clearly superior Swift design for supporting integer field tags is fou, the approach I've taken for now is the simpler string-centric one for "format-agnostic" (such as it is) encoding, This means deferring integer field tag support to types that explicitly adopt a hypothetical future format-specific @ProtobufCodable, etc. macro.