How to use a transform type with Type.Union?

[Gnuxie]

Typically in the decoder for a transform type, i have been throwing TypeErrors when the serialized value being decoded is invalid. However, this stops transform types being used in a union. The examples in the readme don't seem to do anything special to prevent this, so i'm wondering if you have a workaround or some insight (like a custom error type)?

Thanks again for your amazing project.

[sinclairzx81]

@Gnuxie Hi

Typically in the decoder for a transform type, i have been throwing TypeErrors when the serialized value being decoded is invalid. However, this stops transform types being used in a union. The examples in the readme don't seem to do anything special to prevent this, so i'm wondering if you have a workaround or some insight (like a custom error type)?

Yeah, Union types have proven to be somewhat challenging to use with Transform. The reason mostly comes down to the Transform needing to accommodate (test for) multiple possible values within the codec, and where multiple values can lead to ambiguity as to that form the Encode / Decode function should produce (it could be one of many possible forms).

As of writing, the general recommendation is to avoid applying Transform to a Union directly, and instead apply the Transform to each Union variant. By doing this, it's possible to reason about the transformation specific to the union variant (treating the variant schema as the source of truth). For Encode specifically, instead of throwing, try return an invalid value instead such as null, Invalid values will be tested as part of the transform Encode pipeline.

This is a rough example.

import { Value } from '@sinclair/typebox/value'
import { Type } from '@sinclair/typebox'

// Transform Applied to Variants

const A = Type.Transform(Type.Number())
  .Decode(value => `number:${value}`)
  .Encode(value => parseFloat(value.replace('number:', ''))) // note: will return NaN if invalid. This is an
                                                             // example of an invalid value that will be tested
                                                             // against Number() internal to the Encode pipeline.
                                                                                              
const B = Type.Transform(Type.Boolean())
  .Decode(value => `boolean:${value}`)
  .Encode(value => value.replace('boolean:', '') === 'true')

// Compose Union with Interior Transform Variants

const T = Type.Union([A, B]) 

// Test

const D_A = Value.Decode(T, 100)
const D_B = Value.Decode(T, true)

console.log({ D_A,  D_B })       // { D_A: 'number:100', D_B: 'boolean:true' }

const E_A = Value.Encode(T, D_A)
const E_B = Value.Encode(T, D_B)

console.log({ E1: E_A, E2: E_B }) // { E1: 100, E2: true }

---

There is quite a lot of nuance to transforming Union types, but hope the above provides a strategy to help process them. Happy to discuss more on this thread.

Cheers
S

[Gnuxie]

Ah, ok, that makes sense when we are decoding from different types. My issue was that I am trying to decode a string using either of two transforms:

const A = Type.Transform(Type.String())
  .Decode(value => {
    if (value.startsWith('a')) {
      return createA()
    } else {
      throw new TypeError(`I don't know what to do with things that aren't A ${value}` )
    }
  }).Encode(_value => 'a')

const B = Type.Transform(Type.String())
  .Decode(value => {
    if (value.startsWith('b')) {
      return createB()
    } else {
      throw new TypeError(`I don't know what to do with things that aren't B ${value}` )
    }
  }).Encode(_value => 'b')

const AB = Type.Union([A, B])
const value_a = Type.Decode(AB, 'a') // ideally would be the object returned by `createA`
const value_b = Type.Decode(AB, 'b') // ideally would be the object returned by `createB`
const value_c = Type.Decode(AB, 'c')  // TypeError -- neither of the transforms can handle this variant.

[Gnuxie]

I guess I might be stupid for trying this. In the specific context though I was just trying to combine transform types without needing to write a transform that can account for all the variants. And I think I also forgot that the transform is specific to a schema.... and i guess that it could be possible to use the format option in Type.Stringas a workaround?

[sinclairzx81]

@Gnuxie Hiya,

And I think I also forgot that the transform is specific to a schema.... and i guess that it could be possible to use the format option in Type.Stringas a workaround?

You're on the right track. A key thing to consider is that each variant needs to be uniquely distinguishable, and that the schema for each variant needs to (as best as it can) validate the data prior to the value reaching Decode. The following loosely achieves this using pattern

import { Value } from '@sinclair/typebox/value'
import { Type } from '@sinclair/typebox'

const A = Type.Transform(Type.String({ pattern: '^a.*' }))
  .Decode(value => `createA:${value}` as const)
  .Encode(_value => 'a')
                         
const B = Type.Transform(Type.String({ pattern: '^b.*' }))
  .Decode(value => `createB:${value}` as const)
  .Encode(_value => 'b')

const AB = Type.Union([A, B])
const value_a = Value.Decode(AB, 'apple') // ideally would be the object returned by `createA`
const value_b = Value.Decode(AB, 'bicycle') // ideally would be the object returned by `createB`

// this will error as 'c' doesn't match either pattern
// const value_c = Value.Decode(AB, 'c')  // TypeError -- neither of the transforms can handle this variant.

console.log({ value_a, value_b })

For more complex string formats, you can use format. Interestingly, I got a request the other day from a user wanting to upload WebAssembly encoded in base64 strings. In that particular case, there was a requirement to validate the assembly before the rest of the application could use it. In that scenario,

FormatRegistry

1. decode the base64 as a Uint8Array (inside a try/catch)
2. instantiate the WebAssembly from Int8Array (inside a try/catch)
3. Return true if no error

Decode

1. Instantiate the Web Assembly (as it had already been tested via the format)
2. Return it to the application along with a Uint8Array (for later Encode)

Not the most efficient way to approach things, but in lieu of a WebAssembly validator, I thought it was a reasonable example of how to reason about decoding (where the schema (and associated checks) act as a predicate before Decode. A key aspect of transforms is the Decode "must" trust that the schema has done everything it can to check the value before decoding (this way errors technically can't happen, in theory)

---

Sounds like you're on the track. TypeBox transforms have been a challenging thing to get right in the codebase. I am open to revisiting the current implementation in future, but it's difficult to imagine how it could be improved given the way TypeBox is designed. Community insight into how users are approaching Transforms is helpful tho, but Transforms + Union (something that comes up time and time again) I think will always be difficult aspect of the library, but I'm open to suggestions :D

Happy to keep discussing things if you have any additional follow up questions,
Cheers!
S

[sinclairzx81]

@Gnuxie Hiya,

I'll convert this issue into a Discussion as it's more a general enquiry than an issue. Happy to discuss more if you have any follow up questions or comments though. The above strategy (applying transforms to tuples) is generally the recommended way to approach things so should provide a reasonable basis to work from.

Let me know how you go
Cheers
S