feature: Add String Type Support Using Regex and Standard Built-in Functions

[hospotho]

Describe the feature

Basic regex support and built-in string functions should be relatively easy to integrate into validate, parsing and serialization function.
Using the email validation example from your video, adding regex support would only require one additional line:

const $$User = {
  validate(input: unknown): input is User {
    return (
      typeof input === 'object' &&
      input !== null &&
      'id' in input &&
      typeof input.id === 'string' &&
      'name' in input &&
      typeof input.name === 'string' &&
      'email' in input &&
      typeof input.email === 'string' &&
      // add regex test
      /^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/.test(input.email)
    );
  },
  parse,
  serialize,
};

I haven’t gone through the entire ts-schema/src/compiler/, but it seems feasible to do so. Maybe some refactoring on a.string() (or just setter proxy for a.object?) and add some templates for regex and built-in string functions.

Additional context

Would you be willing to implement this feature?
Yes, but I may need some time to familiarize myself with the codebase.

[joshmossas]

I'm open to this. I've been thinking a lot about how to go about handling the most common string cases and my thought is that adding these additional validations is fine so long as we mark them as "non-portable". By "non-portable" I mean these validations won't necessarily included in the compilation to other language clients. This would give us flexibility to make validation easier on the TS server side without forcing us to consider what that would look like in every language Arri supports.

Before we get into actually implementing this I think a discussion probably needs to happen around what the API should look like. I think however these additional validations are implemented, they should feel "separate" from the primitive types available in Arri. Something like:

const Foo = a.object({
  foo: a.email(),
  bar: a.regex(),
})

Probably won't do because it makes email and regex feel like a primitive type. (I'm open to changing my mind on this, I just think it might make things confusing once we get to cross language compilation)

I have a couple ideas of what the API itself could look like, but please I'm open to input as well.

Possibility 1: Add A Refinement Operator

Maybe we could have a refinement operator like a.refine() which accepts a primitive type as it's first parameter followed by one or more "refinements". This API is sort of similar to Valibot's pipe function except that refine cannot transform the input. It can only add additional validation checks.

const User = a.object({
  id: a.refine(a.string(), a.uuid()),
  email: a.refine(
    a.string(),
    a.regex(/^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/, "invalid email"),
  ),
  username: a.refine(
    a.string(),
    a.minLen(5, "too short"),
    a.maxLen(20, "too long"),
  )
})

We could repeat this sort of approach for other types as well.

Internally we can setup some kind of type that if conformed to provides enough information to the compiler that it can be included in the compiled versions.

Possibility 2: Adding functionality on the string function

// String accepts refinements as parameters
const User = a.object({
  id: a.string(a.uuid()),
  email: a.string(a.regex(/^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/, "invalid email")),
  username: a.string(a.minLen(5, "too short"), a.maxLen(20, "too long")
})

// refine is a function on string itself. More Zod like but we risk it feeling foreign compared to the rest of the codebase
const User = a.object({
  id: a.string().refine(a.uuid())
  email: a.string().refine(a.email())
  username: a.string().refine(minLen(5, "too long"), maxLen(20, "too long")
})

[joshmossas]

We could also go even simpler as well. Maybe we give a.string() options that handle the most common use cases.

a.string({
  minLen: 5,
  maxLen: 20,
  type: "email"
})

const User = a.object({
  id: a.string({ type: "uuid" }),
  email: a.string({ type: "email" }),
  username: a.string({ minLen: 5, maxLen: 20 })
})

// "type" could also be called something else like "format" or whatever

Benefit of this is that we could probably more easily integrate something like this into the cross language compilation. Maybe even some of the string types make their way into Arri Type Definition. Just some thoughts

[hospotho]

I have some concerns with the current approaches:

1. Refinement - feels a bit unintuitive, and users might misuse it
2. Extending a.string() - will introduce unnecessary coupling
3. Defining options - seem not a good practice for cross language and could impact performance

I think a builder pattern could be a better fit. It would provide flexibility while keeping cross-language compatibility in mind.
Here’s a rough idea:

1. Introduce StringTypeBuilder with validateTemplateList, parseTemplateList, and serializeTemplateList
2. Can only create or init by a.string() or other primitive types (different TypeBuilder)
3. Add support functions like email(), regex(), optional(), minLen(), etc, accepting fallback values
4. a.compile() can then use the template list to generate the final validation, parsing, and serialization logic
5. This approach could also extend to other types, eg. timestamp validation a.timestamp().before('2025-01-01')

[joshmossas]

I agree that a builder pattern is probably the most ergonomic. Although, I have some concerns.

This schema library was structured around nesting functions similar to valibot and typebox

const Foo = a.object({
  foo: a.optional(a.string()),
  bar: a.nullable(a.boolean()),
  baz: a.array(a.optional(a.float64())),
});

Given that architecture, wouldn't adding things like StringTypeBuilder feel out of place?

const Foo = a.object({
  // Could feel out of place? 
  foo: a.optional(a.string().minLen(5).maxLen(10))
})

Secondly, all of these functions are tree-shakeable. So if you use the modular import syntax as outlined here. The bundle size of Arri schema starts as low as 4kb. Using a builder pattern means, that all of these built-in string functions will be imported whether you use them or not.

Now I'm not hard set on keeping bundle size as minimal as possible, but it is something we should consider.

Also some additional context:

While discussing this we must keep in mind that Arri schema is meant for building Arri Type Definitions. So any schema built with this library must also be a valid Arri Type Definition.

Example this:

const Foo = a.object({
  foo: a.optional(a.string()),
  bar: a.nullable(a.boolean()),
  baz: a.optional(a.array(a.float64())),
})

JSON.stringify(Foo)

Will output

{
  "properties": {
    "bar": {
      "type": "boolean",
      "nullable": true
    }
  },
  "optionalProperties": {
    "foo": {
      "type": "string"
    },
    "baz": {
      "elements": {
        "type": "string"
      }
    }
  }
}

A builder pattern isn't necessarily in opposition to this (since extra methods would get stripped by JSON.stringify), but it's something to keep in mind while designing this API. This specification is also how Arri handles cross-language compilation. So any cross-language validations would need to be represented in this spec somehow, but as I said before I'm actually fine with many of these validations being "non-portable". In that case extra validations would run on the server via Arri Schema, but not the generated clients. Which is still fine IMO. (Example: The client doesn't technically have to check if the string has a minimum length because it's going to be checked on the server anyway)

Defining options - seem not a good practice for cross language and could impact performance

Part of the reason why I was considering adding options to the string type itself is because of the possibility of adding those options the Arri Type Definition specification. This could be similar to how how "minLength", "format": "email", "format": "date-time" are possible to set in OpenAPI.

[hospotho]

I have a better understanding after reading these documents, but I have some new questions now.

For reducing JS bundle size, are we allowing programmers to generate client.g.ts and use it on the client side instead of just calling a.compile()?

// client.g.ts
export type User = {
  id: string;
  name: string;
}

export class UserModel {
  validate(input: unknown): boolean {
    return false; // some validation code
  }
  parse(input: unknown): User | null {
    return null; // some parsing code
  }
  serialize
}

// app.ts
import { UserModel } from './client.g';
const $$user = new UserModel();
const user = $$user.parse(input);

For Arri Type Definition, should we simply add a validation keyword, similar to the metadata keyword?

{
  "properties": {
    "id": {
      "type": "string",
      "validation": [
        {
          "check": "maxLen",
          "arg": 10,
          "fallback": "placeholder1"
        },
        {
          "check": "regex",
          "arg": "^[a-zA-Z0-9]+$",
          "fallback": "placeholder2"
        }
      ]
    }
  }
}

For code generation, since a.compile() has different logic compared to ts/ts-codegen (JIT vs AOT), should we cache anything or build the template before calling a.compile()? If not, it seems we cannot and may not be necessary to apply the builder pattern.

const User = a.object(/* ... */);
console.log(User);
// expected output:
{
  "properties": {
    "id": {
      "type": "string",
      "validation": [
        {
          "check": "maxLen",
          "arg": 10,
          "fallback": "placeholder1"
        }
      ]
    }
  },
  // created before a.compile()
  // symbol will be ignored by JSON.stringify
  [Symbol("arriTemplate")]: {
    "validateTemplate": "",
    "parseTemplate": "",
    "serializeTemplate": ""
  }
}

[joshmossas]

For reducing JS bundle size, are we allowing programmers to generate client.g.ts and use it on the client side instead of just calling a.compile()?

Yes certainly. Although I should note that currently the logic of ts-codegen is not 1-to-1 with the validators created by a.compile. ts-codegen is currently geared specifically for creating Arri RPC clients are thus are more tolerant to schema changes. (Notes on this can be seen here)

I plan to add a strictness toggle for people who want to use it standalone just to get AOT compiled validation, parsing, and serialization functions. (Or maybe it'll get a separate name like instead of typescriptClient generator it's a typescriptType generator. 🤷

That said you can actually get the compiled function body of a.compile like so:

const MySchema = a.object({ foo: a.string() });
 // passing true as the second parameter indicates that you want to include the function body strings
const $$MySchema = a.compile(MySchema, true)

console.log($$MySchema.compiledCode.validate);
console.log($$MySchema.compiledCode.parse);
console.log($$MySchema.compiledCode.coerce);
console.log($$MySchema.compiledCode.serialize);

For Arri Type Definition, should we simply add a validation keyword, similar to the metadata keyword?

I like the idea of that. "validation", "validations", or "refinements" all sound like reasonable keywords to me. We should make a list of what validations we would expect. Another part of me is thinking that maybe it should live under the metadata keyword. Since metadata includes information that may or may not be used by the code generator.

Also I think we should put a pin in implementing support for "default" or "fallback" values. I have to think about the implications that creates for type generation in the RPC clients. For this round let's focus on some of the more common validations that people would want for convenience reasons. (Ex: email, regex, minLen, maxLen, etc)

For code generation, since a.compile() has different logic compared to ts/ts-codegen (JIT vs AOT), should we cache anything or build the template before calling a.compile()? If not, it seems we cannot and may not be necessary to apply the builder pattern.

My initial reaction is "no" because then we'll end up carrying around these potentially large template strings, even if the user never calls a.compile. Or are you planning on caching just the part of the template specific to the additional validations? I.e. if there's an "minLen" validation it just has the string input.length >= {{ minLen }}. That said I think my initial reaction is still no, but I could see that working better if we go that route.