Sketch: allow a variable number of imports/exports of fixed type in Wit

[lukewagner]

Motivation

Let's say I'd like to build a component that consumes 3 configuration values a, b and c (which in a 12-factor app I'd take as 3 environment variables). I could define a component with type:

(component
  (import "get-config" (func (param "key" string) (result (option string))))
  ...
)

However, this loses the fact that my component specifically wants a, b and c. If I'm using or deploying this component, I have to learn about these values from the docs or observe the behavior of get-config at runtime. If I make a mistake, the error will only show up at runtime.

If instead I give my component this type:

(component
  (import "config" (instance
    (export "a" (func (result string)))
    (export "b" (func (result string)))
    (export "c" (func (result string)))
  ))
  ...
)

then there's a lot more declarative information that a host can leverage to provide a better developer experience. E.g., at deployment time, the host can check that there are indeed configuration values a, b and c available and give a deployment-time error if not. There are also new optimizations made available at runtime: a, b and c can be held in a dense array accessed by internally-computed static index, thereby avoiding the hash-table lookups at runtime (which can really matter when the target instantiation-time is in microseconds).

This example is in the domain of configuration, but you can also find analogous examples in:

• metrics (with one (inlineable) (func (param "increment" u32)) import per metric)
• logging (supporting multiple logging backends)
• HTTP upstreams (with one import of wasi:http/outgoing-handler per upstream.
• KV/Blob stores (with one import per distinct storage backend)

In all these cases, the workaround for supporting multiple instances of the same interface inevitably involves using a dynamic string parameter which loses the otherwise-declarative dependency information. (To be clear, some use cases do really need a runtime-dynamic string name; we're talking here about the cases where the string would otherwise be a magic constant in the code.)

So given that we'd like to write components with the above type, how do we capture all these varying types in Wit? Of course we can write the Wit for any single component; for example, a Wit world that supports the above component is:

world my-world {
  import config: interface {
    a: func() -> string
    b: func() -> string
    c: func() -> string
  }
  ...
}

The challenge is writing a single world that captures this component and all the other components, each with their own varying set of configuration values.

The proposal is to allow Wit to express not just a single interface or world, but a family of interfaces/worlds produced by substituting various parameters. Concretely, for the above I'd like to write (roughly; the exact syntax here is open for debate, of course):

world my-world {
  import config: interface {
   *: func() -> string
  }
  ...
}

Using this, it would be natural for WASI standards to use * in standardized interfaces such as:

default interface "wasi:config/values" {
  *: func() -> string
}

which would then allow my-world to be rewritten as:

world my-world {
  import config: "wasi:config/values"
  ...
}

and leverage standard implementations of wasi:config/values.

Sketch

While in general I think we'll want to allow putting parameters everywhere in Wit (types, names, parameters, results, fields, cases, imports, exports, ...), as a first step, I'd like to keep things scoped to just the case I showed above where a * can show up in the name of the lone field of an interface. While this won't be easy (there are a number of interesting producer/consumer design questions to work out), I think it'll be much easier than parameters in types, and so a good starting point. For terminology, I've been calling the general feature "Wit templates", and I think this first milestone could be called "variable imports and exports", but open to hearing alternatives.

From a Wit grammar perspective, the addition is fairly tiny, defining a:

variable-id ::= id | '*'

and then using variable-id in func-item and typedef-item. As an additional validation-time constraint, * would only be allowed inside an interface block as the only item. (Or we could capture this constraint in the grammar; but I think we'll want to loosen this constraint over time.)

To allow encoding Wit documents as .wasm binaries, we'll also need to extend Binary.md to support * in names. One thing to be clear about here is that a concrete component won't be allowed to have any *s in its imports/exports; only Wit documents. In a far-post-MVP future, one could imagine generalizing components with a staged-compilation model that did allow components to talk about * names, so while we don't need to design how that all would work, it would be good to pick an encoding of * that could be retconned into this far-post-MVP future if needed.

Significant design work will be needed per-language-toolchain around how to allow the developer to fill in the *s in the world they are targeting. Riffing on an idea from @dicej and @fibonacci1729, this could come from buildconfig, with each line adding a field. E.g., in the context of cargo component, I could write:

[package.metadata.component.target]
path = "my-world.wit"

[package.metadata.component.parameters]
config.a = "string"
config.b = "string"
config.c = "string"

which would declare 3 imported configuration values a, b and c, looking ahead to a future milestone where non-string types could be allowed. More design work is probably necessary here to think through how to express all the not-so-simple cases. But the idea is that, from this language-specific buildconfig, tooling could derive a language-agnostic substitution which would then be applied to the target world to produce a "monomorphized" world with no *s that is fed into bindings generation, keeping the rest of the build pipeline working like normal.

This is just a sketch, and more work is needed to flesh out the design, but I thought it'd be useful to put up this much now since the interface design questions above are coming up in a number of places at the moment. @dicej and @fibonacci1729 have also done a lot of thinking about this, so I'd invite them to drop in whatever they're thinking too.

[Mossaka]

Nice write-up! The cargo component example really makes sense and makes it easy for developers to express configs. I have a few questions regarding the later part of the write-up in this proposal:

1. Is the scope of star imports restricted to functions without any parameters and single string return type, like *: func() -> string? If not, I'd be interested to understand what *: func(param1, param2, ...) -> (ret1, ret2) would mean for a configuration.
2. The example in cargo component is really nice. I'd be interested to see how other languages would express this in their buildconfig, especially some dyanmic languages like Python and JavaScript.
3. Following up on question 2, I am not entirely sure about what a buildconfig even is. What is a buildconfig for Python or JavaScript? Could a buildconfig be implemented as a language-independent configuration file using YAML. Could it be a ConfigMap in Kubernetes?

[fibonacci1729]

This looks great @lukewagner , thanks for capturing all of this!

+1 to not encoding the rule * would only be allowed inside an interface block as the only item directly into the grammar. Plenty of languages allow more expressiveness per their grammar while enforcing particular constraints during parse.

I might not be making the connection here, but how is the RHS string type of the parameter in Cargo.toml used during the substitution, for instance when substituting in the above *: func()->string?

[lukewagner]

Ah, I think my Cargo.toml example was confusing. Because wasi:config/values as-written has a fixed string result type, there is no right-hand-side necessary. Since TOML doesn't allow empty right-hand-side, maybe I should have written this instead:

[package.metadata.component.target]
path = "my-world.wit"

[package.metadata.component.parameters]
config = [ "a", "b", "c" ]

What I wrote would make sense once type parameters were possible and wasi:config/values was generalized interface { *: func() -> _ }), so I sortof jumped straight there in my example, where I'm substituting string for the _.

I think that answers @fibonacci1729 and partially @Mossaka's question 1. Answering the rest:

1. The * should be usable with any function type (the "string" in the Cargo.toml was a red herring).

2. For JS, I think you'd write the analogous thing but in package.json. I can't speak to the Python though.

3. Yes, great point! I think we could definitely have a language-agnostic TOML (or JSON or ...) file that was fed into wit-bindgen. Maybe this is even our starting point, so it sortof sets up a "reference" format, and then Cargo.toml and package.json are just developer conveniences that avoid one extra config file.

[dicej]

FYI, @fibonacci1729 and I have this pretty much implemented; we'll demo it at tomorrow's Component Model meeting, then clean things up and start opening PRs. I'm sure @alexcrichton will have opinions :)

bytecodealliance/wasm-tools@main...fermyon:wasm-tools:wit-templates
bytecodealliance/wit-bindgen@main...dicej:wit-templates
bytecodealliance/wasmtime@main...dicej:wasmtime:wit-templates

[alexcrichton]

I unfortunately won't be able to make the meeting tomorrow, but I look forward to digging into these bits on Monday!

[dicej]

First draft PR for feedback: bytecodealliance/wasmtime#5925
I anticipate that one will generate the most discussion, since there are a variety of ways host bindings could be generated, and I picked the one that seemed most flexible to me.
@Kylebrown9 FYI

[dicej]

@lukewagner Brian and I ran into what seems to be a pretty fundamental issue while implementing this.

Consider this world:

interface foo {
    *: func() -> u32
}

default world wildcards {
    import imports1: self.foo
    import imports2: self.foo
}

Now imagine we want to build a component that targets that world, using ["a", "b", "c"] to expand imports1's wildcard and ["x", "y", "z"] to expand imports2's wildcard. AFAICT there's currently no way to represent that component type, whether as WIT or in binary -- we can say foo has functions a, b, and c or we can say it has functions x, y, and z, but we can't say both at the same time.

Of course, we could split foo into two separate interfaces, but how would we name them, and how would you do a subtype comparison between the original WIT template and the concretized WIT?