Adding a `use` directive to `*.wit`

[alexcrichton]

Currently as specified the *.wit format doesn't have a use directive or any similar ability to import/share definitions from other files. A historical implementation of this existed in wit-bindgen but was removed since it was never really fully fleshed out and working as one might expect. I wanted to open this issue, though, to serve as a place to talk about it.

@lukewagner, @fibonacci1729, and I just had a long chat about a possible design here and I wanted to additionally write down our current thinking for what this might look like. The current plan is for @fibonacci1729 and I to pair up and implement this both in wit-parser in the wasm-tools repository as well as through a PR here describing the semantics.

Interfaces and use

interface http-types {
  resource request
  resource response
  type headers = list<tuple<string, string>>
  record error {
    kind: error-kind
    message: string
  }
  enum error-kind {
    too-sunny-outside,
    too-cold-outside,
    too-many-cats,
  }
}

interface http-backend {
  use { request, response, error } from http-types

  fetch: func(r: request) -> result<response, error>
}

world my-world {
  import backend1: http-backend
  import backend2: http-backend
}

Here the http-backend interface has a use directive which refers to types defined in the previous interface, http-types. The ordering of the interfaces doesn't matter here so you could also do:

interface http-backend {
  use { request, response, error } from http-types

  // ...
}

interface http-types {
  // ...
}

although use statements do not support cyclic references, they must form a DAG still. The use directive can, for now, only appear in interface and world blocks, not the top-level. This may be added at a future date, however. Names can also be renamed in a use:

interface foo {
    use { bar as different-name } from other-interface 
}

Additionally anything use'd is considered an export of the interface which means that use-s can be chained together:

interface i1 {
  type a = u32
}

interface i2 {
  use { a } from i1
}

interface i3 {
  use { a } from i2
}

This will also enable splitting one *.wit file into multiple *.wit files:

// types.wit
default interface types {
  enum errno {
    // ...
  }
}

// fs.wit
interface fs {
  use { errno } from "types"

  exists: func(path: string) -> result<bool, errno>
}

Here the from target is quoted to represent any filename being loaded from. The precise contents of the string are sort of nebulous for now and are eventually intended to be URL-like but for now will just be path names for local resolution. Note the usage of default interface in types.wit which indicates that when "types" is used from it knows which interface to pick by default. A non-default interface can be selected with:

use { errno } from some-non-default-interface in "types"

For now only types can be referred to by use. Eventually it might make sense to add functions as well but that's seen as a possible future feature.

Worlds and use

World directives can be specified with strings now to refer to other files:

world foo {
    import wasi-fs: "wasi-fs" // requires `wasi-fs.wit` to exist
}

and this is equivalent to:

world foo {
    use { wasi-fs } from "wasi-fs"
    import wasi-fs: wasi-fs
}

In a world its use statements must refer to interfaces as opposed to types. Unlike interfaces they're not imported/export but rather are just available for use in import or export directives.

Translation to a component type

The intention is that a use statement basically equates to an alias in a component. For example:

interface types {
  enum errno {
    i-am-a-teapot,
  }
}

interface thread {
  use { errno } from types
  sleep: func(dur: u64) -> result<_, errno>
}

world my-world {
  import thread: thread
}

would translate to:

(component $C
  (import "types" (instance $types
    (type $errno (enum "i-am-a-teapot"))
    (export "errno" (type (eq $errno)))
  ))
  (alias export $types "errno" (type $errno))

  (import "thread" (instance $thread
    (alias outer $C $errno (type $errno))
    (export "errno" (type (eq $errno)))
    (export "sleep" (func (param "dur" u64) (result (result (error $errno)))))
  ))
)

Note here that my-world only imported thread but ended up also importing types in the final component. This is due to the use of the types interface in thread. Note that the string "types" comes from the name of the interface types. For cases where conflicts arise the import strings must be explicitly disambiguated:

// dir1/foo.wit
default interface foo {
    use { /* ... */ } from "shared"
}

// dir1/shared.wit
default interface shared {
    // ...
}

// dir2/bar.wit
default interface bar {
    use { /* ... */ } from "shared"
}

// dir2/shared.wit
default interface shared {
    // ...
}

// world.wit
world world-with-bug {
  // This generates an error since it requires `import "shared"` in the component for two 
  // different interfaces
  import foo: "dir1/foo"
  import bar: "dir2/bar"
}

world world-that-works {
  // explicitly disambiguate here and this works ok
  import shared1: "dir1/shared"
  import shared2: "dir2/shared"
  import foo: "dir1/foo"
  import bar: "dir2/bar"
}

Resolution

Resolution of use statements is intended to follow a rough algorithm that looks like:

1. First resolve all quoted use statements and interface dependencies in worlds. This will involve reading external files, recursively, and collecting everything together. Note that at this point it should be possible to emit another *.wit file equivalent to the first. Note that cycles are disallowed here
2. Next resolve all use statements now that all interfaces are known. Cycles are again disallowed here and connections are made.
3. Finally resolve each interface individually, disalling name conflicts within an interface. For now everything in an interface` will share a namespace, meaning types, functions, and interfaces all live in the same namespace and can have no duplicates.

The final resulting *.wit data structure still semantically has use statements inside of it. This is useful for generators such as:

• For *.wasm generation a use means "go alias something from somewhere else"
• For code generators printing a type definition for exported types a use means "skip this, import the name from somewhere else"
• For code generators printing the name of a type, a use means "print the name as used into this module"
• For lifting/lowering use is "unwrapped" to look at the structure of the type to determine how to lift and lower

This should provide a straightforward way for, even in code generators, to share types between interfaces since all that's necessary is to use between them.

[lukewagner]

Great job writing up the whole story here! This all looks right to me with two nits:

In the first my-world, assuming that the interface name http-backend ends up implying (in a manner that we haven't fully defined yet) a URL, this URL would go into the component import externname as the optional URL string and since URLs have to be unique when present (even if the kebab-names are different), it would be a validation error. To avoid this error when you want to write roughly the world as you've written it, we'd need a way to import/export an interface without a URL. One direct way is to allow inline interfaces (in the same manner as inline function types as in this comment), e.g., world w { import foo: interface { ... } }. (This is independent of the point of the example, of course, which is to show two imported interfaces sharing a common used interface.)

In the "Worlds and use" section, I'd think that the former world would desugar directly into:

interface wasi-fs {
  ...
}
world foo {
  import wasi-fs: wasi-fs
}

so that we don't have to have any special meaning of use in world context. The addition of inline import/export types opens some new questions around how a world pulls in externally-defined names to use in inline function types, but I think these questions are separable and compatible with everything written above.

[alexcrichton]

Ah yes good point I haven't internalized URLs much, but makes sense.

Otherwise though good point on use in worlds, and sounds like we don't need it at all in that case!

[alexcrichton]

Ok I've attempted to formalize this, plus some other items, in #141