RFC: Type Generation

[dfreeman]

I've been mulling over different approaches to this for a bit, and I'd love to start getting thoughts from the Typed Ember folks and the community in general 🙂

Background

There are a number of cases in a typical Ember app where files other than JS/TS sources produce importable modules in the compiled application. Because TypeScript is unaware of these files, it rejects attempts to import them. An elephant-in-the-room example of this is Handlebars templates, whose import statements are @ts-ignored in our component blueprint to work around this exact issue.

The TypeScript compiler doesn't support any notion of loader plugins or anything like that to directly allow for deriving types from non-JS/TS files, but it does allow for merging 'virtual directories' into a source tree, which is the subject of this RFC.

Proposal

I'd like to propose the following:

1. we introduce a types/generated directory and add it to rootDirs and the appropriate places in paths to allow for both absolute and relative resolution of modules there
2. we create a phase in the build process during which ember-cli-typescript can emit files into types/generated for use by tsc and other TypeScript-aware tooling
3. we provide a mechanism for other addons to provide generated typings as well

For technical details on how these three points might work, see Approaches below.

Possible Applications

Templates

As an immediate win, I expect we would provide out-of-the-box behavior to produce type declarations for any templates found. The simplest approach here would likely be to produce a simple branded default export in order for component files to have something to import, e.g.

// types/generated/addon/templates/my-component.d.ts
import { TemplateFactory } from 'htmlbars-inline-precompile';

declare const template: TemplateFactory;
export default template;

(Note that the 'htmlbars-inline-precompile' module seems to be our source of truth for the type of a template at present...)

config/environment

One non-obvious (or at least it wasn't immediately obvious to me) option we'd get from this is the ability to produce the type declaration for the <app-name>/config/environment module. The contents of this module are generated based on a node file executed at build time, and users are currently responsible for maintaining a one-off declaration file.

Instead, we may be able to inspect the shape of the config object during the build and emit a type declaration for it into the generated types directory.

Addons

A number of addons compile arbitrary files down to JS modules for consumption. By exposing external hooks for providing generated types, we empower the community to enable participation in the type system for any addon that does this kind of compilation.

Two examples that are near and dear to my heart:

• ember-css-modules: This addon provides a JS module for each CSS file which exports the values of any constants defined in that file as well as a mapping from local to mangled names for classes in that file. Generating a type declaration for the shape of that object for each module would be trivial if the hooks to do so were available.
• ember-apollo-client: This addon compiles .graphql files into an AST representation of the query or mutation they contain. Using a tool like apollo-codegen, it could provide the ability for well-typed queries and mutations, using the GraphQL schema to inform required inputs and the type of the returned query result.

Experimentation

Richer Template Types

As mentioned above, I think we'd want to provide simple opaque types for template files out of the box. This opens the door, though, for experimentation outside of ember-cli-typescript itself with different approaches to typing templates, perhaps getting richer guarantees about compatibility between templates, their component implementations, and other templates/helpers available in a project.

Generated Type Registries

Today users need to deal with the boilerplate of adding their container-managed classes to the corresponding registries automatically. With engines and local-lookup, which each introduce their own form of sandboxing for these registries, this management only promises to become more complicated as projects grow and initiatives like Module Unification move forward.

In light of that, we might experiment with generating registry declarations automatically, producing one monolithic generated registry interface for each type that imports and references all discovered implementations of that type.

Approaches

There's one primary technical question that needs to be answered in order to move forward with this: what are the inputs and output of type generation? The fundamental fork in the road I've hit is whether this should all be Broccoli-based or not; there are pros and cons either way.

Broccoli Based

A Broccoli solution for this would likely have the generation hook receive a tree as input and return a tree as output. On the e-c-ts side, we'd then be able to take that output tree and essentially move its contents into place in the generated types folder.

Pros

• Consistent with how most Ember CLI-related things work
• Allows us to use the input trees Ember CLI already generates during a build (guaranteed to have the right files, etc)
• Allows addons access to any other information they might have built up in other Broccoli trees (e.g. for ember-css-modules this would be super useful for reusing parsed CSS)

Cons

• Requires an actual full build to produce types (so for instance we couldn't run type generation as part of ts:precompile)
• Lots of people find Broccoli confusing/underdocumented (though there are efforts to fix that)

Not-Broccoli Based

If we take a non-broccoli approach, we have more flexibility in terms of what we provide as inputs and expect as an output, but I suspect we'd still wind up with some kind of "read from this directory, and write your results over here" approach.

Pros

• We could invoke this as an initial step of ts:precompile, ensuring generated types are always up to date when publishing addons
• FS APIs are generally pretty straightforward and well-understood

Cons

• Actually invoking these hooks outside the context of a running build might make producing types much more difficult for some addons
• We'd have to pretty much just do our best to provide the right inputs (e.g. hope that the app or addon tree actually lives at app/** or addon/** and that the host isn't doing anything weird on the Broccoli side to add files there

Open Questions

• To broccoli or not to broccoli? (In writing this up, I think I've started to lean toward the Broccoli approach, but there are definitely some intricacies we'll need to figure out if we go that route)
• Beyond the broccoli question, what does the actual interface for addons look like?
  • How do they opt into this? Just implement a method? Set a package keyword?
  • Is the hook invoked multiple times, e.g. once for app, once for tests, etc? (This is how ember-cli manages a number of similar APIs)

[chriskrycho]

1. 👏 Thank you for writing this up!

2. I’ll have more thoughts later on the general set of questions.

3. One thing I did want to note:

   Today users need to deal with the boilerplate of adding their container-managed classes to the corresponding registries automatically. With engines and local-lookup, which each introduce their own form of sandboxing for these registries, this management only promises to become more complicated as projects grow and initiatives like Module Unification move forward.

   In light of that, we might experiment with generating registry declarations automatically, producing one monolithic generated registry interface for each type that imports and references all discovered implementations of that type.

   Because of how computed property installation works in Ember 3.1+, it turns we need to actively move people away from the registry pattern for everything but Ember Data.

   It’s still something we should support, assuming we end up supporting “last two LTSes” or something like that, as lots of folks are still going to be in <3.1 and not yet on one or the other of classes and decorators. What’s more, while classes + decorators are viable they’re definitely still not the Ember way to go (even if they kind of are in TS land at this point), and I think we’re broadly agreed that we should support Ember defaults as much as possible, even if we recommend classes and decorators as being better and more reliable as far as the type system goes.

   That doesn’t remove the need for this – at all! – it just means that whatever we do should focus primarily on the Ember Data part of that story specifically IMO. And the reason for that is that the Ember Data lookups just aren’t going away: this.store.findAll('potatoes') is going to be with us for the foreseeable future.

[dfreeman]

Because of how computed property installation works in Ember 3.1+, it turns we need to actively move people away from the registry pattern for everything but Ember Data.

This is because of the field decorators being invisible to the type system? In a rainbows and unicorns world that has the potential to change if/when decorators move forward, right? Just want to make sure I understand the scenario 🙂

[ro0gr]

This all sounds really great!

I expect we would provide out-of-the-box behavior to produce type declarations for any templates found.

Especially this one 🚀

My first impression after a few days of e-c-ts usage is "it works!". Currently the biggest issue I think is a lack of typescript support in templates.

[chriskrycho]

At this point we have solved the “types for templates” problem via Glint, and are intentionally not building anything Ember-specific for other forms of codegen, instead deferring that (appropriately!) to projects like GraphQL Code Generator, native CSS Modules support, etc. Closing this accordingly – thanks for the original thoughts written up here @dfreeman, but even more for solving the most important one of these!