Pluggable loader support for TypeScript imports

[perezd]

Objective: I'd love for Deno to provide a builtin solution to pluggable processing at build time. This would make it easier for tools like Protocol Buffers to integrate more seamlessly with the deno command line tooling without requiring additional steps.

I'm formulating in my mind a more formalized proposal but I wanted to brainstorm a bit with the community to see how this could evolve with more input. Here's some unorganized thinking I've done:

Registering Loaders

I think the best way to do this is using flags, for example: deno run --loaders=path/to/loader1.ts,path/to/loader2.ts

I thought about alternatives here, such as a config file, or a special .loaders folder but there's really no prior art to justify it, so it probably should be expressed as a flag.

loader capabilities should be constrained by the declared permissions specified in the command for the intended entrypoint file.

It didn't seem feasible to have a runtime op that handled registration because I believe that would be evaluated too late in the process.

Loader Definitions

The model that esbuild uses for plugins could be adopted nearly identically, with a slight modification that I'd want to require a loader file to have a single default export which represents the loader that should be installed. Something like this:

export default function Env(build: BuildContext) {
  // Intercept import paths called "env" so esbuild doesn't attempt
  // to map them to a file system location. Tag them with the "env-ns"
  // namespace to reserve them for this plugin.
  build.onResolve({ filter: /^env$/ }, (args) => ({
    path: args.path,
    namespace: "env-ns",
  }));

  // Load paths tagged with the "env-ns" namespace and behave as if
  // they point to a JSON file containing the environment variables.
  build.onLoad({ filter: /.*/, namespace: "env-ns" }, () => ({
    contents: JSON.stringify(Deno.env.toObject()),
    loader: "json", // some other loader defined elsewhere.
  }));
}

And its usage would be:

import { PATH } from 'env'
console.log(`PATH is ${PATH}`)

Loader Execution

Loaders could be spawned off as separate WebWorker instances (or a single instance with all the loaders installed). There's at least 2 modes that need to be supported 1) The initial static resolving/loading of imports before execution begins, and 2) any dynamic calls to import().

Any other thoughts?

[kitsonk]

See #1739 and https://deno.land/manual@v1.11.0/typescript/faqs#why-don#39t-you-support-language-service-plugins-or-transformer-plugins.

Above that, I think we would only support what can be done with [import assertions(https://github.com/tc39/proposal-import-assertions).

Now, having some sort of loader hooks for Deno.emit() in the future though is very likely.

[perezd]

Interesting, could you go into a bit of detail on how you'd imagine using Deno.emit() in the Protocol Buffers compilation scenario? If that's a viable path I'd be happy to explore it.

Also, totally agree that something like this could be a slippery slope to incompatibilities with import syntax, so I appreciate wanting to say no to something like this for purity reasons.

[justinmchase]

What is Deno.emit() doing?

[perezd]

I guess another community ecosystem approach that could be taken would be similar to this: https://github.com/cevek/ttypescript

However I really really dislike this pattern.