Allow "Compiler Plugins"

[mohsen1]

From wiki:

TypeScript Language Service Plugins ("plugins") are for changing the editing experience only

TypeScript plugins are very limited. Plugins should be able to:

• Apply Transformers
• Provide type definitions
• Override module resolver (LanguageServiceHost#resolveModuleNames)
• Emit code beside TypeScript standard output

[DanielRosenwasser]

@rbuckton and I have some offhand thoughts about this

1. The scope of this is fairly broad - does this include generating new nodes for type-checking? If so, that makes this a much larger item.
2. When it comes to module resolution, there are quite a few subtleties.
   1. Custom module resolution also means providing some special module resolution "host" which can provide this behavior in the editing scenarios.
   2. It's not clear how custom module resolution works between dependencies & dependents.

In general this isn't simple but we're open to at least hearing ideas.

[mohsen1]

I'm not familiar with TypeScript well enough to write a proposal. Instead I can list a few plugins that can be useful and exist in the wild in other forms (Webpack plugin, Babel transforms) to make a case for having such extensibility:

Custom module resolution

• Typed CSS Modules
• JSON-loader
• YAML-loader
• Loading any type of serializable file with types

Code transformers

• babel-plugin-jest-hoist
  • issue
• Minifiers
• Removing debug code

Emitting other code

• A Swagger code generator embedded in TypeScript plugin
  • gist about this idea
• Functionality similar to Webpack: emitting image and other asset files

Providing types

• Typed GraphQL query responses
  • This applies to pretty much any database query language. Result of a query can be typed based on query

There are so many other use-cases for compiler plugins that I'm not aware of but I'm sure compiler plugins will make TypeScript ecosystem thrive.

[phra]

i think that this can be a very powerful feature. in particular i'm interested in points 1, 2 and 4 at the moment.

[ryanelian]

Big yes to this feature being supported by TypeScript.

I propose that the plugin system should be implemented using streaming pattern.

// NodeJS stuffs
import * as stream from 'stream';
import * as path from 'path';

export enum TypeScriptModuleEnum {
    CommonJS, AMD, System, UMD, ES6
}

export interface ITypeScriptTransform {
    (filename: string, module: TypeScriptModuleEnum): stream.Transform
}

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// As sample plugin to transform file content into string if matches extensions. Handy for templates.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

// Plugin creators can use this to extend TypeScript!
import * as through2 from 'through2';

export interface IStringifyOptions {
    extensions: string[]
}

function doStringify(filename: string, extensions: string[]) {
    return extensions.includes(path.extname(filename));
}

export function StringTransform(options: IStringifyOptions): ITypeScriptTransform {
    return function (filename, module) {
        return through2(function (file, encoding, next) {

            // Determines whether we should stringify the file.
            // For example, the file name = 'test.txt' and extensions list = ['.txt', '.html']
            if (!doStringify(filename, options.extensions)) {
                return next(null, file);
            }

            let s = JSON.stringify(file);

            if (module === TypeScriptModuleEnum.CommonJS) {
                s = 'module.exports = ' + s + ';\n';
                return next(null, s);
            } else {
                return next({
                    message: 'Module not supported!'
                });
            }
        });
    };
}

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Later...
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

import * as ts from 'typescript';
import { StringTransform } from 'StringTransform';

let tsString = StringTransform({
    extensions: ['.html', '.txt']
});

ts.useTransforms([tsString]);

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Later in application source code...
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

import s = require('./test.txt');

This system has the following benefits:

• The vast majority of browserify and gulp plugins (which are mostly written using through2) can be ported to TypeScript easily!
• Transforms can be executed sequentially by TypeScript during compilation, in-memory, using the order defined in the passed transform array.
• Allows file manipulation freedom by plugin developers.

[cameron-martin]

Something similar to browserify transforms or webpack loaders would be very powerful and cover most of these use cases.

[phra]

looking forward to see this feature implemented! :raised_hands:

[data-ux]

Currently program.emit() already has customTransformers as possible parameter, but it isn't exposed to consumers of the tsc command line program. It would be great to be able to give transformers in compilerOptions as proposed in #14419. Currently to use customTransformers you have to use the Compiler API and re-implement all the functionality in tsc like watching files etc.

[mohsen1]

Transformers do not allow custom module resolution or extra file emit

[data-ux]

@mohsen1 Yes, you're right. I was suggesting it as an approach for your first point "Apply transformers". For a plugin to do all the things suggested in your issue description is too broad of a scope, as @DanielRosenwasser noted. I think exposing custom transformers is the highest value feature of the suggested and it's also the most simple to implement taking into account the way the TS compiler currently works.

[WiseBird]

@Jack-Works Isn't Language Service doing it already?

[geovanisouza92]

Any news on this? I'm writing a plugin, and would like to just plug it on my current setup (tsc & webpack + awesome-typescript-loader).

[fis-cz]

I am sorry. I didn't know it is possible to write language service plugins... I should read before I write. So, for the compiler it can be implemented in the similar way as for ls.

Regarding this topic I have some tips.

It would be great if loading of plugins is configurable through the tsconfig.json file. Thats because i.e. VS code syntax highlighter / lens / intellisense will use the same plugins as the compiler during the regular build. Plugin can be standard Node module and can be resolved in the standard CommonJS way.

When the plugin for the typescript compiler will be defined in the tsconfig.json file it should be loaded during the tsc startup and tsc should provide access to all currently available tsc APIs (it would be also great if extended tsconfig can be read through API too as when I was playing around the API about year ago i had to write custom config reader / extender, what is not good as with next release of ts you can remove or add some options and its hard to maintain the code afterwards ;).

During the init phase, the plugin can replace various stages of the compiler API with custom implementations (such as file reader) or bind event listeners to events occurring during the compilation process (file load, resolve, parse, transform, emit...). Events with "pre" and "post" would be also great in order to pre-process or post-process the stage data while original components are still in use. I.e. preParse is great time to run text preprocessor which can implement #ifdefs and replace them with empty lines to keep it possible to generate source maps properly, or postParse when AST can be searched for dead code and the dead code can be removed from furthermore processing.

From my perspective, it would be much easier to implement call puigin.init(...) with references to all available tsc components and let plugin developer to choose if he will replace it or not what would be specified in the return object. I'll update this later once I'll check tsc sources.

If this would be possible we can simply use various plugins for code preprocessing, death code elimination, output minification or whatever else we can imagine directly under the hood of the compiler "executable" but without touching the compiler code itself. Currently, we have to write everything as a new compiler using the tsc API. Unfortunately, this later means we have to implement the "new" compiler to our development tools (such as VS code or full VS, what is almost impossible ;).

[pcan]

Events with "pre" and "post" would be also great in order to pre-process or post-process stage data (i.e. preParse is great time to run text preprocessor which can implement #ifdefs and replace them with empty lines to keep it possible to generate source maps properly, or postParse when AST can be searched for dead code and remove it from the furthermore processing.)

I would really need this one. The ideal spot for synthetic code injection is after the parsing phase: here you have the AST ready, you can do some enhancements, and they are already available to language service!

[fis-cz]

I would really need this one. The ideal spot for synthetic code injection is after the parsing phase: here you have the AST ready, you can do some enhancements, and they are already available to language service!

I think it is enough if you can replace the parser with custom one and inside of it you would do your pre, call original parser with modified input and your post where you would modify the AST.

[dmitry-agapov]

Please make it happen. There is just too much need in this. Whenever i find myself in need of extending the typescript compiler with some feature (typed css modules, #10866) i always think "This is definetly can/must be done with a plugin". Babel has a plugin system, and people have an opportunity to do anything they need to do to get a work done, without bothering the core team with requests of new features. Besides, all burden of support lies on the plugin author. I mean that the community can add a ton of requested features to the typescript without bloating the core codebase (which will make it hard to support).

[xtuc]

Now that Babel 7 has the support for TypeScript, would it be possible to achieve that trough Babel?

[krzkaczor]

@xtuc i think this would break all existing tooling etc.

Personally i belive that type providers should be a way to extend typescript (only type system).

[mrmckeb]

I've been working on this plugin, and I would love that it would work during build - and not just when working with files in the IDE.

https://github.com/mrmckeb/typescript-plugin-css-modules

[danielkcz]

I would like to invite everyone to the discussion I've started today that tries to find a way how to make a universal plugin/transform that would work in a similar way as babel-plugin-macros, ideally alongside it to avoid the need for multiple plugins that are hard to configure and use.

Please join and bring your ideas and experience.

[ghost]

My usecase for a plugin, would involve emitting a json file with reflection-like metadata for the API in a SDK description in typescript. Right now we need to manually maintain a json file AND a typescript file for our API. All the information in the json file can be derived from typescript. With a plugin, would like to be able to write code that emitted the json file automatically.

[3cL1p5e7]

I want normal function overloading as C# :D

[justinfagnani]

@DanielRosenwasser is there any chance of starting small and just running existing language service plugins and emitting their additions warnings and errors?

Right now we have a really unfortunate situation where a nice plugin can offer additional type checking during editing, but not fail a build if there are errors.

[DanielRosenwasser]

@justinfagnani I personally think that's the place that LS plugins need to move to over time - it definitely feels strange to have language service errors that aren't build errors.

[twop]

@orta suggested to cross link my tweet: https://twitter.com/twopSK/status/1194117666052567043

I would love to create my own subset of ts but with smaller api surface area and ideally have a custom extension: example *.tss. Things I want to remove: fetch, dom etc.

Context: want to build custom DSL for defining css styles in ts.

Example:

// my_style.tss
export const myStyle = defineStyle({
  backColor: "red"
});

// this function is available in *.tss files
declare function defineStyle(s: StyleDef): string;

// some APIs are not allowed
export const noFetch = fetch(...); // TSError, fetch is not defined

and then seamlessly import it in ts(x) files.

import { myStyle } from "./my_style.tss";
console.log(myStyle); // ASJKH$#Q42  <--- some value produced by 'defineStyle'

Essentially I would love to be able to build a custom ts dialect akin to tsx but with no extra syntax.

code snippet: https://codesandbox.io/s/sketch-of-typed-css-5hlyn

Thanks!

[Peeja]

Similarly, this would enable the type checking of MDX files, which is important for folks using the newer Docs features of Storybook.

[fis-cz]

Just for those who need it I wrote simple AST preprocessor for conditional compilation. Before plugins are implemented it must be run as a complete separate compiler.

See #33800 for details.

[axetroy]

Hi.

All operations of the plugin are now synchronized

I want to know if the Typescript team plans to support asynchronous operations

Eg.

interface LanguageServiceHost extends ModuleSpecifierResolutionHost {
-  getScriptFileNames(): string[]
+. getScriptFileNames(): string[] | Promise<string[]>
}

All operations of the plugin are synchronized

It is possible that some plugins will block the entire Typescript Service until make it crash

[nadavwix]

would love to be able to create plugins that transform non typescript files into d.ts files. these can be used while type checking and will provide a better work experience with non js/ts files.

let all our css files be typed

[glen-84]

@DanielRosenwasser,

The TypeScript 3.8 Iteration Plan included "Plugin Investigation" under the Expected Work Items – was any progress made in that regard?

Will this issue be considered for the next roadmap (July - December 2020)?

[oychao]

Here I need to transform some IDL files into type definition files, by creating a custom module loader (if available maybe)? It's January 2021 now, any progress?