Make LiveScript pluggable and backward compatible to backward incompatible changes

[bartosz-m]

I have proposition for first steps to make LiveScript pluggable and backward compatible to backward incompatible changes.

I will try to explain why LiveScript would benefit from plugins, what already I've done in that matter and what more is to do. It'll be a little longish and although I'm totally sober I'm extremely tired, so bear with me and my whim to express my thought in this peculiar way. (But if You can't then go directly to todo section)

Lets start with some boring stuff to calm some fears:

Terminology:

• core - what you get out of the box by installing LiveScript from main repo.
• plugin - some module that can at runtime extend/modify functionality of the core
• system - core combine with installed plugins

Requirements:

• no backward incompatible modification inside core
• only incremental changes
• plugins are developed outside main repo

As You can see it won't be so scary, no revolution. You can stop holding your breath now.

But why?

Lets lean over crystal ball and see if future holds the answer:

user A wants feature F but it is backward incompatible, so maintainers decide to not implement it in the core. User B has to much free time and implements plugin P0 for F.
A is happy because he can install plugin P0 and use F in his projects, maintainers are happy because core is unpolluted by rarely used functionality, unicorns dance on the rainbows and whole world is a happy place.

Then comes the day when A decides that in order to build the most awesome app in entire universe he will need to use another plugin P1. But installing P1 alongside P0 breaks his system and he is screwed.
Meanwhile unicorns continue to dance, maintainers and rest of LiveScript users continue their work uninterrupted. Poor A B and maintainer of P1 will need to resolve A's problem on their own.

[VISION ENDS]

We can learn from it that plugin system would accomplished five goals:

• ensure core is stable and well tested
• introduce bleeding edge innovations
• increase level of diversity across whole system
• give people freedom to choose their own "LiveScript"
• make unicors really happy

Where?

I like what I've seen it that crystall ball. So I gathered couple of my little frieds one of them was plugin that could convert any function with await inside into async-one (very handy fellow) and together we went on a journey to find that "plugin heaven" from the vision...
Ok.Ok. I'll keep it short.
Our party grew bigger - now we travel with a big dude that puts livescript files in a hat and pulls out of it es6 modules - and after many adventures we finally get to place which we call now home. Its not "plugin heaven" but still we like it and we work only here. Going back to core or making expedition to js wastelands is no an option for us. For future generation of explorer We would like to share a map, that contains usefull landmarks we visited which may help you find "plugin heaven":

• core - LiveScript freshly served from main repo without any modification
• compiler - wrapper for core that allows for augmenting of compiling and lexing process by exposing internal parts to plugins.
• system - configures compiler based on individual project requirements and feeds compiler with files and writes transpiled one to disk. I hoped to extend it in future to become central point for serving LiveScript related services e.g. transpilation, linting, preview, autocompletion (right now this functionality is duplicated across many editor specific extensions)
• plugins - few small and one big module for extending and modify core e.g. es6 module, top level await

How

When I and my little plugin friends set of on a journey to "pluggable heaven", We didn't anticipate how many obstacle would we need to overcome. So here's list of things you better prepare here in core before You venture forth

Todo

First step - making LiveScript plugin friendly:

Expose:

• ast:
  • [ ] Node for creating new kind of nodes
  • [ ] UTILS for custom implementation of utility functions
  • [ ] Scope
• lexer:
  • [ ] tokens
  • [ ] regular expressions
  • [ ] keywords

• [ ] Use properties instead of using modules directly e.g assign lexer to compiler.lexer and use it this.lexer.tokenize some-stuff

Next steps - making LiveScript plugin aware

• [ ] - add hooks/point that multiple plugins could use at same time
• [ ] - load plugins defined in config

Distant future:

• [ ] move as much functionality from core to plugins

Boring summary

• plugin system is beneficial to LiveScript
• it doesn't have to be implemented inside core LiveScript
• only little backward compatible changes would need to be done
• it is doable - I've created something that could be called proof-of-concept and I'm using it for my personal projects

[rhendric]

The LiveScript grammar is non-modular, since it would have to be fully regenerated any time you want to add elements of syntax, and that grammar generator currently isn't even shipped with LiveScript. So I think that would mean you can't add new AST node types, at least types that would be created by syntax. You could, I suppose, monkey-patch existing methods to insert new node types in appropriate places. I don't know how far that would get you with your goals.

In general, I don't think there's anything wrong with unofficial, aftermarket modifications to LS that add custom functionality on a may-break-whenever basis, and to the extent that plugins are simply a formalization (but still officially unsupported) of that, I'm all for them. However, exposing more compiler internals in a supported way would constrain maintainers and therefore need to be thought through carefully, which should include making sure you can actually do what you want if those things are exposed. Exposing the lexer's lists of keywords and token types for extension, for example, doesn't help unless the grammar can recognize those tokens.

I would be interested in seeing your proofs of concept. I'd also be very happy to be proven wrong about my pessimism, for the record; I've been doing a lot of thinking about modular language design, particularly in the context of LiveScript, and my conclusion has been that modularity pretty much needs to be a first-order consideration from the language's beginning to be viable. If you do have a viable way to make LiveScript modular without redoing the whole thing, I'd be delighted. But also very, very surprised.

[vendethiel]

If you do have a viable way to make LiveScript modular without redoing the whole thing, I'd be delighted. But also very, very surprised.

Clearly macros and readtables are the solution! /s ^ Actually I did try to have a go at custom operators a while back. But we have too many valid parses already...

[bartosz-m]

Links for my proof of concept are at the end - you need to have really strong stomach because it is example of truly unholy coding.

What it can already do:

• bring to LiveScript es6 modules import and export, extend them by allowing of importing modules base on glob pattern, and importing inside global scope all symbols from modules
• enable using of await at top scope
• automatically detect async functions

What could it do:

• custom mathematic operators per block
• macros system

How it does it:

• rewriting tokens
• rewriting ast
• introducing new custom Nodes

BUT IT ALL IS A ONE BIG HACK! (see next section)

How does it work

Im not calling LiveScript directly instead I call my system that on every stage of LiveScript compilation adds custom post/preprocessors which are powered by plugins.

1. source preprocessor modifies code e.g. ls-code = "return #ls-code" (actual usage from my current project)
2. code is passed to lexer to produce tokens
3. tokens are rewriten by plugins installed in token rewriter
4. rewriten tokens are returned to livescript to produce ast
5. AST rewriter augments ast and allow installed plugins to rewrite it (plugins can introduce new custom Nodes)
6. rewriten ast is used to generated js code
7. js code is postprocess e.g add "//# sourceURL=...

But why does it work

Example

import modules base on glob pattern (taken from one of unit tests)

import 'modules/**'

LiveScript compiles it but generates this code:

import$(this, 'modules/**');
function import$(obj, src){
  var own = {}.hasOwnProperty;
  for (var key in src) if (own.call(src, key)) obj[key] = src[key];
  return obj;
}

I replace all tokens DECL:import with calls to dummy function with name __static-import__ and fix INDENT tokens.
After that tokens are equivalent to that generated by this code:

__static-import__ 'modules/**'

next I walk ast in search of invocation of __static-import__ and I replace whole subtree with my custom Import nodes based on files matching glob pattern.
What remains to be done is simply compile root node of ast which gives us:

import foo from './modules/foo';
import Math from './modules/Math';
import Vector from './modules/Vector';
(function(){

}).call(this);

Proof of concept

I'm only developer and user of this system so I have no idea if it will work out of box on someone-else's computer. I tried to estimate which part are more stable that other. All are dual hosten on github and gitlab, if there is no link for github repo just look for it on my profile

Plugins (they should work) they can load automatically compiler behind the scene so they can be used just by requiring them

• transform-esm - es modules import & export
• transform-implicit-async - automatic async insertion
• transform-object-create - Object.create as an implementation of clone operator ^^
• transform-top-level-await - Enables use of await in top level block.

Webpack loader (should work) plugin-loader loader for webpack with support for plugins

Compiler (will work if I didn't messed dependecies) compiler - main wrapper used for compilation

Loader (no idea - not using it at the moment)

• livescript-esm-loader - node experimental loader for es6 modules and es6 flavored LiveScript

System (experimental needs previous one)

• livescript-system - automatically loads configuration allowing for usage of different plugins and configurations on per directory basis

[bartosz-m]

Also for me main reason for exposing the lexer's lists of keywords was to help writing code completions like this

Right now I have to copy-paste keyword form lexer.ls. It would be much better if i could write lexer.keyword-all and be sure that it is always up-to-date .

[rhendric]

I haven't yet found the time to do a deep dive into this, but it does look impressive. If you have any specific requests to support this and/or your IDE work (like exposing the lexer's list of keywords), please pitch them individually and I'll be happy to entertain them.