Updating the JavaScript dialect used by the TiddlyWiki core

[Jermolene]

Since it was started, TiddlyWiki 5 has targeted the JavaScript language as it was defined in (roughly) 2011.

The advantage of doing so is that it enables TW5 to run on a wide variety of devices, including older and obsolete devices. For example, in the issue that prompted these thoughts, @cdruan mentions using TW5 on an iPad 3 running iOS 9.3.5. That model was launched in March 2012 and discontinued very soon after in October 2012. It runs an operating system that was released in August 2016. It is both impressive and useful that TW5 can run on devices as old as this.

Set against those advantages, there are now quite a few disadvantages that arise from not being able to use newer JavaScript features in the core:

• We are locked out of some potential performance improvements that rely on newer language features. For example #7297 introduces a worthwhile optimisation for the HTML parser that is only available in browsers released since about 2017
• Core code cannot take advantage of new language features that would make the code shorter and easier to understand (for example, arrow functions and templated string literals)
• Generally, it makes contributing to TW5 much harder for contemporary JS developers, because they are not able to use the language features that they are used to

I think that this is a good moment to consider whether it is time to move forwards to a newer dialect of JavaScript.

As a straw man, I propose that we:

• Adopt the policy of targeting the dialect of JavaScript that was supported by the major browsers five years ago. This would be a rolling policy, so each year we'd automatically start using the new features that came in five years before
  • In practice, at this point in time, we would be mostly targeting ECMAScript 2018 as our new baseline. That gives developers modern features like async/await, the spread operator, asynchronous iteration
• Make this switch with v5.3.0. As things stand, despite the version bump, v5.3.0 is actually 100% backwards compatible because the major component planned for it is #6666 which does not break backwards compatibility
• We wouldn't attempt to update all the core code in one go to use more modern JS constructions. In v5.3.0 we would make changes that had a significant performance or readability benefit (for example, using the "sticky" flag with regular expressions), and gradually address more opportunities for optimisation in subsequent releases
• After the release of v5.3.0 we might consider continuing to issue bug fix updates to the v5.2.x branch for a year (or as long as we have the volunteers to manage the process)

The proposal above means that v5.3.0 would still be backwards compatible from an end users perspective; as with v5.2.0, the backwards incompatibility would just be a matter of the internal implementation. We might consider going further, and making some breaking changes that do affect users. For example:

• Switching off CamelCase linking by default
• Removing automatic list de-duplication (see #3790 for an experiment from 2019 that suggests that the impact on backwards compatibility would be limited to some well defined scenarios)

[pmario]

Related to:

• #5192
  • Especially this comment

  • Dropping IE support in the Getting Started tiddler, as proposed in the OP, is very straightforward and I think could be done straightaway.

  • Interesting read mentioned: The baseline for web development in 2022

• #2278

ES 2018 compatibility table:

• https://compat-table.github.io/compat-table/es2016plus/
  • I did find this project interesting: https://github.com/lebab/lebab

Slightly related:

• #5624

[pmario]

I would love to go to ES2020 because of this single feature: optional chaining operator (?.)

https://caniuse.com/mdn-javascript_operators_optional_chaining

[linonetwo]

major browsers five years ago

If we introduce a bundler or compiler, we can use https://github.com/browserslist/browserslist to programmatically define the target, like https://github.com/tiddly-gittly/slate-write/blob/b2f67a124fb36282ef4282087a2f9635bea299f6/esbuild.config.mjs#L33

I would love to go to ES2020 because of this single feature: optional chaining operator (?.)

That is relatively new. I already use these features in the plugin a lot, for example, https://github.com/tiddly-gittly/super-tag/blob/de84647b73d8d85752d34ba6dda480e7a98d3e82/src/widgets/supertag-form/index.ts#L38-L42

But I use Typescript + Esbuild for my code, so it is easy for me to setup these features and compile them to ES5. I can help setup TS+Esbuild if you would like this to happen (AnyScript first, so we don't need to add any typing at first, I can merge our https://github.com/tiddly-gittly/TW5-Typed to the core later.).

[linonetwo]

A benefit of using browserlist can be automatically release multiple version. We can set target to last 10 years, and build an empty.legacy.html for users using old browsers.

[Jermolene]

Hi @linonetwo I always argue against minification or other obfuscation of the core's JavaScript modules because it makes it harder to debug the code with browser developer tools. The usual mitigation is to use source maps, which is not convenient with our single file architecture.

I am also keen to avoid the proliferation of variants of the TiddlyWiki core. A lot of the value of TiddlyWiki comes from the fact that we all use the same TiddlyWiki. Splintering into different, slightly incompatible variants makes it harder for us to provide support, and complicates the documentation. Put another way, the TiddlyWiki core is designed to be universal.

[pmario]

Tiddlywiki already "compiles" itself. eg: We use tiddlywiki .edition/x --build index, which technically is a compilation step.

TW does not use any transpiler or any other packaging system, for the reason Jeremy mentioned. I think tools like babel and webpack where great to develop new JavaScript language features and when JS itself did not have a module system. .. But none of that is needed anymore. ... JS has a module system now, which is already usable by all browsers and TW bundles itself.

I also think that a second compilation step makes building TW much slower. At the moment TW test-server starts more or less instantly. With a building step this will probably need several seconds, which is a big waste of time in my book.

And the biggest disadvantage I see, is dependencies and complexity. At the moment we use npm install tiddlywiki and that's it. Installing transpilers and bundlers will add several 100000 lines of new code, where none of them is known to tiddlywiki users. Keeping those dependencies up to date and secure is a full time job. ... But why .. We don't really need it.

If we do want to explore "transpiling" the TW rendering engine we should have a closer look at bytecodealliance projects which targets WebAssembly components that can run on the server and on the client.

Just some thoughts.

[abesamma]

Great to see this is being considered. We should try to target ES2020 as much as possible whilst setting out developer guidelines/tooling to enable progressive enhancement.

Regarding transpiling, it's quite easy for a developer to set up for themselves a transpiling pipeline using just npm as a build tool/other build tools, and a transpiler of their choice. No need to add transpiling tooling to TiddlyWiki's codebase.

On Mon, Mar 13, 2023, 10:16 Mario Pietsch @.***> wrote:

Tiddlywiki already "compiles" itself. eg: We use tiddlywiki .edition/x --build index, which technically is a compilation step.

TW does not use any transpiler or any other packaging system, for the reason Jeremy mentioned. I think tools like babel and webpack where great to develop new JavaScript language features and when JS itself did not have a module system. .. But none of that is needed anymore. ... JS has a module system now, which is already usable by all browsers https://caniuse.com/es6-module and TW bundles itself.

I also think that a second compilation step makes building TW much slower. At the moment TW test-server starts more or less instantly. With a building step this will probably need several seconds, which is a big waste of time in my book.

And the biggest disadvantage I see, is dependencies and complexity. At the moment we use npm install tiddlywiki and that's it. Installing transpilers and bundlers will add several 100000 lines of new code, where none of them is known to tiddlywiki users. Keeping those dependencies up to date and secure is a full time job. ... But why .. We don't really need it.

If we do want to explore "transpiling" the TW rendering engine we should have a closer look at bytecodealliance projects https://github.com/bytecodealliance/componentize-js which targets WebAssembly components that can run on the server and on the client.

Just some thoughts.

— Reply to this email directly, view it on GitHub https://github.com/Jermolene/TiddlyWiki5/issues/7353#issuecomment-1465864055, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB6NVYXVRECFTG4OTA3BRODW33XXVANCNFSM6AAAAAAVXMNMWQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[Jermolene]

I think that targeting ES2020 would be too restrictive; it is only supported by browsers released since 2020, which doesn't give us much headroom. I'd welcome some robust statistics (as far as I can tell, the stats on caniuse are for accesses to caniuse.com itself which does not seem representative).

My suggestion above was that we should target browsers that are 5 years old, but I think that may need some refinement. In particular, I get the impression that Chrome and Edge have very good adoption of upgrades, with the majority of users on the latest version. That's not true for Safari, where versions of iOS/Mac that are too old to get updates don't get Safari updates either.

With this update we're going to have to address the fact that tiddlywiki.com will not work at all for a much larger percentage of users. We need to try to make it fail gracefully, giving the user enough information so that they understand the problem.

[linonetwo]

You can use website https://browsersl.ist/# with https://github.com/browserslist/caniuse-lite/tree/main/data/features to do statistics.

[Jermolene]

You can use website https://browsersl.ist/# with https://github.com/browserslist/caniuse-lite/tree/main/data/features to do statistics.

I think those are the same statistics shown on https://caniuse.com/ – the problem is that these stats are taken from visitors to caniuse.com which I don't think is remotely representative of our users. It would be better to have stats for wikipedia, for example.

[linonetwo]

As https://caniuse.com/usage-table said, the browser-version vs usage-statistic data is from https://gs.statcounter.com/ . And caniuse only match browser-version with feature-names.

[pmario]

I did find the article some time ago and think it's also relevant. It tackles the problem from several different viewpoints. The baseline for web development in 2022

• Browser Market Share
• Mobile Browser Market Share
• CPU Performance Differences between "flagship phones" and "mid range phones"
  • where the difference is 3 fold :/
• Browser Alignment to Web Standards
  • Luckily browser interoperability was greatly improved in 2022 and will be in the future
• 3G, 4G and 5G Network Availability
• WebSite Size .. Content vs Code vs CSS vs Media

One of the conclusions of the article written in Jan. 2022, related to this thread was:

• Build for modern browsers: Drop IE support and set your build target to ES 2017 or even 2018. Just doing that might get you bundle size improvements of up to 20%.

Which is pretty much in alignment what Jeremy suggests. This table overview will make it possible to lookup JS features and where they belong to.

[buggyj]

Here is a table for node.js javascript version compatibility"

https://node.green/

[rmunn]

#7677 is one issue that could be easily fixed if we used ES2020 as a baseline supported Javascript version. There are lots of places in the code that convert undefined to a default value of the right type with expressions like value || "" or value || []. If value happens to be the number 0, then it gets converted to a string or an empty list, which is often not what was desired. Using the ?? operator, introduced in ES2020, would have allowed those expressions to be value ?? "" or value || [], which would not have converted the number 0. Without ES2020, we would have to write multiple lines of code:

// Old
var value = someFunction() || "";

// New
var tmp = someFunction();
var value = tmp == null ? "" : tmp;

(Note the == null to catch both null and undefined, because that's exactly what ?? does).

[CrossEye]

This is showing up in the new Planning for v5.4.0 project plan. But it's two years later, and that rolling window would now imply the ES2020 that people were hoping for above. To me, that is the also the most recent version that has compelling changes I find difficult to live without.

Would we anticipate simply relaxing the restrictions on newer constructs, or would there be active attempts to replace certain older patterns with newer ones?

[pmario]

Would we anticipate simply relaxing the restrictions on newer constructs, or would there be active attempts to replace certain older patterns with newer ones?

I think it would be an ongoing process. Actively rewriting code, will add issues, we do have no tests for. So for actively new changed code there should be new tests too.

Our code test-coverage at the moment is not good enough to deal with problems, that could be introduced using new coding style globally.

---

There is one construction, which I personally would like to replace, as soon I see it.

• optional chaining operator (?.) .. and may be
• nullish coalescing operator (??)

Eg: if ($tw && $tw.wiki) or in general if (object && object.object && object.object.parameter) -> Replaced with if (object?object?.parameter)

Especially for constructions, that are much more complex. IMO it would improve code readability.

[linonetwo]

Our code test-coverage at the moment is not good enough

What do you mean, I thought it is pretty well, at least unit test.

For e2e test, this is true. Seems I'm the second one use it: https://github.com/TiddlyWiki/TiddlyWiki5/pull/7744/files#diff-28bbbc476ada69d296385c9364b1d94425973bcd9d5dfa9005ebca07387c4d4b

[CrossEye]

@pmario:

There is one construction, which I personally would like to replace, as soon I see it.

optional chaining operator (?.) .. and may be nullish coalescing operator (??)

For me, it's probably the Array.prototype methods such as .map, .filter, .find, .every, .some, .flatMap, and .reduce in place of the for/forEach constructs wherever possible.

But I do also like the optional chaining and nullish coalescing operators. And arrow functions. And template literals. And Object.fromEntries. And ...rest/spread. And... and... and

[sukima]

Class syntax, imports, web streams, iterators, generators, destructuring, abort signals… yes, Yes, and YES!!!! 😍

And 🤞 one day even Explicit Resource Management 💓

[linonetwo]

Nowadays, more new users use mobile phone than PC, many post-00 era human don't even own a PC. And mobile browser's ES support is usually very good. So the browser list's PC part isn't that important.

But I don't really care the syntax now, because AI could follow the existing syntax well. Even ES5 syntax is verbose and not pretty enough, if a human don't need to type it manually then it is fine. Just like I use TS to code, and transpile to ES5 so I don't need to feel bad about that the final result is ES5.

[Jermolene]

I think that we still haven't made a final decision on the target ECMAScript version for v5.4.0.

On balance, I think we should adopt ECMAScript 2017 (also known as ECMAScript 8). That would give us some of the most attractive new features: async/await, classes, modules, and significant syntactic sugar.

I remain concerned about throwing away our deep backwards compatibility, but the concern is mitigated to some extent by the continued availability of v5.3.7.

[pmario]

I did comment about the dialect at: https://github.com/TiddlyWiki/TiddlyWiki5/issues/9065#issuecomment-2983963291

Note that with v5.4.0 we are adopting ECMAScript 2017 as our base JavaScript version. PRs for v5.4.0 are encouraged to adopt the new features that are now available.

There is an overview table, which lists all the newly implemented functions for the different ES versions.

• https://compat-table.github.io/compat-table/es2016plus/

IMO ES2016 and ES2017 do not contain much, that would really improve our code.

I think ES2018 and ES2020 contain 2 features that would really improve our core code.

• ES2018 introduced object rest/spread properties and
• ES2020 introduced optional chaining operator (?.)

[Jermolene]

I think ES2018 and ES2020 contain 2 features that would really improve our core code.

I don't deny that these are useful features, but I think that is balanced by the benefits of significantly deeper backwards compatibility.

[Arlen22]

IMO ES2016 and ES2017 do not contain much, that would really improve our code.

Yeah, there's not a lot there. The ones that stand out to me are:

• async functions

• array.includes
• Object.entries

I can't believe it that we don't have object spread operators. That's such a bummer, but it's just the equivalent of Object.assign so it's actually not a big loss.

[pmario]

I don't deny that these are useful features, but I think that is balanced by the benefits of significantly deeper backwards compatibility.

OK. Then ES2017 it is.

[saqimtiaz]

We should have a minimum version requirement for node.js which will dictate what JavaScript dialect can be used by server only code.

[linonetwo]

This is cool, recommended LTS is node v22. But it will be hard to set eslint rule to check it. We can't tell a js is client or server from its name. Unless we use .server.js for them.

Tiddlyweb plugin might contain both client and server code.

[Arlen22]

Most of the server code is in core or boot. The code is usually still parsed by the browser, which means that parsing errors would happen.

[Leilei332]

I've noticed that some CSS features (like padding-inline-start) has a 2020 baseline, but actually it is related to the former core of Edge: EdgeHTML. IMO we should also drop support for EdgeHTML, since it is considered as deprecated now.

[Jermolene]

I've noticed that some CSS features (like padding-inline-start) has a 2020 baseline, but actually it is related to the former core of Edge: EdgeHTML. IMO we should also drop support for EdgeHTML, since it is considered as deprecated now.

I agree.

We need to start recording these kind of policy decisions for v5.4.0 so that contributors can easily discover what changes have been made. I've started a ticket to record them: https://github.com/TiddlyWiki/TiddlyWiki5/issues/9220