listOf, nonemptyListOf and emptyListOf handling in toAST

[charlespwd]

Docs specify that, by default, array-like things are getting transformed into arrays. This is true of ListOf but not of the lexical version listOf.

I had to add the following to my mappings object for my list of things to be turned into a list.

    listOf: 0,

    nonemptyListOf(first, sep, rest) {
      return [first.toAST(this.args.mapping)].concat(rest.toAST(this.args.mapping));
    },

    emptyListOf() {
      return [];
    },

I suspect that these should be added to the defaults?

Perhaps here ?

[pdubroy]

I started implementing a fix for this, but then I remembered the reason that we don't have defaults for the lexical versions of these rules —

One of the assumptions that toAST makes is that lexical rules represent tokens — so the default handling of lexical rules is to just return a string containing the input text that was consumed. That's a reasonable assumption for most lexical rules, but I think that listOf and friends are an exception. I think in most of the places where you'd use the lexical listOf, it's much more useful to discard the separator. E.g.: slash-separate paths (path = listOf<pathEl, "/">), TSV/CSV formats (line = listOf<col, "\t">, etc.

So, I think we should change the defaults here, but I wanted to record my thinking for posterity :-)

[pdubroy]

This has been fixed on main. However, since it's a breaking change, it won't be released until the next major version (v17), probably in the next few weeks.