Publish the ASTs and parsers for the ASTs for the FsLex and FsYacc tools to enable more robust editor tooling

[baronfel]

Description

Currently, the ASTs and parsers for FsLex and FsYacc are internal to the tools themselves. It would be wonderful to have a published version of the AST structures and parsers for them, to enable me to improve FSAutoComplete (the LSP engine behind Ionide in VSCode) as well as other editors with better tooling around these files.

Having access to these structures would allow:

• better highlighting, navigation, find-references, and tooltips for these structures (imagine being able to CMD+Click on a rule name and go to its definition, for example),
• enable better integration with the F# portions of code in the rule productions, by allowing us to make virtual F# files whose contents are the contents of the 'top code' plus the contents of each rules' specific production,
• enable Web-based tooling via transpilation to fable (imagine something like the fantomas tools but for FsLEx/FsYacc files)
• and any number of other scenarios that I'm not imaginative enough to express.

I propose that a new layer of library be added between each of fsyacc and fslex like so:

• FsLexYacc.Runtime -> FsLex.Core -> fslex
• FsLexYacc.Runtime -> FsYacc.Core -> fsyacc

such that most of the AST, parsers, and logic for each project are pushed down into the core layers, and the outer projects mainly orchestrate calling into this core logic.

[baronfel]

If this is approved, I'd be happy to contribute an MR for it.

[dsyme]

Approved

[baronfel]

The associated MR was merged, can a new release be done that includes the new packages?

[sergey-tihon]

I can help with publishing but I am not sure about dependencies ... They looks wrong

• It is OK tha there is no package that depend on FsLexYacc.Runtime?
• FsLex.Core depends on metapackage FsLexYacc ?
• FsLexYacc has no dependencies?

Metadata:
  id: FsLex.Core
Dependencies:
  .NETStandard2.0:
    FSharp.Core: '>= 4.6.2'
    FsLexYacc: '>= 10.2.0'

Metadata:
  id: FsYacc.Core
Dependencies:
  .NETStandard2.0:
    FSharp.Core: '>= 4.6.2'

Metadata:
  id: FsLexYacc.Runtime
Dependencies:
  .NETStandard2.0:
    FSharp.Core: '>= 4.6.2'

Metadata:
  id: FsLexYacc
Dependencies:

[baronfel]

good eyes, the following packages should be published:

• FsLexYacc.Runtime
• FsLex.Core (new, depends on FsLexYacc.Runtime)
• FsYacc.Core (new, depends on FsLexYacc.Runtime)
• FsLexYacc old combined package, no dependencies

so it looks like there are some package authoring issues introduced with my last PR. I can investigate those.

[sergey-tihon]

FsLexYacc old combined package, no dependencies

why? do we have a goal to abandon this package one day?

Can we make FsLexYacc empty package with dependency on FsLex.Core and FsYacc.Core ?

[baronfel]

It couldn't be empty just yet, because the .Core packages don't contain the actual executable binaries for fslex/fsyacc, only the parsing/generation/tooling code. The 'drivers' still live in their individual projects, and I didn't want to change the distribution mechanism for them (since they are most commonly distributed as the build-tooling package).

[sergey-tihon]

OK got it.

Right now FsLexYacc had dependency on FsLexYacc.Runtime. Why this dependency disappeared after refactoring?

[baronfel]

Because it didn't actually have a dependency on FsLexYacc.Runtime, because it embedded that dll in the metapackage already. And since the meta-package didn't contain libraries for others to reference, it made no sense to me to also include a package reference to the FsLexYacc.Runtime nuget.

[sergey-tihon]

Because it didn't actually have a dependency on FsLexYacc.Runtime, because it embedded that dll in the metapackage already.

When you generate code (fs) from fsl/fsy using (FsLex/FsYacc) you need FsLexYacc.Runtime.dll do build it and to execute it.

• FsLexYacc.Runtime project has lib folder with dll that become target project dependency.
• Metapackage FsLexYacc does not have lib folder and contains only build-time tools.

And since the meta-package didn't contain libraries for others to reference, it made no sense to me to also include a package reference to the FsLexYacc.Runtime nuget.

I believe with this approach user's will have to manually add two NuGet dependencies (one to compile fsl/fsy and one to compile generated code). So, it may sense to keep the dependency between packages to simplify Getting started experience.

[baronfel]

Ah, that explanation makes perfect sense. I'll try to send a PR that re-aligns the packages with our discussion here.

Thank you for taking the time to work through this with me!

[dawedawe]

Hey @baronfel , is there anything still missing here or can this be closed because of #144?

[baronfel]

Good spot @dawedawe 👍