RFC: Authoring DX for Documentation

[kitsunekyo]

Description 📝

I know that the packages are undergoing a big-refactor currently, so I'd like to take the opportunity to propose some food for thought and suggestions on how the libraries could potentially be documented more easily.

In /big-refactor i've noticed that the approach to generating documentation is to create individual example.ts and example.html files, alongside the classes / modules. The ts files contain markdown comments alongside the typescript, with plenty of @ts-ignore.

Theres a few issues I see with this approach.

• individual files that have to be maintained outside of the library code
• the examples are transformed for the docs page only, and are thus very detached from the actual library
• putting markdown inside ts files seems counter intuitive. The files are docs pages, so if anything, they should be markdown files, with code sections. But I understand that you'd potentially lose typechecking for the example code.

I'd appreciate any input or thoughts on this topic.

Suggested solution 💡

Instead of authoring examples and descriptions in extra files, we could use tsdoc/jsdoc comments to annotate and explain classes, methods, etc.

This would allow library users to access documentation directly in their IDE. API Documentation for the docs page could be easily generated with typedoc, and integrated into the docusaurus page without much effort.

i see that there is typedoc config inside the packages, but its unused. was there any reason why you stopped using typedoc?

For the authoring experience, I'd highly suggest using prettier with prettier-plugin-jsdoc. That way any contributor would automatically have the same formatting experience.

  /**
   * Generates geometries and assets from an IFC file buffer.
   *
   * @example
   *   tiler.streamFromBuffer(new Uint8Array(await ifcFile.arrayBuffer()));
   */
  async streamFromBuffer(data: Uint8Array) {
  // ...

With tsdoc-plugin-markdown, we could generate markdown api documentation, which could be consumed by the docs repository.

Additional context ☝️

There's a nice article by deno about this https://deno.com/blog/document-javascript-package I dont have a good solution for having type safety in examples yet. But i'd split guides from api docs anyway.

The guides should be authored in markdown directly in the docs repository. They don't belong in the library source code repos in my opinion.

I'll try to dive further into https://github.com/TanStack/tanstack.com, as they have a stellar docs.

Open questions

• [ ] typechecking for markdown code blocks (something like https://github.com/bbc/typescript-docs-verifier)
• [ ] jsdoc auto-formatting
• [ ] docs versioning

[agviegas]

Hey @kitsunekyo , thanks a lot for the proposal and for taking the time to ellaborate, feedback like this is really useful.

Here's a general overview of our documentation pipeline:

For the Tutorials section of the docs:

1. We create an example.ts file per component.
2. When we build the library (npm run build), we create a production version of the example.ts files. That way, if we change something in the library that breaks any example, it won't compile.
3. In the docs repo (which uses docusaurus) we fetch the example.ts files of all our repositories and automatically generate a nice-looking markdown page for each one. So generally people will never look at the example.ts files directly, but to the tutorials in the docs.
4. At the end of each tutorial, we use an iframe looking at the deployed example in the corresponding repository, so the user can see the live example in action.

For API section of the docs:

1. We add typedoc comments to all classes, methods, properties, etc.
2. In the docs repo we fetch all the repositories and automatically convert them to markdown/docusaurus pages.

Of course, this system is not perfect, but I think it has some benefits to consider:

1. We have all the docs in a single place, regardless of the number of repositories that we have.
2. We can present tutorials in a nice article-like way, instead of just a raw code page.
3. We should also get the same TypeDoc comments you mentioned (we are completing them right now for all our libraries).
4. When we are developing / fixing the libraries, we can use the same examples to debug / test the features and fixes.

We are currently working on some improvements:

• Adding a search functionality with Algolia, which will work through all the API pages and tutorials.
• Separating the tutorials in subfolders by the library to which they belong.

In summary, I agree: we need TypeDoc, and it alone should be so good that a seasoned dev should be able to use the library by just the pop ups in their IDE. But I also think that the docs page add value, especially for people discovering the library and starting to play with it (like the Three.js manual page).

Do you still think that examples should be kept separately? A possible solution would be writing the examples code in the docs repo, but then the integrity of them and the library would be more loose. Also, if we decided to write the examples in markdown directly instead of writing them in TS and then converting them to Markdown programatically, it could be hard to spot errors/inconsistencies when the API changes.

What do you think?

[kitsunekyo]

With the details from your response, I'd summarize the Consumer requirements like this

1. The packages should expose well architected apis, consistent and descriptive naming, etc, so that consumers don't NEED documentation.
2. IDE autocomplete and description with typescript + jsdoc.
3. API docs with all modules, classes, etc.
4. Extensive examples, on how to use certain modules and apis.
5. Descriptive guides and how-to's in text form. (might not be needed)

Authoring requirements could be the following:

1. Typesafety for examples and guides, to catch outdated / broken docs.
2. Single point of truth: We don't want to manage the same information in multiple places.
3. DX: The setup should be based on standards or well-known tools, to reduce contribution barrier.

These are different experiences, that might be fulfilled in different places. 1 and 2 will be within the library repos. 3 would be generated with jsdoc or typedoc. 4 and 5 could be written directly in the documentation repo.

I think it is paramount that the authoring experience of documentation is great, otherwise documentation will be treated as secondary, which quickly kills any open-source project.

Testing vs Documentation

What I would advise against is to use documentation and examples as a replacement for testing. This is a completely separate requirement that should provide confidence for changes.

[kitsunekyo]

API Docs In my opinion, anything related to api documentation should be authored directly inside the packages themselves. This co-location with jsdoc comments provides documentation directly inside the consumers IDEs, while allowing api markdown docs to be generated from them.

Text Docs Text-focused documentation like guides and tutorials should be done with markdown / mdx. The authoring experience is massively superior to writing these in typescript files.

The docs repository is the conduit for accessing documentation through the web. It should provide high-level documentation on the @thatopen ecosystem and an entrypoint, and search for consumers that dont know what they are looking for.

By using docusaurus it should be fairly trivial to pull in the typedoc / jsdoc markdown output from the individual packages. Written guides and tutorials could either be authored in the individual packages and then pulled into the docs repo, or authored directly in docusaurus. I lean towards docusaurus, but dont have a solid reason as to why.

[agviegas]

@kitsunekyo I agree mostly! The API Docs are created inside the packages with comments, like you describe.

I also agree that writting a markdown is more convenient than writting typescript examples, but the fact that we can execute these examples to make sure they work when we make changes to the libraries are a plus check that makes the difference for us right now. Otherwise, it would be hell to locate outdated / broken examples. So the approach we have chosen is to have the tutorials in typescript, and then automatically convert them to markdown using a script from the docs repo.

I also agree that the docs should provide the entry point, and this is what it's doing now.

Pulling the info from the docs is exactly what we are doing now. Specific tutorials are written in the packages to have everything in the same place.

This is the result now (also thanks to you for all the improvements you provided!).

It might be not perfect, but I think it's something that has a decent compromise between automatic generation, tutorials maintenance and ease of onboarding new contributors.

Closing this! Thank you very much for the feedback and for the help.

[kitsunekyo]

great work Antonio. The new docs are a really awesome improvement!