page loaders; jsx + react

[mbostock]

This work-in-progress PR introduces the concept of a “page loader”, which is like a data loader for pages: it allows you to generate a page dynamically rather than being limited to static Markdown. The purpose of page loaders is two-fold. First, it serves as a foundation for server-side rendering #931. Second, it allows pages to be written in languages other than Markdown, such as JSX/MDX #971.

There’s still quite a bit to figure out here, but the general idea is to replace this sort of code:

const source = await readFile(join(root, path), "utf-8");
const page = parseMarkdown(source, options);

with something more generic:

const page = generatePage(root, path);

whereby the user can write whatever logic they like to generate the page.

TODO

• [ ] Resolve the appropriate page loader for the requested page (hello-world.page.js?)
• [ ] Cache the output of a page loader (same rules as data loaders)
• [ ] For the preview socket, watch the page loader source instead of a .md file

Presumably this could be extended to support patterns or parameterized paths, like data loaders, so maybe there would be a generic way to register support for .mdx, .html, or .jsx extensions for the entire project rather than each page having a distinct loader (such as index.page.js).

[mbostock]

We could maybe call this “page generator” (and “generated pages”) instead.

[Fil]

Yay, I wanted to explore something like this too. In my idea (that I hadn't really tried to implement yet) parseMarkdown would have been one of the input helpers, instead of something that is applied by default to all (transformed) pages. In order to outline some design considerations:

my idea was something like this:

md  -> parseMarkdown -> parsed page
mdx -> parseMdx      -> (same)
js  -> runJs         -> (same)

Whereas this PR treats md as the pivot format:

js ---|
md ---|
mdx --|
       \-> md -> parseMarkdown -> parsed page

The only advantage to my approach is that the generated page wouldn't go through markdown at all if your input format was not md, thus making sure there are no surprises after your page generator has run. (However, outputting html with no line breaks should guarantee that too.)

This PR's approach is much better:

• well “contained” in terms of API, and simple to reason about if you want to write a new page format generator.
• You can even run the page generators independently from Framework with node home.page.js > home.md; like you do with data loaders.
• It does not require the page format generator to be aware of how Framework works: you could even use pandoc.

[mbostock]

Indeed that was my original plan which you can see in the first commit, and I’d still like to support that, too — particularly for JSX which is compiled to JavaScript that needs to be evaluated on the server and then serializes to HTML. I was originally thinking of generating JSON using the RenderPage interface, but that won’t quite work as-is because parseMarkdown calls parseJavaScript internally and I don’t want to serialize the entire AST for JavaScript; so I think I’ll pull the parseJavaScript out and have parseMarkdown simply return the unparsed source for code blocks. I also want to support HTML as an extension, perhaps with <script type="observablehq"> for code blocks? But not a high priority.

[mbostock]

Now that we’ve landed JSX and React separately #1429, I want to reboot this PR to focus just on page loaders.

I want to stick with Markdown as the serialized representation of the page for now. We do need a serialized representation because we want to be able to write page loaders in other languages (e.g., index.md.py for a Python page loader) just like we do with data loaders. I think we could also support HTML as a serialized representation (e.g., index.html.py) and even static HTML pages (e.g., index.html instead of index.md). But we’d need to design that representation (for example how do we represent reactive code blocks, probably <script type="observablehq"> elements?) and that can be a future enhancement.

[mbostock]

Rebooted this PR. I marked this as ready for review since I’m planning on merging it into the parent parameterized routing PR and landing both together. Both are close to ready — mostly needing documentation and tests, and porting the theme previews over to parameterized page loaders.