Support for MyST and R Markdown-based notebooks

[teonbrooks]

Recently I've been using jupyter-book as one of my primary notebook environments. It's built on top of MyST, which is a flavor of markdown that is compatible with reStructuredText, and they've built an extension for syntax support.

I really love the integration VSCode has for executing #%% cells in python scripts with and general cells in Jupyter Notebooks. I am wondering if it would be of interest to support other cell paradigms like the ``` code chunk delimiter one in jupyter-book, which is also shared in RMarkdown. If it's a not feature that fits the roadmap, could you make a suggestion how I might be able to build off what is currently there and support it as an extension?

Thanks a bunch! I really love this product

[rchiodo]

This would almost work out of the box. I believe the only changes would be to:

• Add the .md files to our list of files to provide code lenses for
• Add a regex to the cell marker regex we have in our settings to recognize python cells (maybe the end of the cell might be a bit more work).

If you wanted to do this in a separate extension, we'd need an extensibility point for submitting code to the python extension (which you could add to our extension with a PR against us too)

[teonbrooks]

oh awesome. I'm more than happy to have this as a part of this existing extension.

I know that the starting delimiters are:

• MyST (.md) with all combos:
  • ```{code-block} ipython
  • ```{code-cell} python
• Rmd (.Rmd):
  • ```{python}

Let me know if there's anything I can do to help

[rchiodo]

If you're willing to add a PR, I believe we'd love to accept it. You'd have to expand our code that figures out cell ranges and add those regexes to our searching.

I think after that it would just work. (Might be some things that check for .py - like debugging).

[rchiodo]

If you're uncomfortable with a PR, it would likely take a bunch of other people up voting the idea for us to get to it.

[rchiodo]

We're happy to help point out where to add changes to our code though.

[teonbrooks]

@rchiodo, that would be very helpful and i will try to take a stab at it. I was wading through the codebase but some pointers would be welcome

[rchiodo]

First I think you'd go here to change the file extensions we provide code lenses for: https://github.com/microsoft/vscode-python/blob/68aeb2737daa9b86457f1d2df6f3619f7e975aa2/src/client/datascience/datascience.ts#L45

You'd change that PYTHON_ALLFILES to be something else that included your markdown file types.

Then you'd likely change the default regex defined in package.json (and I think in the source too somewhere) https://github.com/microsoft/vscode-python/blob/68aeb2737daa9b86457f1d2df6f3619f7e975aa2/package.json#L2309

so that it included the patterns you specified above. You'll likely have to add another setting for 'end' cell markers or something

Then finally you'd have to change our logic that finds cell ranges. The crux of that is here: https://github.com/microsoft/vscode-python/blob/68aeb2737daa9b86457f1d2df6f3619f7e975aa2/src/client/datascience/cellFactory.ts#L153

Right now it assumes the next begin marker is the end of the previous cell. Obviously you'll have to handle the case where your end marker is there too.

Hope that helps. Feel free to submit a draft PR if you want to ask us more questions when you get farther.

[teonbrooks]

Hi, unfortunately I don't think I will have time to take this on myself but I think this would be a great improvement to notebook ergonomics in vscode.

[CaseyWeiner]

If this could happen at some point, that would be really awesome. One idea I had for HTML export is to expand it to prompt the user to pick either a static file or a jupyter book. For example if I choose jupyter book, it opens a GUI prompt to specify traits of the book and then creates the html pages. Could incorporate Github pages with this as well

[teonbrooks]

for visibility, it looks like the quarto vscode extension now supports its version of markdown, qmd (https://github.com/quarto-dev/quarto-cli/issues/149). I think a similar pattern could be used for the code execution part for other markdown files. looks like there's a related PR (https://github.com/microsoft/vscode-jupyter/pull/4876) to generalize the markdown within the standard vscode library

[arogozhnikov]

Moving suggestion from https://github.com/microsoft/vscode-jupyter/discussions/10174 as recommended by @DonJayamanne

There are a number of already-existing formats that can convert from/to ipynb, implemented as ContentManagers for jupyter. They augment loading/saving process with on-the-fly conversion between filetypes.

While it could be of high value to implement the same for vs code, currently custom NotebookSerializers should provide conversion vscode <> custom format. This requires a lot of logic that is specific to vs code, and requires copying a good chunk of code and dependencies from vs code. (see https://github.com/microsoft/vscode-jupyter/discussions/10174 and https://github.com/microsoft/vscode-jupyter/discussions/10121)

Proposal below should simplify support for custom formats by reusing existing converters between ipynb and custom formats.

I suggest that default vscode NotebookSerializer for ipynb could be provided with a following triple:

1. filename extension (e.g. .Rmd, .ipynb.py and others)
2. callback to_ipynb(custom_format_bytes, filepath) -> ipynb_bytes
3. callback from_ipynb(ipynb_bytes, filepath) -> custom_format_bytes

For instance, when saving a file, extension will convert to ipynb, then use a corresponding conversion.

Another possibility is even simpler, in this case work with FS is offloaded to hooks:

1. filename extension (e.g. .Rmd, .ipynb.py and others)
2. callback serialize_from_ipynb(ipynb_bytes, filepath) -> None
3. callback deserialize_to_ipynb(filepath) -> ipynb_bytes

While filepath argument shouldn't be used in most conversions, it will be useful for extensions that grab additional files (like jupytext) or are configured at the repository level so they could have access to configuration files.

Edited on Oct 26: typo and added a link to one more discussion; added an alternative suggestion on hooks

[parmentelat]

I'd like to +1 here

We have been using (jupytext-powered) text-based notebooks for quite some time now, because ipynb goes along with git so awkwardly

Using jupyter classic notebook mainly, so together with jupytext And it would be tremendous help to be able to use vs-code instead

I've been trying to use the vscode-jupytext extension for a while, but this clearly needs more work, so I ended up here following @DonJayamanne's post in that repo

How can we help ?

[DonJayamanne]

Duplicate of https://github.com/microsoft/vscode-jupyter/issues/1240