Consider adding support for GFM, footnote and frontmatter syntax to default installation

[hukkin]

Context

It seems many users do not have a good understanding of what CommonMark is, and that the Markdown they use (often in GitHub) supports additional syntax such as all the GFM extensions, footnotes and frontmatter. This leads to frustration: people think that mdformat is broken if it breaks their frontmatter syntax, even though it is clearly documented that only CommonMark is supported and the fix is easy -- install a plugin that adds support for the additional syntax.

Proposal

Make pip install mdformat support GFM, footnote and frontmatter syntax in addition to CommonMark. Make pip install mdformat-core (or maybe mdformat-commonmark is better?) install what pip install mdformat currently does.

This means that the current package is moved under mdformat-core. Then we make the mdformat package essentially an empty package that depends on mdformat-core, mdformat-gfm, mdformat-footnote and mdformat-frontmatter.

Tasks and updates

No response

[inktrap]

What if the user had to specify the dialect/flavor of markdown? Then an error message could remind them that the package is not installed.

• something like black's -t, --target-version, but --target-format doesn't really fit?!
• pandoc's -f --from or -t --to or --read --write?
• -f, --format ? would be good because of --from, but typically -f stands for file.

If one works on two unrelated projects it is imagineable that they chose to use different markdown dialects that are incompatible to one-another.

It is great to reduce the confusion of users, so there are less issues and more adaption. If I had to guess, GFM is probably most-widely used and so it is no surprise that markdown-it-py chose these presets as well. But I just want to mention that GitHub is a commercial microsoft-owned platform and personally I wouldn't want to give them an advantage over other dialects, f.e. Gitlab or Pandoc. Yes, I know, this is an issue on GitHub.

Edit: markdown-it uses GFM as a preset, markdown-it-py uses commonmark and supports gfm-like.

FYI: Rendering can be compared here and here is some overview as a table.

It would be more consistent with markdown-it-py though, which would be really awesome!

[inktrap]

And if you are going to change the structure of packages … what about those that contain an underscore?

│ 📂 mdformat_deflist     │ 0.1.1   │ 21-06-2021 │ An mdformat plugin for markdown-it-deflist.                                  │
│ 📂 mdformat_openmmlab   │ 0.0.1   │ 27-05-2022 │ An mdformat plugin for...                                                    │
│ 📂 mdformat_tables      │ 0.4.1   │ 04-05-2021 │ An mdformat plugin for rendering tables.                                     │
│ 📂 mdformat_frontmatter │ 0.4.1   │ 09-06-2021 │ An mdformat plugin for parsing / ignoring frontmatter.                       │
│ 📂 mdformat_footnote    │ 0.1.1   │ 16-02-2022 │ An mdformat plugin for parsing/validating footnotes                          │
│ 📂 mdformat_myst        │ 0.1.5   │ 02-02-2022 │ Mdformat plugin for MyST compatibility.                                      │
│ 📂 mdformat_pelican     │ 0.1.0   │ 29-08-2021 │ An mdformat plugin for pelican markdown items                                │

[rpdelaney]

As a user, I don't consider this an annoyance rooted in mdformat. It's a standard cost of doing business with open standards: "open" means "everyone has their own flavor". I use mdformat because I don't have time or interest to get deep into what is CommonMark and what isn't, and it makes my life much easier.

If you merge these plugins into the trunk, you may end up maintaining them as well. If you are concerned about that, I think it would be enough to detect non-compliant markdown and print a warning suggesting the use of a plugin: then you don't have to close github issues about unexpected behavior with markdown that doesn't conform to the CommonMark spec. :)

[inktrap]

"open" means "everyone has their own flavor"

Can has their own flavor. While there are multiple, some are used more, and are more important. And useage numbers are only one aspect when to judge standards, from e.g. a formal point of view it makes sense to have a common subset of markdown defined as default, which sounds a lot like commonmark.

If you merge these plugins into the trunk, you may end up maintaining them as well.

It is about enabling them by default, because most users are used to gfm + tables + other goodies and not to commonmark only. And most of them, like you, don't want to think about it.

I see no reason why they would be merged into trunk when you can list them as a dependency. They are used when their packages are present, if I remember correctly.

I think it would be enough to detect non-compliant markdown and print a warning suggesting the use of a plugin

But … mdformat doesn't "know" how to handle undefined elements when dealing with "non-compliant markdown". So you would need to check which plugin defines that element. Since multiple plugins can define the same elments (let's say tables for instance) it is difficult to recommend one specific plugin. Then you would be in the position to print either all of them or something generic about informing yourself about plugins and installing the one(s) you may need.

Secondly … non-compliant in relation to what? To the version of markdown the user configured! Which may not be what they want/expect which this issue is about.

I don't have time or interest to get deep into what is CommonMark and what isn't, and it makes life much easier.

And it works because you set it up once (per project?). Which less people would have to do if the flavor would be one that is used more often.

[rpdelaney]

@inktrap I agree with much of what you said, for what that's worth. I really only mean to float this as an alternative solution.

[csala]

I just :-1: 'ed the initial proposal as a form of downvoting.

Personally, I think that the current structure of mdformat supporting the smallest spec set and letting plugins add to it is much better than having it the other way around. This is especially relevant, considering that mdformat is already out there and plugins exist which depend on it.

If a specific preset that is equivalent to the current mdformat with a few plugins is wanted, I would suggest to rather create a new package for that, like mdformat-extended or mdformat-default-plugins, something that implies that it is a superset of what plain mdfromat is.

If one works on two unrelated projects it is imagineable that they chose to use different markdown dialects that are incompatible to one-another.

@inktrap I suppose this is assuming that one uses the same mdformat installation for both projects, which is definitely not what environment isolation tools like pre-commit encourage. I would say that if someone works on two different projects, they should be encouraged to run mdformat from 2 different installation stacks (like 2 different pre-commit configs), with potentially different plugin sets.

[csala]

Here is an interesting approach from the mdformat-mkdocs plugin:

Tip: this package has a pip extra, recommended, of plugins that work well with mkdocs:

• mdformat-admon
• mdformat-beautysh
• mdformat-black
• ...

This could be an option here too: Have mdformat[recommended] or mdformat[featured_plugins]

[inktrap]

Oh wow, this issue still pops up … I haven't thought about this for a long time and the project for which I am using mdformat is currently on hold.

On the one hand I agree that isolation is important and that projects should have their own settings in f.e. their pre-commit configs.

On the other hand it is IMHO painful to install the same tools over and over again!? It is much easier to install something like mdformat with pipx and then have a personal mdformat config that lives in my home directory and is overwritten on a per project basis. If a project specifies mdformat as a development dependency, fine, install it.

If it is installed per project, it definitly should have a per-project config because chances are, the intention is to use it in a specific way, and not with some user-defined config that the author doesn't know about and which might require different versions or plugins anyway.

I don't like the idea that mdformat has a couple of plugins that are implicitly active if they are installed, because this seems opaque. Did I really install x and y? Does it work? Is this really what I want?

An explicit config that defines which flavor of markdown should be formatted would be best, comparable to linting rules. And if there is an explicit config there has to be decision … (a) go for maximal compatibility with what people who don't use commonmark expect (install plugins automatically) or (b) don't introduce changes to the default behaviour that might break things.

I do get that there is a need for tools that have no config, e.g. black has no settings to specify a formatting style. It is opinionated and in being so ends discussions that aim to find some ideal formatting. But I don't think this applies here?

So would I would suggest is: install plugins by default to have gfm compatible markdown (what newcomers might expect) and introduce some default config that either specifies (a) or (b). Both could generate a warning (for (a): default selection of plugins has changed, if you want to old behaviour change the config and for (b), especially if there are undefined markdown elements in the source: you might want to consider installing extra plugins to get something like gfm.