Switch to GitHub Flavored Markdown (Redcarpet) + remove scripts/highlight.js along the way

[mathiasbynens]

@pepelsbey I wanted to run this by you. I’ve enabled GitHub Flavored Markdown, which effectively removes the need for the scripts/highlight.js file. Now, all the syntax highlighting can be done at build time.

All we need to do is use fenced code blocks as described on https://help.github.com/articles/github-flavored-markdown#syntax-highlighting, so that we can (optionally) specify the programming language used for each code block.

The result for an example post that uses this syntax can be found here: http://54.213.240.91/blog/opera-19/

[pepelsbey]

I’m not sure if this is a good markup:

<div>
    <div class="CodeRay">
        <div class="code">
            <pre>
                <span class="keyword">var</span> promise…

It’s not even proper markup for code blocks. I would rather keep using highlight.js, really.

[mathiasbynens]

Agreed, that markup sucks. But that’s an implementation detail. I’ve just tweaked the config file to use Redcarpet instead. Now the output is much cleaner:

<div class="highlight">
  <pre>
    <code class="js language-js javascript" data-lang="js">
      <span class="keyword">var</span> …

Here’s some default CSS: https://github.com/mojombo/tpw/blob/master/css/syntax.css

What do you think?

[pepelsbey]

Moving my comments from commit 262d83f0196d99f62737e7cc3aa175ba9bff0f52:

Please don’t switch MD parser without discussion. I spent a few really nervous hours today trying to figure out why build task is failing. I don’t want to do this again. I tested a lot of pages with kramdown and some of used features doesn’t work right now (such as markdown="span" attributes). So I just reverted config to Kramdown without GFM.

The same with Coderay. It’s not the best way to highlight code, from my point of view. I would like to keep Dev.Opera code quality as good as possible. Highlight.js is good enough for this.

Let’s just finish the current stage of Dev.Opera development with existing libraries and approaches.

[mathiasbynens]

The same with Coderay. It’s not the best way to highlight code, from my point of view. I would like to keep Dev.Opera code quality as good as possible.

Yeah, we already agreed on that: https://github.com/operasoftware/devopera/issues/38#issuecomment-36019407. No argument there.

I spent a few really nervous hours today trying to figure out why build task is failing.

Sorry about that. What was failing exactly? I rebuilt the whole site after switching to RedCarpet and making a small change to markdownline (7c12f921eaa57f7fa1673da3861f2a604c022e03) and didn’t get any build errors after that. (I would have reverted it myself if I saw any kind of error!)

I tested a lot of pages with kramdown and some of used features doesn’t work right now (such as markdown="span" attributes) […] Let’s just finish the current stage of Dev.Opera development with existing libraries and approaches.

Fair enough, but I’d like to work on switching to Redcarpet in the future, if only for fenced code blocks syntax (switching back to Kramdown ‘broke’ e.g. http://54.213.240.91/blog/opera-19/). So could you please list the issues you were having here so that I can take a look? For example, I have no idea what markdown=span is.

[pepelsbey]

What was failing exactly?

Number of errors related to gsub in markdownline and htmlmin output.

markdown attribute is a way to use markdown inside HTML and to control span or block type of it.

But why Redcarpet, why fenced blocks? You need to just indent your code sample by tab or 4 spaces and that’s it — and as it’s indented it becomes much more readable (and also highlighted in code editors). You don’t need to mark you code with language — it will be auto-detected by Highlight.js.

Kramdown is really well-documented: quick reference and full documentation. Kramdown allows you to do more flexible code, mixing HTML and MD. And would really like to have more control over Dev.Opera code. To use <figure>s with <figcaption>s for pictures and inline demos and code samples, as it’s supposed to be. Not just <img style="float:left; margin:1em 0 0 1em">.

[mathiasbynens]

All that is possible with Redcarpet! I ran some tests with the following _config.yml and Redcarpet 3:

title: Dev.Opera
markdown: redcarpet
redcarpet:
  extensions: [
    "lax_spacing",
    "no_intra_emphasis",
    "strikethrough",
    "tables",
    "with_toc_data"
  ]
pygments: true
permalink: /:categories/:title/
include:
- '.htaccess'
exclude:
- 'node_modules'
- 'Gruntfile.js'
- 'package.json'
- 'README.md'
- '*.scss'
# limit_posts: 150

Then I compared the HTML output of jekyll build for this repository: Kramdown vs. Redcarpet. I used a private GitHub repository for this so I could easily look at the diffs and detect possible issues.

Reasons to switch to Redcarpet some day

• supports fenced code blocks, which enables copy-pasting chunks of Markdown from anywhere on GitHub
• faster build times (~15 minutes to build the entire site vs. ~18 minutes using Kramdown on my machine)
• pretty HTML output (prettier than Kramdown, even – no XHTML-style /> nonsense)
• allows us to specify the language for each code block in Markdown (useful metadata)
• allows us to get rid of highlight.js and do the syntax highlighting work at build time, based on the programming language for each block (= more accurate results)
• …

Reasons not to switch to Redcarpet

• ~~no Markdown syntax for <abbr>~~ not really needed as we can just use HTML directly
• ~~build issues~~ – these are fixed now
• ~~we want Redcarpet 3, but Jekyll 1.x uses Redcarpet 2. Workaround: https://github.com/jekyll/jekyll/pull/1299#issuecomment-36118919 (This won’t be a problem anymore when Jekyll 2 is released.)~~ Jekyll 2 is now released, so this is not a problem anymore.
• anything else?

[pepelsbey]

Why not sundown which is even more faster? Not sure though it Jekyll support non-Ruby parsers.

[mathiasbynens]

The commit message here is a pretty good reason IMHO: https://github.com/vmg/sundown/commit/37728fb2d7137ff7c37d0a474cb827a8d6d846d8 Anyway, does it support fenced code blocks?

[pepelsbey]

Reasons to switch to Redcarpet some day supports fenced code blocks, which enables copy-pasting chunks of Markdown from anywhere on GitHub

Since we’re mostly using code editors for editing articles, is it just a matter of copying code without opening and closing backticks and pressing indent Cmd ] shortcut? Weak reason, from my point of view. But faster build time is essential for us, I agree.

Let’s do it this way: since I have converted a tons of articles to MD and developing content styles and general style guide, I’ll have a look at Redcarpet 3.0 features and how are they aligned with our needs. Kramdown allows us to do much more than any other MD engine, that’s why we’re using it now.

[pepelsbey]

Also I personally don’t like the idea of code highlighting rendered right in HTML during the build:

Output code really sucks.

<div>
    <div class="CodeRay">
        <div class="code">
            <pre>
                <span class="keyword">var</span> promise…

And

<div class="highlight">
  <pre>
    <code class="js language-js javascript" data-lang="js">
      <span class="keyword">var</span> …

We should have less building things in our process since we’re aiming to a shorter build time.

Highlight.js is pretty good at code detection so you don’t have to specify language (I haven’t seen any detection failure). Also it’s not messing up markup with extra divs and nonsense classes.

[mathiasbynens]

Since we’re mostly using code editors for editing articles, is it just a matter of copying code without opening and closing backticks and pressing indent Cmd + ] shortcut? Weak reason, from my point of view.

You can still do that if you prefer. But IMHO typing a few backticks is much easier, and either way it’s nice to be compatible with GitHub’s syntax (copying from comments/gists/etc. without having to convert the fenced code block syntax manually).

Output code really sucks.

We agree the CodeRay output (first example) sucks. But Redcarpet’s output is really nice. The class="language-*" format is actually the recommended way to do this as per the HTML standard:

Authors who wish to mark code elements with the language used, e.g. so that syntax highlighting scripts can use the right rules, can use the class attribute, e.g. by adding a class prefixed with language- to the element.

Highlight.js is pretty good at code detection so you don’t have to specify language (I haven’t seen any detection failure). Also it’s not messing up markup with extra divs and nonsense classes.

It is, though. The only difference is that the extra <div>s and <span>s are added at run-time (which requires JS and is slower) and thus not visible in the HTML source. They’re still in the DOM. Why use JS for something that can be done at build time without any noticeable effect on build time?

We should have less building things in our process since we’re aiming to a shorter build time.

I thought tags were the big bottleneck here? Enabling Pygments doesn’t seem to make much of a difference in terms of build times. (Of course, currently all code blocks are marked as text and don’t get highlighting.)

Maybe we should move the code highlighting discussion to a separate thread, as it doesn’t block switching Markdown engines.

[odinho]

hoedown was the sundown continuation, btw.