commonmark: Pure Haskell commonmark parser.

[ bsd3, library, text ] [ Propose Tags ]

This library provides the core data types and functions for parsing commonmark (https://spec.commonmark.org). The parser is fully commonmark-compliant and passes the test suite. It is designed to be customizable and easily extensible. To customize the output, create an AST, or support a new output format, one need only define some new typeclass instances. It is also easy to add new syntax elements or modify existing ones.

Accurate information about source positions is available for all block and inline elements. Thus the library can be used to create an accurate syntax highlighter or an editor with live preview.

The parser has been designed for robust performance even in pathological cases that tend to cause stack overflows or exponential slowdowns in other parsers, with parsing speed that varies linearly with input length.

Related packages:


[Skip to Readme]
Versions [RSS] [faq] 0.1.0.0, 0.1.0.1, 0.1.0.2, 0.1.1, 0.1.1.1, 0.1.1.2, 0.1.1.3, 0.1.1.4, 0.2, 0.2.1
Change log changelog.md
Dependencies base (>=4.9 && <5), bytestring, containers, parsec, text, transformers, unicode-transforms [details]
License BSD-3-Clause
Copyright 2018-2021 John MacFarlane
Author John MacFarlane
Maintainer jgm@berkeley.edu
Category Text
Home page https://github.com/jgm/commonmark-hs
Bug tracker https://github.com/jgm/commonmark-hs/issues
Source repo head: git clone https://github.com/jgm/commonmark-hs
Uploaded by JohnMacFarlane at 2021-06-20T22:22:54Z
Distributions Arch:0.2, LTSHaskell:0.2, NixOS:0.1.1.4, Stackage:0.2, openSUSE:0.1.0.2
Downloads 7056 total (891 in the last 30 days)
Rating 2.0 (votes: 1) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Hackage Matrix CI
Docs available [build log]
Last success reported on 2021-06-20 [all 1 reports]

Modules

[Index] [Quick Jump]

Downloads

Maintainer's Corner

For package maintainers and hackage trustees

Candidates


Readme for commonmark-0.2.1

[back to package description]

commonmark

hackage release

This package provides the core parsing functionality for commonmark, together with HTML renderers.

🚧 This library is still in an experimental state. Comments on the API and implementation are very much welcome. Further changes should be expected.

The library is fully commonmark-compliant and passes the test suite for version 0.30 of the commonmark spec. It is designed to be customizable and easily extensible. To customize the output, create an AST, or support a new output format, one need only define some new typeclass instances. It is also easy to add new syntax elements or modify existing ones.

Accurate information about source positions is available for all block and inline elements. Thus the library can be used to create an accurate syntax highlighter or an editor with synced live preview.

Finally, the library has been designed for robust performance even in pathological cases. The parser behaves well on pathological cases that tend to cause stack overflows or exponential slowdowns in other parsers, with parsing speed that varies linearly with input length.

  • commonmark-extensions provides a set of useful extensions to core commonmark syntax, including all GitHub-flavored Markdown extensions and many pandoc extensions. For convenience, the package of extensions defining GitHub-flavored Markdown is exported as gfmExtensions.

  • commonmark-pandoc defines type instances for parsing commonmark as a Pandoc AST.

  • commonmark-cli is a command-line program that uses this library to convert and syntax-highlight commonmark documents.

Simple usage example

This program reads commonmark from stdin and renders HTML to stdout:

{-# LANGUAGE ScopedTypeVariables #-}
import Commonmark
import Data.Text.IO as TIO
import Data.Text.Lazy.IO as TLIO

main = do
  res <- commonmark "stdin" <$> TIO.getContents
  case res of
    Left e                  -> error (show e)
    Right (html :: Html ()) -> TLIO.putStr $ renderHtml html

Notes on the design

The input is a token stream ([Tok]), which can be be produced from a Text using tokenize. The Tok elements record source positions, making these easier to track.

Extensibility is emphasized throughout. There are two ways in which one might want to extend a commonmark converter. First, one might want to support an alternate output format, or to change the output for a given format. Second, one might want to add new syntactic elements (e.g., definition lists).

To support both kinds of extension, we export the function

parseCommonmarkWith :: (Monad m, IsBlock il bl, IsInline il)
                    => SyntaxSpec m il bl -- ^ Defines syntax
                    -> [Tok] -- ^ Tokenized commonmark input
                    -> m (Either ParseError bl)  -- ^ Result or error

The parser function takes two arguments: a SyntaxSpec which defines parsing for the various syntactic elements, and a list of tokens. Output is polymorphic: you can convert commonmark to any type that is an instance of the IsBlock typeclass. This gives tremendous flexibility. Want to produce HTML? You can use the Html () type defined in Commonmark.Types for basic HTML, or Html SourceRange for HTML with source range attributes on every element.

GHCI> :set -XOverloadedStrings
GHCI>
GHCI> parseCommonmarkWith defaultSyntaxSpec (tokenize "source" "Hi there") :: IO (Either ParseError (Html ()))
Right <p>Hi there</p>
> parseCommonmarkWith defaultSyntaxSpec (tokenize "source" "Hi there") :: IO (Either ParseError (Html SourceRange))
Right <p data-sourcepos="source@1:1-1:9">Hi there</p>

Want to produce a Pandoc AST? You can use the type Cm a Text.Pandoc.Builder.Blocks defined in commonmark-pandoc.

GHCI> parseCommonmarkWith defaultSyntaxSpec (tokenize "source" "Hi there") :: Maybe (Either ParseError (Cm () B.Blocks))
Just (Right (Cm {unCm = Many {unMany = fromList [Para [Str "Hi",Space,Str "there"]]}}))
GHCI> parseCommonmarkWith defaultSyntaxSpec (tokenize "source" "Hi there") :: Maybe (Either ParseError (Cm SourceRange B.Blocks))
Just (Right (Cm {unCm = Many {unMany = fromList [Div ("",[],[("data-pos","source@1:1-1:9")]) [Para [Span ("",[],[("data-pos","source@1:1-1:3")]) [Str "Hi"],Span ("",[],[("data-pos","source@1:3-1:4")]) [Space],Span ("",[],[("data-pos","source@1:4-1:9")]) [Str "there"]]]]}}))

If you want to support another format (for example, Haddock's DocH), just define typeclass instances of IsBlock and IsInline for your type.

Supporting a new syntactic element generally requires (a) adding a SyntaxSpec for it and (b) defining relevant type class instances for the element. See the examples in Commonmark.Extensions.*. Note that SyntaxSpec is a Monoid, so you can specify myNewSyntaxSpec <> defaultSyntaxSpec.

Performance

Here are some benchmarks on real-world commonmark documents, using make benchmark. To get benchmark.md, we concatenated a number of real-world commonmark documents. The resulting file was 355K. The bench tool was used to run the benchmarks.

program time (ms)
cmark 12
cheapskate 105
commonmark.js 217
commonmark-hs 229
pandoc -f commonmark 948

It would be good to improve performance. I'd welcome help with this.