json-tokens: Tokenize JSON

Convert JSON to a token stream. This libary focuses on high performance and minimal allocations. This library is distinguished from aeson in the following ways:

• In aeson, decode parses JSON by building an AST that resembles the ABNF given in RFC 7159. Notably, this converts every JSON object to a HashMap. (This choice of intermediate data structure may not be appropritae depending on how the user wants to interpret the object). By constrast, json-tokens converts a document to a token sequence.

• For numbers, aeson uses scientific, but json-tokens uses scientific-notation. Although scientific and scientific-notation have similar APIs, scientific-notation includes a parser that is about 4x faster and conversion functions that are 10x faster than those found in scientific and aeson.

• For text, aeson uses the UTF-16-backed text library, but json-tokens uses the UTF-8-backed text-short library.

• Parsing is resumable in aeson, which uses attoparsec, but not in json-tokens, which uses bytesmith.

• In aeson, all batteries are included. In particular, the combination of typeclasses and GHC Generics (or Template Haskell) make it possible to elide lots of boilerplate. None of these are included in json-tokens.

The difference in design decisions means that solutions using json-tokens are able to decode JSON twice as fast as solutions with `aeson. In the zeek-json benchmark suite, a json-tokens-based decoding outperforms aeson's decode by a factor of two. This speed comes at a cost. Users must write more code to use json-tokens than they do for aeson. If high-throughput parsing of small JSON documents is paramount, this cost may be worth bearing. It is always possible to go a step further and forego tokenization entirely, parsing the desired Haskell data type directly from a byte sequence. Doing this in a low-allocation way while retaining both the ability the handle JSON object keys in any order and the ability to handle escape sequences in object keys is fiendishly difficult. Kudos to the brave soul that goes down that path. For the rest of us, json-tokens is a compromise worth considering.

Modules

[Index] [Quick Jump]

Downloads

• json-tokens-0.1.0.1.tar.gz [browse] (Cabal source package)
• Package description (as included in the package)

Maintainer's Corner

For package maintainers and hackage trustees

• edit package information