hw-json-simd: SIMD-based JSON semi-indexer

[ bsd3, data, library, program ] [ Propose Tags ]

Please see the README on GitHub at https://github.com/haskell-works/hw-json-simd#readme

[Skip to Readme]
Versions [faq],,,,,
Change log ChangeLog.md
Dependencies base (>=4.11 && <5), bytestring (>=0.10.6 && <0.11), hw-json-simd, hw-prim (>= && <0.7), lens (==4.*), optparse-applicative (>=0.14 && <0.17), vector (==0.12.*) [details]
License BSD-3-Clause
Copyright 2018-2020 John Ky
Author John Ky
Maintainer newhoggy@gmail.com
Revised Revision 1 made by newhoggy at 2020-08-17T12:39:20Z
Category Data
Home page https://github.com/haskell-works/hw-json-simd#readme
Bug tracker https://github.com/haskell-works/hw-json-simd/issues
Source repo head: git clone https://github.com/haskell-works/hw-json-simd
Uploaded by haskellworks at 2020-04-14T05:08:43Z
Distributions NixOS:, Stackage:
Executables hw-json-simd
Downloads 2001 total (26 in the last 30 days)
Rating (no votes yet) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Hackage Matrix CI
Docs available [build log]
Last success reported on 2020-04-14 [all 1 reports]


[Index] [Quick Jump]



Enable avx2 instruction set


Enable bmi2 instruction set


Enable SSE 4.2 optimisations.


Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info


Note: This package has metadata revisions in the cabal description newer than included in the tarball. To unpack the package including the revisions, use 'cabal get'.

Maintainer's Corner

For package maintainers and hackage trustees

Readme for hw-json-simd-

[back to package description]


This library/tool will generate semi-indexes on JSON files as per the paper: Semi-Indexing Semi-Structured Data in Tiny Space.

The command line tool

For a given JSON file, hw-json-simd will generate two semi-index files, which both together can be loaded into a single in-memory semi-index.

The semi-index files can be generated using two methods, which will be called standard and simple, which correspond to sections 4 and 5 of the Semi-Indexing Semi-Structured Data in Tiny Space paper respectively.

Navigation of the JSON text using the standard index is suppored by the hw-json project for more information. There is currently no support for navigation of the JSON text using the simple index.

Using on the command line

cat test.json | pv -t -e -b -a | time hw-json-simd create-index \
  -i /dev/stdin
  --output-ib-file test.json.ib.idx
  --output-bp-file test.json.bp.idx
  --method standard
cat test.json | pv -t -e -b -a | time hw-json-simd create-index \
  -i /dev/stdin
  --output-ib-file test.json.ib.idx
  --output-bp-file test.json.bp.idx
  --method simple


From hackage

cabal new-install hw-json-simd