hw-json-simd: SIMD-based JSON semi-indexer

[ bsd3, data, library, program ] [ Propose Tags ]

Please see the README on GitHub at https://github.com/haskell-works/hw-json-simd#readme

[Skip to Readme]
Versions [faq],
Change log ChangeLog.md
Dependencies base (==4.*), bytestring (>=0.10.6 && <0.11), hw-json-simd, hw-prim (>= && <0.7), lens (==4.*), optparse-applicative (==0.14.*), semigroups (>=0.8.4 && <0.19), vector (==0.12.*) [details]
License BSD-3-Clause
Copyright 2018 - 2019 John Ky
Author John Ky
Maintainer newhoggy@gmail.com
Category Data
Home page https://github.com/haskell-works/hw-json-simd#readme
Bug tracker https://github.com/haskell-works/hw-json-simd/issues
Source repo head: git clone https://github.com/haskell-works/hw-json-simd
Uploaded by haskellworks at Wed Jan 30 22:32:29 UTC 2019
Distributions NixOS:, Stackage:
Executables hw-json-simd
Downloads 49 total (49 in the last 30 days)
Rating (no votes yet) [estimated by rule of succession]
Your Rating
  • λ
  • λ
  • λ
Status Hackage Matrix CI
Docs available [build log]
Last success reported on 2019-01-30 [all 1 reports]


[Index] [Quick Jump]



Enable avx2 instruction set


Enable bmi2 instruction set


Enable SSE 4.2 optimisations.


Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info


Maintainer's Corner

For package maintainers and hackage trustees

Readme for hw-json-simd-

[back to package description]


This library/tool will generate semi-indexes on JSON files as per the paper: Semi-Indexing Semi-Structured Data in Tiny Space.

The command line tool

For a given JSON file, hw-json-simd will generate two semi-index files, which both together can be loaded into a single in-memory semi-index.

The semi-index files can be generated using two methods, which will be called standard and simple, which correspond to sections 4 and 5 of the Semi-Indexing Semi-Structured Data in Tiny Space paper respectively.

Navigation of the JSON text using the standard index is suppored by the hw-json project for more information. There is currently no support for navigation of the JSON text using the simple index.

Using on the command line

cat test.json | pv -t -e -b -a | time hw-json-simd create-index \
  -i /dev/stdin
  --output-ib-file test.json.ib.idx
  --output-bp-file test.json.bp.idx
  --method standard
cat test.json | pv -t -e -b -a | time hw-json-simd create-index \
  -i /dev/stdin
  --output-ib-file test.json.ib.idx
  --output-bp-file test.json.bp.idx
  --method simple


From hackage

cabal new-install hw-json-simd