hw-json: Memory efficient JSON parser

[ bsd3, data, library, program ] [ Propose Tags ]

Memory efficient JSON parser. Please see README.md


[Skip to Readme]
Versions [faq] 0.0.0.1, 0.0.0.2, 0.0.0.3, 0.0.0.4, 0.1.0.0, 0.2.0.0, 0.2.0.1, 0.2.0.2, 0.2.0.3, 0.2.0.4, 0.3.0.0, 0.4.0.0, 0.5.0.0, 0.6.0.0, 0.7.0.1, 0.8.0.0, 0.8.1.0, 0.9.0.0, 0.9.0.1, 1.0.0.0, 1.0.0.1, 1.0.0.2, 1.1.0.0, 1.1.0.1, 1.1.0.2, 1.2.0.0, 1.2.0.1, 1.2.0.2, 1.3.0.0, 1.3.0.1, 1.3.1.0, 1.3.1.1, 1.3.2.0, 1.3.2.1
Dependencies aeson (>=1.4.3.0 && <1.5), ansi-wl-pprint (>=0.6.8.2 && <0.7), attoparsec (==0.13.*), base (>=4.11 && <5), bits-extra (>=0.0.1.3 && <0.1), bytestring (>=0.10.6 && <0.11), dlist (==0.8.*), generic-lens (>=1.2.0.1 && <2.1), hw-balancedparens (>=0.3.0.0 && <0.5), hw-bits (>=0.7.0.5 && <0.8), hw-json, hw-json-simd (>=0.1.0.2 && <0.2), hw-json-simple-cursor (>=0.1.0.1 && <0.2), hw-json-standard-cursor (>=0.2.0.1 && <0.3), hw-mquery (>=0.2.0.0 && <0.3), hw-parser (==0.1.*), hw-prim (>=0.6.2.32 && <0.7), hw-rankselect (==0.13.*), hw-rankselect-base (>=0.3.2.1 && <0.4), hw-simd (>=0.1.1.2 && <0.2), lens (==4.*), mmap (==0.5.*), optparse-applicative (>=0.14 && <0.16), text (==1.2.*), unordered-containers (==0.2.*), vector (==0.12.*), word8 (==0.1.*) [details]
License BSD-3-Clause
Copyright 2016-2020 John Ky
Author John Ky
Maintainer newhoggy@gmail.com
Revised Revision 2 made by newhoggy at 2020-04-26T14:32:46Z
Category Data
Home page http://github.com/haskell-works/hw-json#readme
Bug tracker https://github.com/haskell-works/hw-json/issues
Source repo head: git clone https://github.com/haskell-works/hw-json
Uploaded by haskellworks at 2020-04-14T05:34:28Z
Distributions LTSHaskell:0.9.0.1, NixOS:1.3.2.1, Stackage:1.0.0.2
Executables hw-json
Downloads 15147 total (486 in the last 30 days)
Rating 2.0 (votes: 1) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Hackage Matrix CI
Docs available [build log]
Last success reported on 2020-04-14 [all 1 reports]

Modules

[Index] [Quick Jump]

Flags

NameDescriptionDefaultType
bmi2

Enable bmi2 instruction set

DisabledAutomatic
sse42

Enable sse4.2 instruction set

DisabledAutomatic

Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info

Downloads

Note: This package has metadata revisions in the cabal description newer than included in the tarball. To unpack the package including the revisions, use 'cabal get'.

Maintainer's Corner

For package maintainers and hackage trustees


Readme for hw-json-1.3.2.0

[back to package description]

hw-json

master

hw-json is a succinct JSON parsing library.

It uses succinct data-structures to allow traversal of large JSON strings with minimal memory overhead.

For an example, see app/Main.hs

Prerequisites

  • cabal version 2.2 or later

Memory benchmark

Parsing large Json files in Scala with Argonaut

      S0U       EU           OU       MU     CCSU CMD
--------- --------- ----------- -------- -------- ---------------------------------------------------------------
      0.0  80,526.3    76,163.6 72,338.6 13,058.6 sbt console
      0.0 536,660.4    76,163.6 72,338.6 13,058.6 import java.io._, argonaut._, Argonaut._
      0.0 552,389.1    76,163.6 72,338.6 13,058.6 val file = new File("/Users/jky/Downloads/78mbs.json"
      0.0 634,066.5    76,163.6 72,338.6 13,058.6 val array = new Array[Byte](file.length.asInstanceOf[Int])
      0.0 644,552.3    76,163.6 72,338.6 13,058.6 val is = new FileInputStream("/Users/jky/Downloads/78mbs.json")
      0.0 655,038.1    76,163.6 72,338.6 13,058.6 is.read(array)
294,976.0 160,159.7 1,100,365.0 79,310.8 13,748.1 val json = new String(array)
285,182.9 146,392.6 1,956,264.5 82,679.8 14,099.6 val data = Parse.parse(json)
                    ***********

Parsing large Json files in Haskell with Aeson

-- CMD                                                     -- Mem (MB)
---------------------------------------------------------- -- --------
import Control.DeepSeq                                     --      345
import Data.Aeson                                          --      371
import qualified Data.ByteString.Lazy as LBS               --      376
!bs <- LBS.readFile "corpus/bench/hospitalisation.json"    --      380
let !y = decode bs :: Maybe Value                          --      928

Parsing large Json files in Haskell with hw-json

-- CMD                                                                -- Mem (MB)
--------------------------------------------------------------------- -- --------
import qualified HaskellWorks.Data.ByteString                as BS    --      351
import qualified HaskellWorks.Data.Json.Standard.Cursor.Fast as JCF   --      353

!jsonBs <- BS.mmap "corpus/bench/hospitalisation.json"                --      355
let !ibip = JCF.simdToIbBp jsonBs                                     --      358
let !c    = JCF.fromBsIbBp jsonBs ibip                                --      495

Examples

Navigation example

import Control.Monad

import qualified Data.ByteString                             as BS
import qualified HaskellWorks.Data.Json.Standard.Cursor.Fast as JCF
import qualified HaskellWorks.Data.TreeCursor                as TC

let fc = TC.firstChild
let ns = TC.nextSibling
let pn = TC.parent
let ss = TC.subtreeSize
let jsonBs  = "[null, {\"field\": 1}]" :: BS.ByteString
let ibip    = JCF.simdToIbBp jsonBs
let cursor  = JCF.fromBsIbBp jsonBs ibip
fc cursor
(fc >=> ns) cursor

Querying example

import Control.Monad
import Data.Function
import Data.List
import HaskellWorks.Data.Json.PartialValue
import HaskellWorks.Data.Json.Standard.Cursor.Load.Cursor
import HaskellWorks.Data.Json.Standard.Load.Partial
import HaskellWorks.Data.MQuery
import HaskellWorks.Data.MQuery.Micro

import qualified Data.DList as DL

!cursor <- loadPartial "corpus/bench/78mb.json"
!cursor <- loadCursorWithIndex "corpus/bench/78mb.json"
!cursor <- loadCursor "corpus/bench/78mb.json"
!cursor <- loadCursorWithCsPoppyIndex "corpus/bench/78mb.json"
let !json = jsonPartialJsonValueAt cursor
let q = MQuery (DL.singleton json)

putPretty $ q >>= item & limit 10
putPretty $ q >>= item & page 10 1
putPretty $ q >>= item >>= hasKV "founded_year" (JsonPartialNumber 2005) & limit 10
putPretty $ q >>= item >>= entry
putPretty $ q >>= item >>= entry >>= named "name" & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code")
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code") & onList (uniq . sort)
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount") & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount" >=> castAsInteger ) & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount" >=> castAsInteger ) & aggregate sum

putPretty $ q >>= item & limit 10
putPretty $ q >>= item & page 10 1
putPretty $ q >>= item >>= entry
putPretty $ q >>= item >>= entry >>= named "name" & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code" >=> asString)
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code" >=> asString) & onList (uniq . sort)
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount") & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount" >=> castAsInteger ) & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount" >=> castAsInteger ) & aggregate sum

References

Special mentions