hw-json: Conduits for tokenizing streams.

[ bsd3, data, library, program ] [ Propose Tags ]

Please see README.md


[Skip to Readme]

Modules

[Last Documentation]

  • HaskellWorks
    • Data
      • HaskellWorks.Data.Json
        • HaskellWorks.Data.Json.Extract
        • HaskellWorks.Data.Json.Succinct
          • HaskellWorks.Data.Json.Succinct.Cursor
            • HaskellWorks.Data.Json.Succinct.Cursor.BalancedParens
            • HaskellWorks.Data.Json.Succinct.Cursor.BlankedJson
            • HaskellWorks.Data.Json.Succinct.Cursor.CursorType
            • HaskellWorks.Data.Json.Succinct.Cursor.FromJsonCursor
            • HaskellWorks.Data.Json.Succinct.Cursor.InterestBits
            • HaskellWorks.Data.Json.Succinct.Cursor.Internal
            • HaskellWorks.Data.Json.Succinct.Cursor.Token
        • HaskellWorks.Data.Json.Token
          • HaskellWorks.Data.Json.Token.Tokenize
          • HaskellWorks.Data.Json.Token.Types
        • HaskellWorks.Data.Json.Type
        • HaskellWorks.Data.Json.Value

Downloads

Note: This package has metadata revisions in the cabal description newer than included in the tarball. To unpack the package including the revisions, use 'cabal get'.

Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees

Candidates

Versions [RSS] 0.0.0.1, 0.0.0.2, 0.0.0.3, 0.0.0.4, 0.1.0.0, 0.2.0.0, 0.2.0.1, 0.2.0.2, 0.2.0.3, 0.2.0.4, 0.3.0.0, 0.4.0.0, 0.5.0.0, 0.6.0.0, 0.7.0.1, 0.8.0.0, 0.8.1.0, 0.9.0.0, 0.9.0.1, 1.0.0.0, 1.0.0.1, 1.0.0.2, 1.1.0.0, 1.1.0.1, 1.1.0.2, 1.2.0.0, 1.2.0.1, 1.2.0.2, 1.3.0.0, 1.3.0.1, 1.3.1.0, 1.3.1.1, 1.3.2.0, 1.3.2.1, 1.3.2.2, 1.3.2.3, 1.3.2.4, 1.3.2.5 (info)
Dependencies attoparsec, base (>=4.7 && <5), bytestring, conduit, containers, criterion, hw-bits, hw-conduit (<0.0.0.11), hw-diagnostics, hw-json, hw-parser, hw-prim (<0.0.0.11), hw-rankselect (<0.0.0.3), mmap, mono-traversable, resourcet, text, vector, word8 [details]
License BSD-3-Clause
Copyright 2016 John Ky
Author John Ky
Maintainer newhoggy@gmail.com
Revised Revision 1 made by GeorgeWilson at 2018-07-12T01:19:57Z
Category Data, Conduit
Home page http://github.com/haskell-works/hw-json#readme
Source repo head: git clone https://github.com/haskell-works/hw-json
Uploaded by haskellworks at 2016-05-06T09:13:36Z
Distributions
Reverse Dependencies 5 direct, 2 indirect [details]
Executables hw-json-example
Downloads 21394 total (91 in the last 30 days)
Rating 2.0 (votes: 1) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs not available [build log]
All reported builds failed as of 2020-10-29 [all 5 reports]

Readme for hw-json-0.0.0.1

[back to package description]

hw-json

v0.0-branch

Conduits for tokenizing streams.

hw-json is a succinct JSON parsing library. It uses succinct data-structures to allow traversal of large JSON strings with minimal memory overhead.

It is currently considered experimental.

For an example, see app/Main.hs

Prerequisites

  • Install haskell-stack.
  • Install hlint (eg. stack install hlint)

Building

Run the following in the shell:

git clone git@github.com:haskell-works/hw-json.git
cd hw-json
stack setup
stack build
stack test
stack ghci --ghc-options -XOverloadedStrings \
  --main-is hw-json:exe:hw-json-example

Memory benchmark

Parsing large Json files in Scala with Argonaut

      S0U       EU           OU       MU     CCSU CMD
--------- --------- ----------- -------- -------- ---------------------------------------------------------------
      0.0  80,526.3    76,163.6 72,338.6 13,058.6 sbt console
      0.0 536,660.4    76,163.6 72,338.6 13,058.6 import java.io._, argonaut._, Argonaut._
      0.0 552,389.1    76,163.6 72,338.6 13,058.6 val file = new File("/Users/jky/Downloads/78mbs.json"
      0.0 634,066.5    76,163.6 72,338.6 13,058.6 val array = new Array[Byte](file.length.asInstanceOf[Int])
      0.0 644,552.3    76,163.6 72,338.6 13,058.6 val is = new FileInputStream("/Users/jky/Downloads/78mbs.json")
      0.0 655,038.1    76,163.6 72,338.6 13,058.6 is.read(array)
294,976.0 160,159.7 1,100,365.0 79,310.8 13,748.1 val json = new String(array)
285,182.9 146,392.6 1,956,264.5 82,679.8 14,099.6 val data = Parse.parse(json)
                    ***********

Parsing large Json files in Haskell with Aeson

Mem (MB) CMD
-------- ---------------------------------------------------------
     302 import Data.Aeson
     302 import qualified  Data.ByteString.Lazy as BSL
     302 json78m <- BSL.readFile "/Users/jky/Downloads/78mbs.json"
    1400 let !x = decode json78m :: Maybe Value

Parsing large Json files in Haskell with hw-json

Mem (MB) CMD
-------- ---------------------------------------------------------
     274 import Foreign
     274 import qualified Data.Vector.Storable as DVS
     274 import qualified Data.ByteString as BS
     274 import System.IO.MMap
     274 import Data.Word
     274 (fptr :: ForeignPtr Word8, offset, size) <- mmapFileForeignPtr "/Users/jky/Downloads/78mbs.json" ReadOnly Nothing
     601 cursor <- measure (fromForeignRegion (fptr, offset, size) :: JsonCursor BS.ByteString (BitShown (DVS.Vector Word64)) (SimpleBalancedParens (DVS.Vector Word64)))

Examples

Performance example

import Foreign
import qualified Data.Vector.Storable as DVS
import qualified Data.ByteString as BS
import qualified Data.ByteString.Internal as BSI
import System.IO.MMap
import Data.Word
import System.CPUTime
(fptr :: ForeignPtr Word8, offset, size) <- mmapFileForeignPtr "/Users/jky/Downloads/78mbs.json" ReadOnly Nothing
cursor <- measure (fromForeignRegion (fptr, offset, size) :: JsonCursor BS.ByteString (BitShown (DVS.Vector Word64)) (SimpleBalancedParens (DVS.Vector Word64)))
let !bs = BSI.fromForeignPtr (castForeignPtr fptr) offset size
x <- measure $ jsonBsToInterestBs bs
let !y = runListConduit [bs] (unescape' "")

import Foreign
import qualified Data.Vector.Storable as DVS
import qualified Data.ByteString as BS
import qualified Data.ByteString.Internal as BSI
import System.IO.MMap
import Data.Word
import System.CPUTime
(fptr :: ForeignPtr Word8, offset, size) <- mmapFileForeignPtr "/Users/jky/Downloads/part40.json" ReadOnly Nothing
let !bs = BSI.fromForeignPtr (castForeignPtr fptr) offset size
x <- measure $ BS.concat $ runListConduit [bs] (blankJson =$= blankedJsonToInterestBits)
x <- measure $ jsonBsToInterestBs bs

jsonTokenAt $ J.nextSibling $ J.firstChild $ J.nextSibling $ J.firstChild $ J.firstChild  cursor
$  cabal repl --ghc-option='-package mmap'
λ> :set -XNoMonomorphismRestriction
λ> import qualified Data.ByteString                                            as BS
λ> import           Data.String
λ> import qualified Data.Vector.Storable                                       as DVS
λ> import           Data.Word
λ> import           HaskellWorks.Data.Bits.BitShow
λ> import           HaskellWorks.Data.Bits.BitShown
λ> import           HaskellWorks.Data.FromForeignRegion
λ> import           HaskellWorks.Data.Json.Succinct.Cursor                     as C
λ> import           HaskellWorks.Data.Json.Token
λ> import           HaskellWorks.Data.Succinct.BalancedParens.Internal
λ> import           HaskellWorks.Data.Succinct.BalancedParens.Simple
λ> import           HaskellWorks.Data.Succinct.RankSelect.Binary.Basic.Rank0
λ> import           HaskellWorks.Data.Succinct.RankSelect.Binary.Basic.Rank1
λ> import           HaskellWorks.Data.Succinct.RankSelect.Binary.Basic.Select1
λ> import           HaskellWorks.Data.Succinct.RankSelect.Binary.Poppy512
λ> import qualified HaskellWorks.Data.TreeCursor as TC
λ> import           System.IO.MMap
λ> let fc = TC.firstChild
λ> let ns = TC.nextSibling
λ> let pn = TC.parent
λ> let cd = TC.depth
λ> let ss = TC.subtreeSize
λ> let cursor = "[null, {\"field\": 1}]" :: JsonCursor BS.ByteString (BitShown (DVS.Vector Word64)) (SimpleBalancedParens (DVS.Vector Word64))
λ> cursor

References

Special mentions