streamly-lmdb: Stream data to or from LMDB databases using the streamly library.

[ bsd3, database, library, streaming, streamly ] [ Propose Tags ]

Please see the README on GitHub at https://github.com/shlok/streamly-lmdb#readme


[Skip to Readme]

Downloads

Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees

Candidates

Versions [RSS] 0.0.1, 0.0.1.1, 0.1.0, 0.2.0, 0.2.1, 0.3.0, 0.4.0, 0.5.0, 0.6.0, 0.7.0, 0.8.0
Change log ChangeLog.md
Dependencies async (>=2.2.2 && <2.3), base (>=4.7 && <5), bytestring (>=0.10.10.0 && <0.12), streamly (>=0.9 && <0.10), streamly-core (>=0.1 && <0.2) [details]
License BSD-3-Clause
Copyright 2023 Shlok Datye
Author Shlok Datye
Maintainer sd-haskell@quant.is
Category Database, Streaming, Streamly
Home page https://github.com/shlok/streamly-lmdb
Bug tracker https://github.com/shlok/streamly-lmdb/issues
Source repo head: git clone https://github.com/shlok/streamly-lmdb
Uploaded by shlok at 2023-05-05T07:49:29Z
Distributions
Downloads 1143 total (42 in the last 30 days)
Rating (no votes yet) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs uploaded by user
Build status unknown [no reports yet]

Readme for streamly-lmdb-0.7.0

[back to package description]

streamly-lmdb

Hackage CI

Stream data to or from LMDB databases using the Haskell streamly library.

Requirements

Install LMDB on your system:

  • Debian Linux: sudo apt-get install liblmdb-dev.
  • macOS: brew install lmdb.

Quick start

{-# LANGUAGE OverloadedStrings #-}

module Main where

import Data.Function ((&))
import qualified Streamly.Data.Fold as F
import qualified Streamly.Data.Stream.Prelude as S
import Streamly.External.LMDB
  ( Limits (mapSize),
    WriteOptions (writeTransactionSize),
    defaultLimits,
    defaultReadOptions,
    defaultWriteOptions,
    getDatabase,
    openEnvironment,
    readLMDB,
    tebibyte,
    writeLMDB,
  )

main :: IO ()
main = do
  -- Open an environment. There should already exist a file or
  -- directory at the given path. (Empty for a new environment.)
  env <-
    openEnvironment "/path/to/lmdb-database" $
      defaultLimits {mapSize = tebibyte}

  -- Get the main database.
  -- Note: It is common practice with LMDB to create the database
  -- once and reuse it for the remainder of the program’s execution.
  db <- getDatabase env Nothing

  -- Stream key-value pairs into the database.
  let fold' = writeLMDB db defaultWriteOptions {writeTransactionSize = 1}
  let writeStream = S.fromList [("baz", "a"), ("foo", "b"), ("bar", "c")]
  _ <- S.fold fold' writeStream

  -- Stream key-value pairs out of the
  -- database, printing them along the way.
  -- Output:
  --     ("bar","c")
  --     ("baz","a")
  --     ("foo","b")
  let unfold' = readLMDB db Nothing defaultReadOptions
  let readStream = S.unfold unfold' undefined
  S.mapM print readStream
    & S.fold F.drain

Benchmarks

See bench/README.md. Summary (with rough figures from our machine):

  • Reading:

    • For iterating through a fully cached LMDB database, this library has roughly a 110 ns/pair overhead compared to C. (Plain Haskell IO code has roughly a 70 ns/pair overhead compared to C. The two preceding figures being similar fulfills the promise of streamly and stream fusion.)
    • By using unsafeReadLMDB instead of readLMDB, we can get the overhead down to roughly 100 ns/pair.
    • By additionally using the readUnsafeFFI option (to use unsafe FFI calls under the hood), we can get the overhead down to roughly 40 ns/pair.
  • Writing:

    • For writing to an LMDB database, this library has roughly a 210 ns/pair overhead compared to C. (Plain Haskell IO code has roughly a 100 ns/pair overhead compared to C. The two preceding figures being similar fulfills the promise of streamly and stream fusion.)
    • By using the writeUnsafeFFI option (to use unsafe FFI calls under the hood), we can get the overhead down to roughly 140 ns/pair.
  • For most Haskell programs, these differences will not cause problems. (For instance, note that merely opening and reading 1 byte from a file with C already takes us tens of microseconds.)

May 2023; Linode; Debian 11, Dedicated 32GB: 16 CPU, 640GB SSD storage, 32GB RAM.