Streamly.External.LMDB

The functionality for the limits and getting the environment and database were mostly obtained from the lmdb-simple library.

Types

data Database mode

data Environment mode

data Limits

LMDB environments have various limits on the size and number of databases and concurrent readers.

Constructors

Limits

Fields

• mapSize :: !Int

  Memory map size, in bytes (also the maximum size of all databases).

• maxDatabases :: !Int

  Maximum number of named databases.

• maxReaders :: !Int

  Maximum number of concurrent ReadOnly transactions (also the number of slots in the lock table).

data LMDB_Error

Constructors

LMDB_Error

Fields

• e_context :: String
• e_description :: String
• e_code :: Either Int MDB_ErrCode

Instances

Show LMDB_Error

Defined in Streamly.External.LMDB.Internal.Foreign

Methods

showsPrec :: Int -> LMDB_Error -> ShowS

show :: LMDB_Error -> String

showList :: [LMDB_Error] -> ShowS

Exception LMDB_Error

Defined in Streamly.External.LMDB.Internal.Foreign

Methods

toException :: LMDB_Error -> SomeException

fromException :: SomeException -> Maybe LMDB_Error

displayException :: LMDB_Error -> String

data MDB_ErrCode

Constructors

MDB_KEYEXIST

MDB_NOTFOUND

MDB_PAGE_NOTFOUND

MDB_CORRUPTED

MDB_PANIC

MDB_VERSION_MISMATCH

MDB_INVALID

MDB_MAP_FULL

MDB_DBS_FULL

MDB_READERS_FULL

MDB_TLS_FULL

MDB_TXN_FULL

MDB_CURSOR_FULL

MDB_PAGE_FULL

MDB_MAP_RESIZED

MDB_INCOMPATIBLE

MDB_BAD_RSLOT

MDB_BAD_TXN

MDB_BAD_VALSIZE

MDB_BAD_DBI

Instances

Eq MDB_ErrCode

Defined in Streamly.External.LMDB.Internal.Foreign

Methods

(==) :: MDB_ErrCode -> MDB_ErrCode -> Bool

(/=) :: MDB_ErrCode -> MDB_ErrCode -> Bool

Show MDB_ErrCode

Defined in Streamly.External.LMDB.Internal.Foreign

Methods

showsPrec :: Int -> MDB_ErrCode -> ShowS

show :: MDB_ErrCode -> String

showList :: [MDB_ErrCode] -> ShowS

class Mode a

Minimal complete definition

isReadOnlyMode

Instances

Mode ReadOnly

Defined in Streamly.External.LMDB.Internal

Methods

isReadOnlyMode :: ReadOnly -> Bool

Mode ReadWrite

Defined in Streamly.External.LMDB.Internal

Methods

isReadOnlyMode :: ReadWrite -> Bool

data ReadWrite

Instances

Mode ReadWrite

Defined in Streamly.External.LMDB.Internal

Methods

isReadOnlyMode :: ReadWrite -> Bool

data ReadOnly

Instances

Mode ReadOnly

Defined in Streamly.External.LMDB.Internal

Methods

isReadOnlyMode :: ReadOnly -> Bool

data ReadDirection

Direction of key iteration.

Constructors

Forward

Backward

Instances

Show ReadDirection

Defined in Streamly.External.LMDB

Methods

showsPrec :: Int -> ReadDirection -> ShowS

show :: ReadDirection -> String

showList :: [ReadDirection] -> ShowS

data ReadOptions

Constructors

ReadOptions

Fields

• readDirection :: !ReadDirection
• readStart :: !(Maybe ByteString)

  If Nothing, a forward [backward] iteration starts at the beginning [end] of the database. Otherwise, it starts at the first key that is greater [less] than or equal to the Just key.

Instances

Show ReadOptions

Defined in Streamly.External.LMDB

Methods

showsPrec :: Int -> ReadOptions -> ShowS

show :: ReadOptions -> String

showList :: [ReadOptions] -> ShowS

data OverwriteOptions

Constructors

OverwriteAllow

OverwriteAllowSame

OverwriteDisallow

Instances

Eq OverwriteOptions

Defined in Streamly.External.LMDB

Methods

(==) :: OverwriteOptions -> OverwriteOptions -> Bool

(/=) :: OverwriteOptions -> OverwriteOptions -> Bool

data WriteOptions

Constructors

WriteOptions

Fields

• writeTransactionSize :: !Int
• overwriteOptions :: !OverwriteOptions
• writeAppend :: !Bool

Environment and database

defaultLimits :: Limits

The default limits are 1 MiB map size, 0 named databases, and 126 concurrent readers. These can be adjusted freely, and in particular the mapSize may be set very large (limited only by available address space). However, LMDB is not optimized for a large number of named databases so maxDatabases should be kept to a minimum.

The default mapSize is intentionally small, and should be changed to something appropriate for your application. It ought to be a multiple of the OS page size, and should be chosen as large as possible to accommodate future growth of the database(s). Once set for an environment, this limit cannot be reduced to a value smaller than the space already consumed by the environment, however it can later be increased.

If you are going to use any named databases then you will need to change maxDatabases to the number of named databases you plan to use. However, you do not need to change this field if you are only going to use the single main (unnamed) database.

openEnvironment :: Mode mode => FilePath -> Limits -> IO (Environment mode)

Open an LMDB environment in either ReadWrite or ReadOnly mode. The FilePath argument may be either a directory or a regular file, but it must already exist. If a regular file, an additional file with "-lock" appended to the name is used for the reader lock table.

Note that an environment must have been opened in ReadWrite mode at least once before it can be opened in ReadOnly mode.

An environment opened in ReadOnly mode may still modify the reader lock table (except when the filesystem is read-only, in which case no locks are used).

isReadOnlyEnvironment :: Mode mode => Environment mode -> Bool

getDatabase :: Mode mode => Environment mode -> Maybe String -> IO (Database mode)

Utility

gibibyte :: Int

tebibyte :: Int

clearDatabase :: Mode mode => Database mode -> IO ()

Clears, i.e., removes all key-value pairs from, the given database.

Reading

defaultReadOptions :: ReadOptions

readLMDB :: (MonadIO m, Mode mode) => Database mode -> ReadOptions -> Unfold m Void (ByteString, ByteString)

Creates an unfold with which we can stream key-value pairs from the given database.

A read transaction is kept open for the duration of the unfold; one should therefore bear in mind LMDB's caveats regarding long-lived transactions.

If you don’t want the overhead of intermediate ByteStrings (on your way to your eventual data structures), use unsafeReadLMDB instead.

unsafeReadLMDB :: (MonadIO m, Mode mode) => Database mode -> ReadOptions -> (CStringLen -> IO k) -> (CStringLen -> IO v) -> Unfold m Void (k, v)

Creates an unfold with which we can stream key-value pairs from the given database.

A read transaction is kept open for the duration of the unfold; one should therefore bear in mind LMDB's caveats regarding long-lived transactions.

To ensure safety, make sure that the memory pointed to by the CStringLen for each key/value mapping function call is (a) only read (and not written to); and (b) not used after the mapping function has returned. One way to transform the CStringLens to your desired data structures is to use unsafePackCStringLen.

Writing

defaultWriteOptions :: WriteOptions

writeLMDB :: MonadIO m => Database ReadWrite -> WriteOptions -> Fold m (ByteString, ByteString) ()

Creates a fold with which we can stream key-value pairs into the given database.

It is the responsibility of the user to execute the fold on a bound thread.

The fold currently cannot be used with a scan. (The plan is for this shortcoming to be remedied with or after a future release of streamly that addresses the underlying issue.)

Please specify a suitable transaction size in the write options; the default of 1 (one write transaction for each key-value pair) could yield suboptimal performance. One could try, e.g., 100 KB chunks and benchmark from there.