franz: Append-only database

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.

[maintain] [Publish]

Warnings:

Please see the README on GitHub at https://github.com/fumieval/franz#readme


[Skip to Readme]

Properties

Versions 0.2.1, 0.3, 0.3, 0.3.0.1
Change log ChangeLog.md
Dependencies base (>=4.7 && <5), bytestring, cereal, concurrent-resource-map (==0.2.*), containers, cpu, deepseq, directory, fast-builder (>=0.1.2.0 && <0.2), filepath, franz, fsnotify, mtl, network, optparse-applicative, process, retry, sendfile, stm, stm-delay, transformers, unboxed-ref, unordered-containers, vector [details]
License BSD-3-Clause
Copyright Copyright (c) 2019 Fumiaki Kinoshita
Author Fumiaki Kinoshita
Maintainer fumiexcel@gmail.com
Category Database
Home page https://github.com/fumieval/franz#readme
Bug tracker https://github.com/fumieval/franz/issues
Source repo head: git clone https://github.com/fumieval/franz
Uploaded by FumiakiKinoshita at 2020-06-03T04:00:21Z

Modules

Downloads

Maintainer's Corner

For package maintainers and hackage trustees


Readme for franz-0.3

[back to package description]

Franz

Haskell CI

Franz is an append-only container format, forked from liszt.

Each stream is stored as a pair of concatenated payloads with an array of their byte offsets.

Design requirements

Usecase

Format details

The on-disk representation of a franz stream comprises the following files:

A stream is stored as a directory containing the files above.

The Franz reader also supports a squashfs image, provided that the content is a valid franz stream.

franzd

franzd is a read-only server which follows franz files and gives access on wire. Where to look for streams can be specified as a command-line argument, separately for live streams and squashfs images.

Each stream is stored as a pair of concatenated payloads with an array of their byte offsets.

Why not Kafka

Design requirements

Usecase

Format details

The on-disk representation of a franz stream comprises the following files:

A stream is stored as a directory containing the files above.

The Franz reader also supports a squashfs image, provided that the content is a valid franz stream.

franzd

franzd is a read-only server which follows franz files and gives access on wire. Where to look for streams can be specified as a command-line argument, separately for live streams and squashfs images.

franzd --live /path/to/live --archive /path/to/archive

You can obtain a Connection to a remote franz file with withConnection. It tries to mount a squashfs image at path. This is shared between connections, and unmounts when the last client closes the connection.

withConnection :: (MonadIO m, MonadMask m)
  => String -- host
  -> Int -- port
  -> ByteString -- path
  -> (Connection -> m r) -> m r

fetch returns a list of triples of offsets, tags, and payloads.

data RequestType = AllItems | LastItem deriving (Show, Generic)

data ItemRef = BySeqNum !Int -- ^ sequential number
  | ByIndex !B.ByteString Int -- ^ index name and value

data Query = Query
  { reqStream :: !B.ByteString
  , reqFrom :: !ItemRef -- ^ name of the index to search
  , reqTo :: !ItemRef -- ^ name of the index to search
  , reqType :: !RequestType
  } deriving (Show, Generic)

type SomeIndexMap = HM.HashMap B.ByteString Int64

type Contents = [(Int, SomeIndexMap, B.ByteString)]

-- | When it is 'Right', it blocks until the content is available on the server.
type Response = Either Contents (STM Contents)

fetch :: Connection
  -> Query
  -> (STM Response -> IO r)
  -- ^ running the STM action blocks until the response arrives
  -> IO r

franz CLI: reading

Read 0th to 9th elements

franz test -r 0:9

Follow a stream

franz test -b _1
``