System.TrailDB

Haskell bindings to trailDB library.

Minimal program that lists a TrailDB:

  import qualified Data.ByteString as B
  import System.TrailDB

  main :: IO ()
  main = do
    tdb <- openTrailDB "wikipedia-history-small.tdb"
    forEachTrailID tdb $ tid -> print =<< getTrailBytestring tdb tid

Example program that reads a TrailDB using low-level (faster) cursor API:

  import qualified Data.ByteString as B
  import qualified Data.Vector.Unboxed as V
  import System.TrailDB

  main :: IO ()
  main = do
    tdb <- openTrailDB "some-trail-db"
    number_of_trails <- getNumTrails tdb

    let arbitrarily_chosen_trail_id = 12345 `mod` number_of_trails

    cursor <- makeCursor tdb
    setCursor cursor arbitrarily_chosen_trail_id

    -- Read the first event in the arbitrary chosen trail
    crumb <- stepCursor cursor
    case crumb of
      Nothing -> putStrLn "Cannot find this particular trail."
      Just (timestamp, features) ->
        V.forM_ features $ feature -> do
          field_name <- getFieldName tdb (feature^.field)
          putStr "Field: "
          B.putStr field_name
          putStr " contains value "
          value <- getValue tdb feature
          B.putStrLn value

Another example program that writes a TrailDB:

  {-# LANGUAGE OverloadedStrings #-}

  import System.TrailDB

  main :: IO ()
  main = do
    cons <- newTrailDBCons "some-trail-db" (["currency", "order_amount", "item"] :: [String])
    addTrail cons ("aaaaaaaaaaaaaa00")   -- UUIDs are 16 bytes in length
                  1457049455             -- This is timestamp
                  ["USD", "10.14", "Bacon & Cheese" :: String]
    addTrail cons ("aaaaaaaaaaaaaa00")   -- Same UUID as above, same customer ordered more
                  1457051221
                  ["USD", "8.90", "Avocado Sandwich" :: String]
    addTrail cons ("aaaaaaaaaaaaaa02")
                  1457031239
                  ["JPY", "2900", "Sun Lotion" :: String]
    closeTrailDBCons cons

    -- TrailDB has been written to 'some-trail-db'

Constructing new TrailDBs

newTrailDBCons :: (FieldNameLike a, MonadIO m) => FilePath -> [a] -> m TdbCons

Create a new TrailDB and return TrailDB construction handle.

Close it with closeTrailDBCons. Garbage collector will close it eventually if you didn't do it yourself. You won't be receiving FinalizationFailure exception though if that fails when using the garbage collector.

closeTrailDBCons :: MonadIO m => TdbCons -> m ()

Close a TdbCons

Does nothing if it's closed already.

withTrailDBCons :: (FieldNameLike a, MonadIO m, MonadMask m) => FilePath -> [a] -> (TdbCons -> m b) -> m b

Runs an IO action with an opened TdbCons. The TdbCons is closed after the action has been executed.

addTrail :: (MonadIO m, ToTdbRow r) => TdbCons -> UUID -> UnixTime -> r -> m ()

Add a cookie with timestamp and values to TdbCons.

appendTdbToTdbCons :: MonadIO m => Tdb -> TdbCons -> m ()

Appends a Tdb to an open TdbCons.

finalizeTrailDBCons :: MonadIO m => TdbCons -> m ()

Finalizes a TdbCons.

You usually don't need to call this manually because this is called automatically by closeTrailDBCons before actually closing TdbCons.

Supplying values to construction

class ToTdbRow r where

Class of things that can be turned into rows and added with addTrail.

The native type inside traildb is the ByteString. The use of this typeclass can eliminate some noise when converting values to ByteString.

Methods

toTdbRow :: r -> [ByteString]

Instances

ToTdbRowField f => ToTdbRow [f]

(ToTdbRowField f1, ToTdbRowField f2) => ToTdbRow (f1, f2)

(ToTdbRowField a, ToTdbRow b) => ToTdbRow (TdbConsRow a b)

(ToTdbRowField f1, ToTdbRowField f2, ToTdbRowField f3) => ToTdbRow (f1, f2, f3)

(ToTdbRowField f1, ToTdbRowField f2, ToTdbRowField f3, ToTdbRowField f4) => ToTdbRow (f1, f2, f3, f4)

(ToTdbRowField f1, ToTdbRowField f2, ToTdbRowField f3, ToTdbRowField f4, ToTdbRowField f5) => ToTdbRow (f1, f2, f3, f4, f5)

class ToTdbRowField f where

Class of things that can be turned into a field in a TrailDB.

Methods

toTdbField :: f -> ByteString

Instances

ToTdbRowField String

ToTdbRowField ByteString

ToTdbRowField ByteString

ToTdbRowField Text

ToTdbRowField Text

Show a => ToTdbRowField (TdbShowable a)

data TdbConsRow a b

Convenience type that lets you arbitrarily make heterogenous list of things that implement ToTdbRowField and subsequently ToTdbRow. Use this if plain lists are not suitable (because they are monotyped) or your tuples are too long to implement ToTdbRow.

Constructors

(:.) a b infixr 7

Instances

(ToTdbRowField a, ToTdbRow b) => ToTdbRow (TdbConsRow a b)

newtype TdbShowable a

Convenience newtype to put things that you can Show in TrailDB. It implements ToTdbRowField.

Constructors

TdbShowable a

Instances

Functor TdbShowable

Foldable TdbShowable

Traversable TdbShowable

Eq a => Eq (TdbShowable a)

Ord a => Ord (TdbShowable a)

Read a => Read (TdbShowable a)

Show a => Show (TdbShowable a)

Generic (TdbShowable a)

Show a => ToTdbRowField (TdbShowable a)

type Rep (TdbShowable a)

pattern TShow :: t -> TdbShowable t

Short-cut pattern synonym for TdbShowable

Opening existing TrailDBs

openTrailDB :: MonadIO m => FilePath -> m Tdb

Opens an existing TrailDB.

It can open file TrailDBs and also directory TrailDBs.

In case of files you can use format "traildb.tdb" or just "traildb".

closeTrailDB :: MonadIO m => Tdb -> m ()

Closes a TrailDB.

Does nothing if Tdb is already closed.

getTdbVersion :: MonadIO m => Tdb -> m TdbVersion

dontneedTrailDB :: MonadIO m => Tdb -> m ()

Hints that Tdb will not be accessed in near future.

Internally may invoke system call 'madvise' behind the scenes to operating system.

This has no effect on semantics, only performance.

willneedTrailDB :: MonadIO m => Tdb -> m ()

Hints that Tdb will be walked over in near future.

Internally may invoke system call 'madvise' behind the scenes to operating system.

This has no effect on semantics, only performance.

withTrailDB :: (MonadIO m, MonadMask m) => FilePath -> (Tdb -> m a) -> m a

Opens a Tdb and then closes it after action is over.

Accessing TrailDBs

High-level, slow, access

class FromTrail a where

Class of things that can be result from getTrail.

Methods

fromBytestringList :: [(UnixTime, [(ByteString, ByteString)])] -> a

Makes a result from a list of trails.

One crumb of a trail has timestamp and associative list of fields.

    [(timestamp, [(fieldname1, value1), (fieldname2, value2), ...]
    , ... ]

Instances

FromTrail [[ByteString]]

Throws away timestamps and field names.

FromTrail [(UnixTime, [(ByteString, ByteString)])]

FromTrail [Map ByteString ByteString]

Throws away timestamps.

FromTrail (Set ByteString)

Set of all values that appear in the trail.

FromTrail (Vector (Map ByteString ByteString))

Vector version.

getTrail :: (FromTrail a, MonadIO m) => Tdb -> TrailID -> m a

Convenience function that returns a full trail in human-readable format.

This is quite a bit slower than using a cursor (makeCursor, setCursor, stepCursor) but is simpler. If you don't need to go through bazillions of data really fast you might want to use this one.

See FromTrail for things that can be taken as a result.

getTrailBytestring :: MonadIO m => Tdb -> TrailID -> m [(UnixTime, [(ByteString, ByteString)])]

Same as getTrail but not polymorphic in output.

Meant to be used for very quick throw-away programs where you don't want to spell out the type to be used (e.g. if you all you do is print the trail).

Lowerish-level, fast, access

makeCursor :: MonadIO m => Tdb -> m Cursor

Creates a cursor to a trailDB.

stepCursor :: MonadIO m => Cursor -> m (Maybe Crumb)

Steps cursor forward in its trail.

Returns Nothing if there are no more crumbs in the trail.

setCursor :: MonadIO m => Cursor -> TrailID -> m ()

Puts cursor at the start of some trail.

Iterating over TrailDB

forEachTrailID :: (Applicative m, MonadIO m) => Tdb -> (TrailID -> m ()) -> m ()

Convenience function that runs a function for each TrailID in TrailDB.

forEachTrailIDUUID :: (Applicative m, MonadIO m) => Tdb -> (TrailID -> UUID -> m ()) -> m ()

Same as forEachTrailID but passes UUID as well.

traverseEachTrailID :: (Applicative m, MonadIO m) => (TrailID -> m ()) -> Tdb -> m ()

Same as forEachTrailID but arguments flipped.

traverseEachTrailIDUUID :: (Applicative m, MonadIO m) => (TrailID -> UUID -> m ()) -> Tdb -> m ()

Same as traverseEachTrailID but passes UUID as well.

foldTrailDB :: MonadIO m => (a -> TrailID -> m a) -> a -> Tdb -> m a

Fold TrailDB for each TrailID.

This is like traverseEachTrailID but lets you carry a folding value.

foldTrailDBUUID :: MonadIO m => (a -> TrailID -> UUID -> m a) -> a -> Tdb -> m a

Same as foldTrailDB but passes UUID as well.

Basic querying

getNumTrails :: MonadIO m => Tdb -> m Word64

Returns the number of cookies in Tdb

getNumEvents :: MonadIO m => Tdb -> m Word64

Returns the number of events in Tdb

getNumFields :: MonadIO m => Tdb -> m Word64

Returns the number of fields in Tdb

getMinTimestamp :: MonadIO m => Tdb -> m UnixTime

Returns the minimum timestamp in Tdb

getMaxTimestamp :: MonadIO m => Tdb -> m UnixTime

Returns the maximum timestamp in Tdb

UUID handling

getUUID :: MonadIO m => Tdb -> TrailID -> m UUID

Finds a uuid by trail ID.

getTrailID :: MonadIO m => Tdb -> UUID -> m TrailID

Finds a trail ID by uuid.

Fields

getFieldName :: MonadIO m => Tdb -> FieldID -> m FieldName

Given a field ID, returns its human-readable field name.

getFieldID :: (FieldNameLike a, MonadIO m) => Tdb -> a -> m FieldID

Given a field name, returns its FieldID.

getItemByField :: (FieldNameLike a, MonadIO m) => Tdb -> a -> ByteString -> m Feature

Same as getItem but uses a resolved field name rather than raw FieldID.

This is implemented in terms of getFieldID and getItem inside.

getValue :: MonadIO m => Tdb -> Feature -> m ByteString

Given a Feature, returns a string that describes it.

Values in a TrailDB are integers which need to be mapped back to strings to be human-readable.

getItem :: MonadIO m => Tdb -> FieldID -> ByteString -> m Feature

Given a field ID and a human-readable value, turn it into Feature for that field ID.

Time handling

utcTimeToUnixTime :: UTCTime -> UnixTime

Converts UTCTime to UnixTime

posixSecondsToUnixTime :: POSIXTime -> UnixTime

Converts POSIXTime to UnixTime

dayToUnixTime :: Day -> UnixTime

Converts Day to UnixTime.

The time will be the first second of the Day.

C interop

withRawTdb :: MonadIO m => Tdb -> (Ptr TdbRaw -> IO a) -> m a

Run an action with a raw pointer to Tdb.

The Tdb is guaranteed not to be garbage collected while the given action is running.

getRawTdb :: MonadIO m => Tdb -> m (Ptr TdbRaw)

Returns the raw pointer to a TrailDB.

You can pass this pointer to C code and use the C API of TrailDB to use it.

You become responsible for ensuring Haskell doesn't clean up and close the managed Tdb handle. You can use touchTdb or withRawTdb to deal with this.

touchTdb :: MonadIO m => Tdb -> m ()

Touch a Tdb.

Ensures that Tdb has not been garbage collected at the point touchTdb is invoked. Has no other effect.

withRawTdbCons :: MonadIO m => TdbCons -> (Ptr TdbConsRaw -> IO a) -> m a

Run an action with a raw pointer to TdbCons.

The TdbCons is guaranteed not to be garbage collected while the given action is running.

getRawTdbCons :: MonadIO m => TdbCons -> m (Ptr TdbConsRaw)

Returns the raw pointer to a TrailDB construction handle.

Just as with getRawTdb, this pointer can be passed to C and used with the TrailDB C API.

Use touchTdbCons or withRawTdbCons to ensure the pointer is not garbage collected while you are using it.

touchTdbCons :: MonadIO m => TdbCons -> m ()

Touch a TdbCons.

Ensures that TdbCons has not been garbage collected at the point touchTdbCons is invoked. Has no other effect.

data TdbRaw

Represents the raw TrailDB handle as used in C.

data TdbConsRaw

Represents the raw TrailDB construction as used in C.

Taking apart Feature

field :: Lens' Feature FieldID

value :: Lens' Feature TdbVal

(^.) :: s -> Getting a s a -> a

Data types

type UUID = ByteString

UUIDs should be 16 bytes in size. It can be converted to TrailID within a traildb.

type TrailID = Word64

TrailID indexes a trail in a traildb. It can be converted to and back to UUID within a traildb.

type FieldID = TdbField

FieldID is indexes a field number.

type Crumb = (UnixTime, Vector Feature)

A single crumb is some event at certain time.

The vector always has length as told by getNumFields.

data Feature

Feature is a value in traildb. getValue can turn it into a human-readable value within a traildb.

Instances

Eq Feature

Data Feature

Ord Feature

Read Feature

Show Feature

Generic Feature

Storable Feature

Unbox Feature

Feature is isomorphic to Word64 so it's safe to coerce between them. (see featureWord).

Vector Vector Feature

MVector MVector Feature

type Rep Feature

data Vector Feature = V_Feature (Vector Word64)

data MVector s Feature = VM_Feature (MVector s Word64)

type TdbField = Word32

type TdbVal = Word64

type TdbVersion = Word64

TrailDB version

type FieldName = ByteString

Fields names are bytestring and can contain nulls.

class FieldNameLike a where

Class of things that can be used as a field name.

The strict bytestring is the native type. Other types are converted, encoding with UTF-8.

Methods

encodeToFieldName :: a -> ByteString

Instances

FieldNameLike String

FieldNameLike ByteString

FieldNameLike ByteString

FieldNameLike Text

FieldNameLike Text

featureWord :: Iso' Feature Word64

featureTdbVal :: Iso' Feature TdbVal

data Cursor

Instances

Eq Cursor

Generic Cursor

type Rep Cursor

data TdbCons

Instances

Generic TdbCons

type Rep TdbCons

data Tdb

Instances

Generic Tdb

type Rep Tdb

Time

type UnixTime = Word64

The type of time used in traildbs.

getUnixTime :: MonadIO m => m UnixTime

A helper function to get the current unix time.

May be useful when building TrailDBs if you don't have timestamps already.

Exceptions

data TrailDBException

Exceptions that may happen with TrailDBs.

Some programming errors may throw with error instead.

Constructors

CannotAllocateTrailDBCons

Failed to allocate TdbCons.

CannotAllocateTrailDB

Failed to allocate Tdb.

TrailDBError !CInt String

Errors reported by error code from TrailDB C library. includes numerical error and human-readable error.

NoSuchTrailID

A TrailID was used that doesn't exist in Tdb.

NoSuchUUID

A UUID was used that doesn't exist in Tdb.

NoSuchFieldID

A FieldID was used that doesn't exist in Tdb.

NoSuchValue

A Feature was used that doesn't contain a valid value.

NoSuchFeature

Attempted to find Feature for human readable name that doesn't exist.

FinalizationFailure

For some reason, finalizing a TdbCons failed.

Instances

Eq TrailDBException

Ord TrailDBException

Read TrailDBException

Show TrailDBException

Generic TrailDBException

Exception TrailDBException

type Rep TrailDBException

Multiple TrailDBs

Operations in this section are conveniences that may be useful if you have many TrailDB directories you want to query.

findTrailDBs

Arguments

:: (MonadIO m, MonadMask m)

=> FilePath

-> Bool

Follow symbolic links?

-> m [FilePath]

Given a directory, find all valid TrailDB paths inside it, recursively.

filterTrailDBDirectories :: (MonadIO m, MonadMask m) => [FilePath] -> m [FilePath]

Given a list of directories, filters it, returning only directories that are valid TrailDB directories.

Used internally by findTrailDBs but can be useful in general so we export it.