Database.DynamoDB

Type-safe library for accessing DynamoDB database.

Introduction

This library is operated in the following way:

• Create instances for your custom types using Database.DynamoDB.Types
• Create ordinary datatypes with records
• Use functions from Database.DynamoDB.TH to derive appropriate instances
• Optionally call generated migration function to automatically create tables and indices
• Call functions from this module to access the database

The library does its best to ensure that only correct DynamoDB operations are allowed. There are some limitations of DynamoDB regarding access to empty values, but the library takes care of this reasonably well.

Example of use

You may need to set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

data Test = Test {
    category :: T.Text
  , messageid :: T.Text
  , subject :: T.Text
} deriving (Show)
mkTableDefs "migrate" (tableConfig "" (''Test, WithRange) [] [])

This code creates appropriate instances for the table and the columns. It creates global variables colCategory, colMessageid and colSubject that can be used in filtering conditions or update queries.

main = do
   lgr <- newLogger Info stdout
   env <- newEnv NorthVirginia Discover
   -- Override, use DynamoDD on localhost
   let dynamo = setEndpoint False "localhost" 8000 dynamoDB
   let newenv = env & configure dynamo
                    & set envLogger lgr
   runResourceT $ runAWS newenv $ do
       -- Create tables and indexes
       migrate mempty Nothing
       -- Save data to database
       putItem (Test "news" "1-2-3-4" "New subject")
       -- Fetch data given primary key
       item <- getItem Eventually tTest ("news", "1-2-3-4")
       liftIO $ print item       -- (item :: Maybe Test)
       -- Scan data using filter condition, return 10 results
       items <- scanCond tTest (subject' ==. "New subejct") 10
       print items         -- (items :: [Test])

See examples and test directories for more detail examples.

Proxies

In order to avoid ambiguity errors, most API calls need a Proxy argument to find out on which table or index to operate. These proxies are automatically generated as a name of type prepended with "t" for tables and "i" for indexes.

A proxy for table Test will have name tTest, for index TestIndex the name will be iTestIndex.

Lens support

If the field names in the table record start with an underscore, the lens get automatically generated for accessing the fields. The lens are polymorphic, you can use them to access the fields of both main table and all the indexes.

data Test = Test {
    _messageid :: T.Text
  , _category :: T.Text
  , _subject :: T.Text
} deriving (Show)
data TestIndex = TestIndex {
    i_category :: T.Text
  , i_messageid :: T.Text
}
mkTableDefs "migrate" (tableConfig "" (''Test, WithRange) [(''TestIndex, NoRange)] [])

doWithTest :: Test -> ...
doWithTest item = (item ^. category) ...

doWithItemIdx :: TestIndex -> ..
getCategoryIdx item = (item ^. category) ...

Conversion support

Given a type Test and an index type TestIndex, a function toTest is created that converts from TestIndex to Test. Such function is created only if TestIndex projects all fields from Test.

Data types

data DynamoException

Exceptions thrown by some dynamodb-simple actions.

Constructors

DynamoException Text

Instances

Show DynamoException

Methods

showsPrec :: Int -> DynamoException -> ShowS

show :: DynamoException -> String

showList :: [DynamoException] -> ShowS

Exception DynamoException

Methods

toException :: DynamoException -> SomeException

fromException :: SomeException -> Maybe DynamoException

displayException :: DynamoException -> String

data Consistency

Parameter for queries involving read consistency settings.

Constructors

Eventually

Strongly

Instances

Show Consistency

Methods

showsPrec :: Int -> Consistency -> ShowS

show :: Consistency -> String

showList :: [Consistency] -> ShowS

data Direction

Query direction

Constructors

Forward

Backward

Instances

Eq Direction

Methods

(==) :: Direction -> Direction -> Bool

(/=) :: Direction -> Direction -> Bool

Show Direction

Methods

showsPrec :: Int -> Direction -> ShowS

show :: Direction -> String

showList :: [Direction] -> ShowS

data Column typ (coltype :: ColumnType) col

Representation of a column for filter queries

• typ - datatype of column (Int, Text..)
• coltype - TypColumn or TypSize (result of size(column))
• col - instance of ColumnInfo, uniquely identify a column

Attribute path combinators

(<.>) :: InCollection col2 (UnMaybe typ) NestedPath => Column typ TypColumn col1 -> Column typ2 TypColumn col2 -> Column typ2 TypColumn col1 infixl 7

Combine attributes from nested structures.

address' <.> street'

(<!>) :: Column [typ] TypColumn col -> Int -> Column typ TypColumn col infixl 8

Access an index in a nested list.

users' <!> 0 <.> name'

(<!:>) :: IsText key => Column (HashMap key typ) TypColumn col -> key -> Column typ TypColumn col infixl 8

Access a key in a nested hashmap.

phones' <!:> "mobile" <.> number'

Fetching items

getItem :: forall m a r. (MonadAWS m, DynamoTable a r) => Consistency -> Proxy a -> PrimaryKey a r -> m (Maybe a)

Read item from the database; primary key is either a hash key or (hash,range) tuple depending on the table.

getItemBatch :: forall m a r. (MonadAWS m, DynamoTable a r) => Consistency -> [PrimaryKey a r] -> m [a]

Get batch of items.

Query options

data QueryOpts a hash range

Options for a generic query.

queryOpts :: hash -> QueryOpts a hash range

Default settings for query options.

qConsistentRead :: forall a hash range. Lens' (QueryOpts a hash range) Consistency

qStartKey :: forall a hash range. Lens' (QueryOpts a hash range) (Maybe (hash, range))

qDirection :: forall a hash range. Lens' (QueryOpts a hash range) Direction

qFilterCondition :: forall a hash range a. Lens (QueryOpts a hash range) (QueryOpts a hash range) (Maybe (FilterCondition a)) (Maybe (FilterCondition a))

qHashKey :: forall a hash range. Lens' (QueryOpts a hash range) hash

qRangeCondition :: forall a hash range. Lens' (QueryOpts a hash range) (Maybe (RangeOper range))

qLimit :: forall a hash range. Lens' (QueryOpts a hash range) (Maybe Natural)

Performing query

query

Arguments

:: (CanQuery a t hash range, MonadAWS m)

=> Proxy a

-> QueryOpts a hash range

-> Int

Maximum number of items to fetch

-> m ([a], Maybe (PrimaryKey a WithRange))

Fetch exactly the required count of items even when it means more calls to dynamodb. Return last evaluted key if end of data was not reached. Use qStartKey to continue reading the query.

querySimple

Arguments

:: (CanQuery a t hash range, MonadAWS m)

=> Proxy a

Proxy type of a table to query

-> hash

Hash key

-> Maybe (RangeOper range)

Range condition

-> Direction

Scan direction

-> Int

Maximum number of items to fetch

-> m [a]

Perform a simple, eventually consistent, query.

Simple to use function to query limited amount of data from database.

queryCond

Arguments

:: (CanQuery a t hash range, MonadAWS m)

=> Proxy a

-> hash

Hash key

-> Maybe (RangeOper range)

Range condition

-> FilterCondition a

-> Direction

Scan direction

-> Int

Maximum number of items to fetch

-> m [a]

Query with condition

querySource :: forall a t m hash range. (CanQuery a t hash range, MonadAWS m) => Proxy a -> QueryOpts a hash range -> Source m a

Generic query function. You can query table or indexes that have a range key defined. The filter condition cannot access the hash and range keys.

querySourceChunks :: forall a t m hash range. (CanQuery a t hash range, MonadAWS m) => Proxy a -> QueryOpts a hash range -> Source m [a]

Same as querySource, but return data in original chunks.

querySourceByKey :: forall a parent hash rest v1 m r. (DynamoIndex a parent NoRange, Code a ~ '[hash ': rest], DynamoScalar v1 hash, MonadAWS m, DynamoTable parent r) => Proxy a -> hash -> Source m a

Conduit to query global indexes with no range key; in case anyone needed it

queryOverIndex :: forall a t m v1 v2 hash r2 range rest parent. (CanQuery a t hash range, MonadAWS m, Code a ~ '[hash ': (range ': rest)], DynamoIndex a parent WithRange, ContainsTableKey a parent (PrimaryKey parent r2), DynamoTable parent r2, DynamoScalar v1 hash, DynamoScalar v2 range) => Proxy a -> QueryOpts a hash range -> Source m parent

Query an index, fetch primary key from the result and immediately read full items from the main table.

You cannot perform strongly consistent reads on Global indexes; if you set the qConsistentRead to Strongly, fetch from global indexes is still done as eventually consistent. Queries on local indexes are performed according to the settings.

Scan options

data ScanOpts a r

Record for defining scan command. Use lenses to set the content.

sParallel: (Segment number, Total segments)

scanOpts :: ScanOpts a r

sFilterCondition :: forall a r. Lens' (ScanOpts a r) (Maybe (FilterCondition a))

sConsistentRead :: forall a r. Lens' (ScanOpts a r) Consistency

sLimit :: forall a r. Lens' (ScanOpts a r) (Maybe Natural)

sParallel :: forall a r. Lens' (ScanOpts a r) (Maybe (Natural, Natural))

sStartKey :: forall a r r. Lens (ScanOpts a r) (ScanOpts a r) (Maybe (PrimaryKey a r)) (Maybe (PrimaryKey a r))

Performing scan

scan

Arguments

:: (MonadAWS m, TableScan a r t)

=> Proxy a

-> ScanOpts a r

Scan settings

-> Int

Required result count

-> m ([a], Maybe (PrimaryKey a r))

list of results, lastEvalutedKey or Nothing if end of data reached

Function to call bounded scans. Tries to return exactly requested number of items.

Use sStartKey to continue the scan.

scanSource :: (MonadAWS m, TableScan a r t) => Proxy a -> ScanOpts a r -> Source m a

Conduit source for running a scan.

scanSourceChunks :: (MonadAWS m, TableScan a r t) => Proxy a -> ScanOpts a r -> Source m [a]

Conduit source for running scan; the same as scanSource, but return results in chunks as they come.

scanCond :: forall a m r t. (MonadAWS m, TableScan a r t) => Proxy a -> FilterCondition a -> Int -> m [a]

Scan table using a given filter condition.

scanCond (colAddress <!:> "Home" <.> colCity ==. "London") 10

Helper conduits

leftJoin

Arguments

:: (MonadAWS m, DynamoTable a r, Ord (PrimaryKey a r), ContainsTableKey a a (PrimaryKey a r))

=> Consistency

-> Proxy a

Proxy type for the right table

-> (b -> Maybe (PrimaryKey a r))

-> Conduit [b] m [(b, Maybe a)]

Conduit to do a left join on the items being sent; supposed to be used with querySourceChunks.

The 'foreign key' must have an Ord to facilitate faster searching.

innerJoin

Arguments

:: (MonadAWS m, DynamoTable a r, Ord (PrimaryKey a r), ContainsTableKey a a (PrimaryKey a r))

=> Consistency

-> Proxy a

Proxy type for the right table

-> (b -> Maybe (PrimaryKey a r))

-> Conduit [b] m [(b, a)]

The same as leftJoin, but discard items that do not exist in the right table.

Data entry

putItem :: (MonadAWS m, DynamoTable a r) => a -> m ()

Write item into the database; overwrite any previously existing item with the same primary key.

putItemBatch :: forall m a r. (MonadAWS m, DynamoTable a r) => [a] -> m ()

Batch write into the database.

The batch is divided to 25-item chunks, each is sent and retried separately. If a batch fails on dynamodb exception, it is raised.

Note: On exception, the information about which items were saved is unavailable

insertItem :: forall a r m. (MonadAWS m, DynamoTable a r) => a -> m ()

Write item into the database only if it doesn't already exist.

Data modification

updateItemByKey :: forall a m r. (MonadAWS m, DynamoTable a r) => Proxy a -> PrimaryKey a r -> Action a -> m a

Update item in a database, return an updated version of the item.

updateItemByKey_ :: forall a m r. (MonadAWS m, DynamoTable a r) => Proxy a -> PrimaryKey a r -> Action a -> m ()

Update item in a table.

updateItem (Proxy :: Proxy Test) (12, "2") (colCount +=. 100)

updateItemCond_ :: forall a m r. (MonadAWS m, DynamoTable a r) => Proxy a -> PrimaryKey a r -> FilterCondition a -> Action a -> m ()

Update item in a table while specifying a condition.

Deleting data

deleteItemByKey :: forall m a r. (MonadAWS m, DynamoTable a r) => Proxy a -> PrimaryKey a r -> m ()

Delete item from the database by specifying the primary key.

deleteItemCondByKey :: forall m a r. (MonadAWS m, DynamoTable a r) => Proxy a -> PrimaryKey a r -> FilterCondition a -> m ()

Delete item from the database by specifying the primary key and a condition. Throws AWS exception if the condition does not succeed.

deleteItemBatchByKey :: forall m a r. (MonadAWS m, DynamoTable a r) => Proxy a -> [PrimaryKey a r] -> m ()

Batch version of deleteItemByKey.

Note: Because the requests are chunked, the information about which items were deleted in case of exception is unavailable.

Delete table

deleteTable :: (MonadAWS m, DynamoTable a r) => Proxy a -> m ()

Delete a table from DynamoDB.

Utility functions

tableKey :: forall a parent key. ContainsTableKey a parent key => a -> key

Extract primary key from a record.

You can use this on both main table or on index tables if they contain the primary key from the main table. Table key is always projected to indexes anyway, so just define it in every index.

Typeclasses

class (DynamoCollection a r IsTable, HasPrimaryKey a r IsTable) => DynamoTable a (r :: RangeType) | a -> r

Descritpion of dynamo table

Minimal complete definition

tableName

class DynamoCollection a r IsIndex => DynamoIndex a parent (r :: RangeType) | a -> parent r

Class representing a Global Secondary Index

Minimal complete definition

indexName, indexIsLocal

type PrimaryKey a r = PrimaryKey' (Code a) r

Type family that returns a primary key of a table/index depending on the RangeType parameter.

class ContainsTableKey a parent key | a -> parent key

Class for indexes that contain primary key of the parent table

Minimal complete definition

dTableKey

class (DynamoCollection a WithRange t, HasPrimaryKey a WithRange t) => CanQuery a (t :: TableType) hash range | a -> hash range

Class for tables/indexes that can be queried

Minimal complete definition

qTableName, qIndexName, dQueryKey, dQueryKeyToAttr

class (DynamoCollection a r t, HasPrimaryKey a r t) => TableScan a (r :: RangeType) (t :: TableType)

Class for tables/indexes that can be scanned

Minimal complete definition

qsTableName, qsIndexName