Kafka

Kafka is a module for _writing_ to Kafka

See Kafka.Worker for the basic building blocks of a CLI app that will poll & process kafka messages

Setup

data Handler

A handler for writing to Kafka

data Settings

Settings required to write to Kafka

decoder :: Decoder Settings

decodes Settings from environmental variables KAFKA_BROKER_ADDRESSES=localhost:9092 # comma delimeted list KAFKA_LOG_LEVEL=Debug KAFKA_DELIVERY_TIMEOUT=120000 KAFKA_BATCH_SIZE=10000

handler :: Settings -> Acquire Handler

Function for creating a Kafka handler.

See Settings for potential customizations.

Creating messages

data Msg

A message that can be written to Kafka

Instances

Show Msg

Defined in Kafka.Internal

Methods

showsPrec :: Int -> Msg -> ShowS

show :: Msg -> String

showList :: [Msg] -> ShowS

Generic Msg

Defined in Kafka.Internal

Associated Types

type Rep Msg :: Type -> Type

Methods

from :: Msg -> Rep Msg x

to :: Rep Msg x -> Msg

ToJSON Msg

Defined in Kafka.Internal

Methods

toJSON :: Msg -> Value

toEncoding :: Msg -> Encoding

toJSONList :: [Msg] -> Value

toEncodingList :: [Msg] -> Encoding

type Rep Msg

Defined in Kafka.Internal

type Rep Msg

emptyMsg :: Text -> Msg

Creates a Kafka-writable message for a topic.

msg =
  emptyMsg "groceries"
    |> addPayload "broccoli"
    |> addKey "vegetables"

addPayload :: (FromJSON a, ToJSON a) => a -> Msg -> Msg

addKey :: Text -> Msg -> Msg

Sending messags

sendAsync :: Handler -> Task Never () -> Msg -> Task Text ()

sends messages asynchronously with to Kafka

This is the recommended approach for high throughput. The C++ library behind hte scenes, librdkafka, will batch messages together.

sendSync :: Handler -> Msg -> Task Text ()

sends messages synchronously with to Kafka

This can have a large negative impact on throughput. Use sparingly!

Reading messages

topic :: Msg -> Text

The topic of a message. This function might sometimes be useful in tests.

payload :: FromJSON a => Msg -> Maybe a

The payload of a message. This function might sometimes be useful in tests.

key :: Msg -> Maybe Text

The key of a message. This function might sometimes be useful in tests.