nri-kafka: Functions for working with Kafka

[ bsd3, library, web ] [ Propose Tags ]

Downloads

Maintainer's Corner

For package maintainers and hackage trustees

Candidates

  • No Candidates
Versions [RSS] 0.1.0.0, 0.1.0.1, 0.1.0.2, 0.1.0.4
Change log CHANGELOG.md
Dependencies aeson (>=1.4.6.0 && <2.1), async (>=2.2.2 && <2.3), base (>=4.12.0.0 && <4.17), bytestring (>=0.10.8.2 && <0.12), conduit (>=1.3.0 && <1.4), containers (>=0.6.0.1 && <0.7), hw-kafka-client (>=4.0.3 && <5.0), nri-env-parser (>=0.1.0.0 && <0.2), nri-observability (>=0.1.1.1 && <0.2), nri-prelude (>=0.1.0.0 && <0.7), safe-exceptions (>=0.1.7.0 && <1.3), stm (>=2.4 && <2.6), text (>=1.2.3.1 && <2.1), time (>=1.8.0.2 && <2), unix (>=2.7.2.2 && <2.8.0.0), uuid (>=1.3.0 && <1.4) [details]
License BSD-3-Clause
Copyright 2022 NoRedInk Corp.
Author NoRedInk
Maintainer haskell-open-source@noredink.com
Category Web
Home page https://github.com/NoRedInk/haskell-libraries/tree/trunk/nri-kafka#readme
Bug tracker https://github.com/NoRedInk/haskell-libraries/issues
Source repo head: git clone https://github.com/NoRedInk/haskell-libraries(nri-kafka)
Uploaded by julianobs at 2022-02-01T21:56:02Z
Distributions NixOS:0.1.0.4
Downloads 219 total (3 in the last 30 days)
Rating 2.0 (votes: 1) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs uploaded by user
Build status unknown [no reports yet]

Readme for nri-kafka-0.1.0.4

[back to package description]

Kafka integration

Reviewed last on 2021-05-28

This library exposes an Elm-like API to Kafka. It exports two main modules:

  • Kafka, for writing to Kafka.
  • Kafka.Worker, For building long-running worker apps that process Haskell messages.

At NoRedInk, we use this to power our high-throughput quiz-engine service. If you work at NoRedInk: look there for a simple example app.

Otherwise: here's the gist of it:

import qualified Environment -- from nri-env-parser
import qualified Kafka.Worker

-- your long running app
main :: IO ()
main =
  settings <- Environment.decode Kafka.Worker.decoder
  Kafka.Worker.process
    Kafka.Worker.Description
      settings
      "this worker's group id"
      (Kafka.Worker.subscription "my.topic" processMessage,)

data MyKafkaMessageType =
  ReticulateSplines Int
  AddHiddenAgenda Text
  CalculateLlamaExpectorationTrajectory Llamas
  deriving (generic)

instance Aeson.ToJSON Envelope
instance Aeson.FromJSON Envelope

-- the meat and potatoes: handles all MyKafkaMessageTypes
processMessage ::
  Kafka.Worker.Envelope MyKafkaMessageType ->
  Task Text ()
processMessage record myMessage =
  -- process your message in here
  -- because of our usage of `Task` you probably want to pass in any handlers
  case myMessage of
    AddHiddenAgenda agenda ->
    	Debug.todo "Add the agenda"
    _ ->
    	Debug.todo "and also handle the other cases"