# conduit-tokenize-attoparsec
Conduits for tokenizing streams.

`conduit-tokenize-attoparsec` is an extension to the `conduit` library based on
`Data.Conduit.Attoparsec` that allows the tokenization of a stream whilst also
tracking arbitrary state.  The library includes in-built support for tracking
such as line-column and stream offset, but can be extended with type-classes to
support alternative state tracking.

For an example, see [`app/Main.hs`](../master/app/Main.hs)

## Building
Prerequisites: `haskell-stack` is installed.

Run the following in the shell:

    git clone git@github.com:haskell-works/conduit-tokenize-attoparsec.git
    cd conduit-tokenize-attoparsec
    stack setup
    stack build
    stack test
    stack ghci --ghc-options -XOverloadedStrings \
      --main-is conduit-tokenize-attoparsec:exe:conduit-tokenize-attoparsec-example