Pipes.ByteString

This module provides pipes utilities for "byte streams", which are streams of strict ByteStrings chunks. Use byte streams to interact with both Handles and lazy ByteStrings.

To stream to or from Handles, use fromHandle or toHandle. For example, the following program copies data from one file to another:

 import Pipes
 import qualified Pipes.ByteString as P
 import System.IO

 main =
     withFile "inFile.txt"  ReadMode  $ \hIn  ->
     withFile "outFile.txt" WriteMode $ \hOut ->
     runEffect $ P.fromHandle hIn >-> P.toHandle hOut

You can stream to and from stdin and stdout using the predefined stdin and stdout proxies, like in the following "echo" program:

 main = runEffect $ P.stdin >-> P.stdout

You can also translate pure lazy ByteStrings to and from proxies:

 import qualified Data.ByteString.Lazy.Char8 as BL

 main = runEffect $ P.fromLazy (BL.pack "Hello, world!\n") >-> P.stdout

In addition, this module provides many functions equivalent to lazy ByteString functions so that you can transform or fold byte streams. For example, to stream only the first three lines of stdin to stdout you would write:

 import Pipes
 import qualified Pipes.ByteString as PB
 import qualified Pipes.Parse      as PP

 main = runEffect $ takeLines 3 PB.stdin >-> PB.stdout
   where
     takeLines n = PB.unlines . PP.takeFree n . PB.lines

The above program will never bring more than one chunk (~ 32 KB) into memory, no matter how long the lines are.

Note that functions in this library are designed to operate on streams that are insensitive to chunk boundaries. This means that they may freely split chunks into smaller chunks and discard empty chunks. However, they will never concatenate chunks in order to provide strict upper bounds on memory usage.

Producers

fromLazy :: Monad m => ByteString -> Producer' ByteString m ()

Convert a lazy ByteString into a Producer of strict ByteStrings

stdin :: MonadIO m => Producer' ByteString m ()

Stream bytes from stdin

fromHandle :: MonadIO m => Handle -> Producer' ByteString m ()

Convert a Handle into a byte stream using a default chunk size

hGetSome :: MonadIO m => Int -> Handle -> Producer' ByteString m ()

Convert a handle into a byte stream using a maximum chunk size

hGetSome forwards input immediately as it becomes available, splitting the input into multiple chunks if it exceeds the maximum chunk size.

hGet :: MonadIO m => Int -> Handle -> Producer' ByteString m ()

Convert a handle into a byte stream using a fixed chunk size

hGet waits until exactly the requested number of bytes are available for each chunk.

pack :: Monad m => Producer Word8 m () -> Producer ByteString m ()

Convert a Word8 producer into a byte stream using a default chunk size

Servers

hGetSomeN :: MonadIO m => Handle -> Int -> Server' Int ByteString m ()

Like hGetSome, except you can vary the maximum chunk size for each request

hGetN :: MonadIO m => Handle -> Int -> Server' Int ByteString m ()

Like hGet, except you can vary the chunk size for each request

Consumers

stdout :: MonadIO m => Consumer' ByteString m ()

Stream bytes to stdout

Unlike toHandle, stdout gracefully terminates on a broken output pipe.

Note: For best performance, use (for source (liftIO . putStr)) instead of (source >-> stdout).

toHandle :: MonadIO m => Handle -> Consumer' ByteString m r

Convert a byte stream into a Handle

Note: For best performance, use (for source (liftIO . hPutStr handle)) instead of (source >-> toHandle handle).

Pipes

map :: Monad m => (Word8 -> Word8) -> Pipe ByteString ByteString m r

Apply a transformation to each Word8 in the stream

concatMap :: Monad m => (Word8 -> ByteString) -> Pipe ByteString ByteString m r

Map a function over the byte stream and concatenate the results

take :: (Monad m, Integral a) => a -> Pipe ByteString ByteString m ()

(take n) only allows n bytes to pass

drop :: (Monad m, Integral a) => a -> Pipe ByteString ByteString m r

(drop n) drops the first n bytes

takeWhile :: Monad m => (Word8 -> Bool) -> Pipe ByteString ByteString m ()

Take bytes until they fail the predicate

dropWhile :: Monad m => (Word8 -> Bool) -> Pipe ByteString ByteString m r

Drop bytes until they fail the predicate

filter :: Monad m => (Word8 -> Bool) -> Pipe ByteString ByteString m r

Only allows Word8s to pass if they satisfy the predicate

elemIndices :: (Monad m, Num n) => Word8 -> Pipe ByteString n m r

Stream all indices whose elements match the given Word8

findIndices :: (Monad m, Num n) => (Word8 -> Bool) -> Pipe ByteString n m r

Stream all indices whose elements satisfy the given predicate

scan :: Monad m => (Word8 -> Word8 -> Word8) -> Word8 -> Pipe ByteString ByteString m r

Strict left scan over the bytes

unpack :: Monad m => Pipe ByteString Word8 m ()

Unpack the bytes

Folds

toLazy :: Producer ByteString Identity () -> ByteString

Fold a pure Producer of strict ByteStrings into a lazy ByteString

toLazyM :: Monad m => Producer ByteString m () -> m ByteString

Fold an effectful Producer of strict ByteStrings into a lazy ByteString

Note: toLazyM is not an idiomatic use of pipes, but I provide it for simple testing purposes. Idiomatic pipes style consumes the chunks immediately as they are generated instead of loading them all into memory.

fold :: Monad m => (x -> Word8 -> x) -> x -> (x -> r) -> Producer ByteString m () -> m r

Reduce the stream of bytes using a strict left fold

head :: Monad m => Producer ByteString m () -> m (Maybe Word8)

Retrieve the first Word8

last :: Monad m => Producer ByteString m () -> m (Maybe Word8)

Retrieve the last Word8

null :: Monad m => Producer ByteString m () -> m Bool

Determine if the stream is empty

length :: (Monad m, Num n) => Producer ByteString m () -> m n

Count the number of bytes

any :: Monad m => (Word8 -> Bool) -> Producer ByteString m () -> m Bool

Fold that returns whether Any received Word8s satisfy the predicate

all :: Monad m => (Word8 -> Bool) -> Producer ByteString m () -> m Bool

Fold that returns whether All received Word8s satisfy the predicate

maximum :: Monad m => Producer ByteString m () -> m (Maybe Word8)

Return the maximum Word8 within a byte stream

minimum :: Monad m => Producer ByteString m () -> m (Maybe Word8)

Return the minimum Word8 within a byte stream

elem :: Monad m => Word8 -> Producer ByteString m () -> m Bool

Determine whether any element in the byte stream matches the given Word8

notElem :: Monad m => Word8 -> Producer ByteString m () -> m Bool

Determine whether all elements in the byte stream do not match the given Word8

find :: Monad m => (Word8 -> Bool) -> Producer ByteString m () -> m (Maybe Word8)

Find the first element in the stream that matches the predicate

index :: (Monad m, Integral a) => a -> Producer ByteString m () -> m (Maybe Word8)

Index into a byte stream

elemIndex :: (Monad m, Num n) => Word8 -> Producer ByteString m () -> m (Maybe n)

Find the index of an element that matches the given Word8

findIndex :: (Monad m, Num n) => (Word8 -> Bool) -> Producer ByteString m () -> m (Maybe n)

Store the first index of an element that satisfies the predicate

count :: (Monad m, Num n) => Word8 -> Producer ByteString m () -> m n

Store a tally of how many elements match the given Word8

Splitters

splitAt :: (Monad m, Integral n) => n -> Producer ByteString m r -> Producer' ByteString m (Producer ByteString m r)

Splits a Producer after the given number of bytes

chunksOf :: (Monad m, Integral n) => n -> Producer ByteString m r -> FreeT (Producer ByteString m) m r

Split a byte stream into FreeT-delimited byte streams of fixed size

span :: Monad m => (Word8 -> Bool) -> Producer ByteString m r -> Producer' ByteString m (Producer ByteString m r)

Split a byte stream in two, where the first byte stream is the longest consecutive group of bytes that satisfy the predicate

break :: Monad m => (Word8 -> Bool) -> Producer ByteString m r -> Producer ByteString m (Producer ByteString m r)

Split a byte stream in two, where the first byte stream is the longest consecutive group of bytes that don't satisfy the predicate

splitWith :: Monad m => (Word8 -> Bool) -> Producer ByteString m r -> FreeT (Producer ByteString m) m r

Split a byte stream into sub-streams delimited by bytes that satisfy the predicate

split :: Monad m => Word8 -> Producer ByteString m r -> FreeT (Producer ByteString m) m r

Split a byte stream using the given Word8 as the delimiter

groupBy :: Monad m => (Word8 -> Word8 -> Bool) -> Producer ByteString m r -> FreeT (Producer ByteString m) m r

Group a byte stream into FreeT-delimited byte streams using the supplied equality predicate

group :: Monad m => Producer ByteString m r -> FreeT (Producer ByteString m) m r

Group a byte stream into FreeT-delimited byte streams of identical bytes

lines :: Monad m => Producer ByteString m r -> FreeT (Producer ByteString m) m r

Split a byte stream into FreeT-delimited lines

Note: This function is purely for demonstration purposes since it assumes a particular encoding. You should prefer the Text equivalent of this function from the upcoming pipes-text library.

words :: Monad m => Producer ByteString m r -> FreeT (Producer ByteString m) m r

Split a byte stream into FreeT-delimited words

Note: This function is purely for demonstration purposes since it assumes a particular encoding. You should prefer the Text equivalent of this function from the upcoming pipes-text library.

Transformations

intersperse :: Monad m => Word8 -> Producer ByteString m r -> Producer ByteString m r

Intersperse a Word8 in between the bytes of the byte stream

Joiners

intercalate :: Monad m => Producer ByteString m () -> FreeT (Producer ByteString m) m r -> Producer ByteString m r

intercalate concatenates the FreeT-delimited byte streams after interspersing a byte stream in between them

unlines :: Monad m => FreeT (Producer ByteString m) m r -> Producer ByteString m r

Join FreeT-delimited lines into a byte stream

Note: This function is purely for demonstration purposes since it assumes a particular encoding. You should prefer the Text equivalent of this function from the upcoming pipes-text library.

unwords :: Monad m => FreeT (Producer ByteString m) m r -> Producer ByteString m r

Join FreeT-delimited words into a byte stream

Note: This function is purely for demonstration purposes since it assumes a particular encoding. You should prefer the Text equivalent of this function from the upcoming pipes-text library.

Low-level Parsers

The following parsing utilities are single-byte analogs of the ones found in pipes-parse.

nextByte :: Monad m => Producer ByteString m r -> m (Either r (Word8, Producer ByteString m r))

Consume the first byte from a byte stream

next either fails with a Left if the Producer has no more bytes or succeeds with a Right providing the next byte and the remainder of the Producer.

drawByte :: Monad m => StateT (Producer ByteString m r) m (Either r Word8)

Draw one Word8 from the underlying Producer, returning Left if the Producer is empty

unDrawByte :: Monad m => Word8 -> StateT (Producer ByteString m r) m ()

Push back a Word8 onto the underlying Producer

peekByte :: Monad m => StateT (Producer ByteString m r) m (Either r Word8)

peekByte checks the first Word8 in the stream, but uses unDrawByte to push the Word8 back

 peekByte = do
     x <- drawByte
     case x of
         Left  _  -> return ()
         Right w8 -> unDrawByte w8
     return x

isEndOfBytes :: Monad m => StateT (Producer ByteString m r) m Bool

Check if the underlying Producer has no more bytes

Note that this will skip over empty ByteString chunks, unlike isEndOfInput from pipes-parse.

 isEndOfBytes = liftM isLeft peekByte

takeWhile' :: Monad m => (Word8 -> Bool) -> Pipe ByteString ByteString (StateT (Producer ByteString m r) m) ()

Deprecated: Use Pipes.ByteString.Parse.takeWhile instead

Take bytes until they fail the predicate

Unlike takeWhile, this unDraws unused bytes

Re-exports

Pipes.ByteString.Parse re-exports nextByte, drawByte, unDrawByte, peekByte, and isEndOfBytes.

Data.ByteString re-exports the ByteString type.

Data.Word re-exports the Word8 type.

Pipes.Parse re-exports input, concat, and FreeT (the type).

module Data.ByteString

module Data.Word

module Pipes.Parse