streaming-bytestring: effectful byte steams, or: lazy bytestring done right

[ bsd3, data, library, pipes, streaming ] [ Propose Tags ]

This is an implementation of effectful, memory-constrained bytestrings (byte streams) and functions for streaming bytestring manipulation, adequate for non-lazy-io.

Interoperation with pipes uses this isomorphism:

Streaming.unfoldrChunks :: Monad m => Producer ByteString m r -> ByteString m r
Pipes.unfoldr Streaming.nextChunk  :: Monad m => ByteString m r -> Producer ByteString m r

Interoperation with io-streams is thus:

IOStreams.unfoldM Streaming.unconsChunk :: ByteString IO () -> IO (InputStream ByteString)
Streaming.reread         :: InputStream ByteString -> ByteString IO ()

and similarly for other rational streaming io libraries.

Problems and questions about the library can be put as issues on the github page, or mailed to the pipes list.

A tutorial module is in the works; here is a sequence of simplified implementations of familiar shell utilities. It closely follows those at the end of the io-streams tutorial. It is generally much simpler; in some case simpler than what you would write with lazy bytestrings. Here is a simple GET request that returns a byte stream.

The implementation follows the details of Data.ByteString.Lazy and Data.ByteString.Lazy.Char8 as far as is possible, replacing the lazy bytestring type:

data ByteString     = Empty   | Chunk Strict.ByteString ByteString

with the minimal effectful variant:

data ByteString m r = Empty r | Chunk Strict.ByteString (ByteString m r) | Go (m (ByteString m r))

(Constructors are necessarily hidden in internal modules in both cases.)

That's it. As a lazy bytestring is implemented internally by a sort of list of strict bytestring chunks, a streaming bytestring is simply implemented as a producer or generator of strict bytestring chunks. Most operations are defined by simply adding a line to what we find in Data.ByteString.Lazy.

Something like this alteration of type is of course obvious and mechanical, once the idea of an effectful bytestring type is contemplated and lazy io is rejected. Indeed it seems that this is the proper expression of what was intended by lazy bytestrings to begin with. The documentation, after all, reads

... which is very much the idea of this library: the default chunk size for hGetContents and the like follows Data.ByteString.Lazy and operations like lines and append and so on are tailored not to increase chunk size.

It is natural to think that the direct, naive, monadic formulation of such a type would necessarily make things much slower. This appears to be a prejudice. For example, passing a large file of short lines through this benchmark transformation

Lazy.unlines      . map    (\bs -> "!"       <> Lazy.drop 5 bs)       . Lazy.lines
Streaming.unlines . S.maps (\bs -> chunk "!" >> Streaming.drop 5 bs)  . Streaming.lines

gives pleasing results like these

$  time ./benchlines lazy >> /dev/null
real	0m2.097s
$  time ./benchlines streaming >> /dev/null
real	0m1.930s

More typical, perhaps, are the results for the more sophisticated operation

Lazy.intercalate "!\n"      . Lazy.lines
Streaming.intercalate "!\n" . Streaming.lines
time ./benchlines lazy >> /dev/null
real	0m1.250s
time ./benchlines streaming >> /dev/null
real	0m1.531s

The pipes environment would express the latter as

Pipes.intercalates (Pipes.yield "!\n") . view Pipes.lines

meaning almost exactly what we mean above, but with results like this

 time ./benchlines pipes >> /dev/null
 real	0m6.353s

The difference is not intrinsic to pipes, but is mostly that this library depends the streaming library, which is used in place of free to express the (streaming) splitting and division of byte streams. Those elementary concepts are catastrophically mishandled in the streaming io libraries other than pipes. Already the enumerator and iteratee libraries were completely defeated by it: see e.g. the enumerator implementation of splitWhen and lines. This will concatenate strict text forever, if that's what is coming in. It becomes torture to express elementary distinctions.

Though we barely alter signatures in Data.ByteString.Lazy more than is required by the types, the point of view that emerges is very much that of pipes-bytestring and pipes-group. In particular we have these correspondences:

Lazy.splitAt      :: Int -> ByteString              -> (ByteString, ByteString)
Streaming.splitAt :: Int -> ByteString m r          -> ByteString m (ByteString m r)
Pipes.splitAt     :: Int -> Producer ByteString m r -> Producer ByteString m (Producer ByteString m r)


Lazy.lines      :: ByteString -> [ByteString]
Streaming.lines :: ByteString m r -> Stream (ByteString m) m r
Pipes.lines     :: Producer ByteString m r -> FreeT (Producer ByteString m) m r

where the Stream type expresses the sequencing of ByteString m _ layers with the usual 'free monad' sequencing.

If you are unfamiliar with this way of structuring material you might take a look at the tutorial for pipes-group and the examples in the documentation for the streaming library. See also simple implementations of the shell-like examples mentioned above. Or, again, put a question on the issues page or to the pipes list.

Versions [faq],,,,,,,,,,,,,,,,,,, 0.1.5, 0.1.6 (info)
Change log
Dependencies base (>=4.7 && <4.9), bytestring (==0.10.*), deepseq, mmorph (>=1.0 && <1.2), mtl (>=2.1 && <2.3), streaming (> && <0.1.1), transformers (>=0.3 && <0.5) [details]
License BSD-3-Clause
Author michaelt
Category Data, Pipes, Streaming
Uploaded by MichaelThompson at 2015-09-13T16:18:33Z
Distributions LTSHaskell:0.1.6, NixOS:0.1.6, Stackage:0.1.6
Downloads 13954 total (833 in the last 30 days)
Rating 2.25 (votes: 2) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Hackage Matrix CI
Docs uploaded by user
Build status unknown [no reports yet]




Maintainer's Corner

For package maintainers and hackage trustees