streaming-bytestring: effectful byte steams, or: lazy bytestring done right
This is an implementation of effectful, memory-constrained bytestrings (byte streams) and functions for streaming bytestring manipulation, adequate for non-lazy-io.
pipes uses this isomorphism:
Streaming.unfoldrChunks Pipes.next :: Monad m => Producer ByteString m r -> ByteString m r Pipes.unfoldr Streaming.nextChunk :: Monad m => ByteString m r -> Producer ByteString m r
io-streams is thus:
IOStreams.unfoldM Streaming.unconsChunk :: ByteString IO () -> IO (InputStream ByteString) Streaming.reread IOStreams.read :: InputStream ByteString -> ByteString IO ()
and similarly for other rational streaming io libraries.
Problems and questions about the library can be put as issues on the github page, or mailed to the pipes list.
A tutorial module is in the works; here is a sequence of simplified implementations of familiar shell utilities. It closely follows those at the end of the io-streams tutorial. It is generally much simpler; in some case simpler than what you would write with lazy bytestrings. Here is a simple GET request that returns a byte stream.
The implementation follows the
as far as is possible, replacing the lazy bytestring type:
data ByteString = Empty | Chunk Strict.ByteString ByteString
with the minimal effectful variant:
data ByteString m r = Empty r | Chunk Strict.ByteString (ByteString m r) | Go (m (ByteString m r))
(Constructors are necessarily hidden in internal modules in both cases.)
That's it. As a lazy bytestring is implemented internally
by a sort of list of strict bytestring chunks, a streaming bytestring is
simply implemented as a producer or generator of strict bytestring chunks.
Most operations are defined by simply adding a line to what we find in
Something like this alteration of type is of course obvious and mechanical, once the idea of an effectful bytestring type is contemplated and lazy io is rejected. Indeed it seems that this is the proper expression of what was intended by lazy bytestrings to begin with. The documentation, after all, reads
"A key feature of lazy ByteStrings is the means to manipulate large or unbounded streams of data without requiring the entire sequence to be resident in memory. To take advantage of this you have to write your functions in a lazy streaming style, e.g. classic pipeline composition. The default I/O chunk size is 32k, which should be good in most circumstances."
... which is very much the idea of this library: the default chunk size for
hGetContents and the like follows
Data.ByteString.Lazy and operations
append and so on are tailored not to increase chunk size.
It is natural to think that the direct, naive, monadic formulation of such a type would necessarily make things much slower. This appears to be a prejudice. For example, passing a large file of short lines through this benchmark transformation
Lazy.unlines . map (\bs -> "!" <> Lazy.drop 5 bs) . Lazy.lines Streaming.unlines . S.maps (\bs -> chunk "!" >> Streaming.drop 5 bs) . Streaming.lines
gives pleasing results like these
$ time ./benchlines lazy >> /dev/null real 0m2.097s ... $ time ./benchlines streaming >> /dev/null real 0m1.930s
More typical, perhaps, are the results for the more sophisticated operation
Lazy.intercalate "!\n" . Lazy.lines Streaming.intercalate "!\n" . Streaming.lines
time ./benchlines lazy >> /dev/null real 0m1.250s ... time ./benchlines streaming >> /dev/null real 0m1.531s
The pipes environment would express the latter as
Pipes.intercalates (Pipes.yield "!\n") . view Pipes.lines
meaning almost exactly what we mean above, but with results like this
time ./benchlines pipes >> /dev/null real 0m6.353s
The difference is not intrinsic to pipes, but is mostly that
this library depends the
streaming library, which is used in place
free to express the (streaming) splitting and division of byte streams.
Those elementary concepts are catastrophically mishandled in the streaming io libraries
other than pipes. Already the
were completely defeated by it: see e.g. the
enumerator implementation of
splitWhen and lines.
This will concatenate strict text forever, if that's what is coming
in. It becomes torture to express elementary distinctions.
Though we barely alter signatures in
more than is required by the types,
the point of view that emerges is very much that of
pipes-group. In particular
we have these correspondences:
Lazy.splitAt :: Int -> ByteString -> (ByteString, ByteString) Streaming.splitAt :: Int -> ByteString m r -> ByteString m (ByteString m r) Pipes.splitAt :: Int -> Producer ByteString m r -> Producer ByteString m (Producer ByteString m r)
Lazy.lines :: ByteString -> [ByteString] Streaming.lines :: ByteString m r -> Stream (ByteString m) m r Pipes.lines :: Producer ByteString m r -> FreeT (Producer ByteString m) m r
Stream type expresses the sequencing of
ByteString m _ layers
with the usual 'free monad' sequencing.
If you are unfamiliar with this way of structuring material you might take a look at the tutorial for pipes-group and the examples in the documentation for the streaming library. See also simple implementations of the shell-like examples mentioned above. Or, again, put a question on the issues page or to the pipes list.
|Versions [faq]||0.1.0.0, 0.1.0.1, 0.1.0.2, 0.1.0.3, 0.1.0.4, 0.1.0.5, 0.1.0.6, 0.1.0.7, 0.1.0.8, 0.1.1.0, 0.1.2.0, 0.1.2.2, 0.1.3.0, 0.1.4.0, 0.1.4.2, 0.1.4.3, 0.1.4.4, 0.1.4.5, 0.1.4.6, 0.1.5, 0.1.6 (info)|
|Dependencies||base (>=4.7 && <4.9), bytestring (==0.10.*), deepseq, mmorph (>=1.0 && <1.2), mtl (>=2.1 && <2.3), streaming (>0.1.0.8 && <0.1.1), transformers (>=0.3 && <0.5) [details]|
|Category||Data, Pipes, Streaming|
|Uploaded||by MichaelThompson at Sun Sep 13 16:18:33 UTC 2015|
|Distributions||LTSHaskell:0.1.6, NixOS:0.1.6, Stackage:0.1.6|
|Downloads||7348 total (358 in the last 30 days)|
|Rating||2.25 (votes: 2) [estimated by rule of succession]|
Docs uploaded by user
Build status unknown [no reports yet]
For package maintainers and hackage trustees