streaming-bytestring: effectful byte steams, or: bytestring io done right.
This is an implementation of effectful, memory-constrained bytestrings (byte streams) and functions for streaming bytestring manipulation, adequate for non-lazy-io.
pipes uses this isomorphism:
Streaming.unfoldrChunks Pipes.next :: Monad m => Producer ByteString m r -> ByteString m r Pipes.unfoldr Streaming.nextChunk :: Monad m => ByteString m r -> Producer ByteString m r
io-streams is thus:
IOStreams.unfoldM Streaming.unconsChunk :: ByteString IO () -> IO (InputStream ByteString) Streaming.reread IOStreams.read :: InputStream ByteString -> ByteString IO ()
and similarly for other rational streaming io libraries.
Problems and questions about the library can be put as issues on the github page, or mailed to the pipes list.
A tutorial module is in the works; here, for the moment, is a sequence of simplified implementations of familiar shell utilities. The same programs are implemented at the end of the excellent io-streams tutorial. It is generally much simpler; in some case simpler than what you would write with lazy bytestrings. Here is a simple GET request that returns a byte stream.
The implementation is idiot-simple; it follows the
as far as is possible, replacing the lazy bytestring type:
data ByteString = Empty | Chunk Strict.ByteString ByteString
with the minimal effectful variant:
data ByteString m r = Empty r | Chunk Strict.ByteString (ByteString m r) | Go (m (ByteString m r))
(Constructors are necessarily hidden in internal modules in both cases.)
That's it. As a lazy bytestring is implemented internally
by a sort of list of strict bytestring chunks, a streaming bytestring is
simply implemented as a producer or generator of strict bytestring chunks.
Most operations are defined by simply adding a line to what we find in
Something like this alteration of type is of course obvious and mechanical, once the idea of an effectful bytestring type is contemplated and lazy io is rejected. Indeed it seems that this is the proper expression of what was intended by lazy bytestrings to begin with. The documentation, after all, reads
"A key feature of lazy ByteStrings is the means to manipulate large or unbounded streams of data without requiring the entire sequence to be resident in memory. To take advantage of this you have to write your functions in a lazy streaming style, e.g. classic pipeline composition. The default I/O chunk size is 32k, which should be good in most circumstances."
... which is very much the idea of this library: the default chunk size for
hGetContents and the like follows
Data.ByteString.Lazy and operations
append and so on are tailored not to increase chunk size.
The present library is thus nothing but lazy bytestring done right.
The authors of
Data.ByteString.Lazy must have supposed that
the directly monadic formulation of such their type
would necessarily make things slower. This appears to be a prejudice.
For example, passing a large file of short lines through
this benchmark transformation
Lazy.unlines . map (\bs -> "!" <> Lazy.drop 5 bs) . Lazy.lines Streaming.unlines . S.maps (\bs -> chunk "!" >> Streaming.drop 5 bs) . Streaming.lines
gives pleasing results like these
$ time ./benchlines lazy >> /dev/null real 0m2.097s ... $ time ./benchlines streaming >> /dev/null real 0m1.930s
For a more sophisticated operation like
Lazy.intercalate "!\n" . Lazy.lines Streaming.intercalate "!\n" . Streaming.lines
we get results like these:
time ./benchlines lazy >> /dev/null real 0m1.250s ... time ./benchlines streaming >> /dev/null real 0m1.531s
The pipes environment would express the latter as
Pipes.intercalates (Pipes.yield "!\n") . view Pipes.lines
meaning almost exactly what we mean above, but with results like this
time ./benchlines pipes >> /dev/null real 0m6.353s
The difference, however, is emphatically not intrinsic to pipes;
it is just that
this library depends the
streaming library, which is used in place
free to express the
splitting and iterated division or "chunking" of byte streams.
These concepts belong to the ABCs of streaming;
lines is just
a textbook example, and it is of course handled correctly in
But the concepts are catastrophically mishandled in the streaming io libraries
other than pipes. Already the
were completely defeated by
see e.g. the
enumerator implementation of
splitWhen and lines.
This will concatenate strict text forever, if that's what is coming
in. The rot spreads from there.
It is just a fact that in all of the general streaming io
frameworks other than pipes,
it becomes torture to express elementary distinctions
that are transparently
and immediately contained in any idea of streaming whatsoever.
Though we barely alter signatures in
more than is required by the types,
the point of view that emerges is very much that of
pipes-group. In particular
we have these correspondences:
Lazy.splitAt :: Int -> ByteString -> (ByteString, ByteString) Streaming.splitAt :: Int -> ByteString m r -> ByteString m (ByteString m r) Pipes.splitAt :: Int -> Producer ByteString m r -> Producer ByteString m (Producer ByteString m r)
Lazy.lines :: ByteString -> [ByteString] Streaming.lines :: ByteString m r -> Stream (ByteString m) m r Pipes.lines :: Producer ByteString m r -> FreeT (Producer ByteString m) m r
Stream type expresses the sequencing of
ByteString m _ layers
with the usual 'free monad' sequencing.
If you are unfamiliar with this way of structuring material you might take a look at the tutorial for pipes-group and the examples in the documentation for the streaming library. See also simple implementations of the shell-like examples mentioned above. Or, again, put a question on the issues page or to the pipes list.
[Skip to Readme]
|Versions||0.1.0.0, 0.1.0.1, 0.1.0.2, 0.1.0.3, 0.1.0.4, 0.1.0.5, 0.1.0.6, 0.1.0.7, 0.1.0.8, 0.1.1.0, 0.1.2.0, 0.1.2.2, 0.1.3.0, 0.1.4.0, 0.1.4.2, 0.1.4.3, 0.1.4.4, 0.1.4.5, 0.1.4.6, 0.1.5, 0.1.6 (info)|
|Dependencies||base (<4.9), bytestring, bytestring-builder, deepseq, mmorph (>=1.0 && <1.2), mtl (>=2.1 && <2.3), streaming (>0.1.1.1 && <0.1.2.2), transformers (>=0.3 && <0.5) [details]|
|Category||Data, Pipes, Streaming|
|Source repo||head: git clone https://github.com/michaelt/streaming-bytestring|
|Uploaded||by MichaelThompson at Tue Oct 20 22:47:11 UTC 2015|
|Distributions||LTSHaskell:0.1.6, NixOS:0.1.6, Stackage:0.1.6|
|Downloads||6170 total (57 in the last 30 days)|
|Rating||2.25 (votes: 2) [estimated by rule of succession]|
|Status||Docs uploaded by user
Build status unknown [no reports yet]
Hackage Matrix CI
For package maintainers and hackage trustees