Data.Conduit.Algorithms.Async.ByteString

Higher level async processing interfaces for handling ByteString objects.

Documentation

asyncMapLineGroupsC :: (MonadIO m, NFData a) => Int -> ([ByteString] -> a) -> ConduitT ByteString a m ()

Apply a function to groups of lines

Note that this is much more efficient than the (more or less equivalent, except that the intermediate lists can be of varying sizes):

     CB.lines .| CC.conduitVector N .| CAlg.asyncMapC nthreads (f . V.toList)

The reason being that splitting into lines then becomes the bottleneck and processing a single line is typically a tiny chunk of work so that the threading overhead overwhelms the advantage of using multiple cores. Instead, asyncMapLineGroupsC will pass big chunks to the worker thread and perform most of the line splitting _in the worker thread_.

Only Unix-style ASCII lines are supported (splitting at Bytes with value 10, i.e., \n). When Windows lines (\r\n) are passed to this function, this results in each element having an extra \r at the end.

asyncFilterLinesC :: MonadIO m => Int -> (ByteString -> Bool) -> ConduitT ByteString ByteString m ()

Filter lines using multiple threads

It is not clear from the types but the input is taken to unbroken lines, while the output will be yielded line by line. This conduit is equivalent to

     CB.lines .| CL.filer f