System.Process.Streaming.Tutorial

process-streaming uses the CreateProcess record to describe the program to be executed. The user doesn't need to set the std_in, std_out and std_err fields, as these are set automatically according to the Piping.

Piping is a datatype that specifies what standard streams to pipe and what to do with them. It has several constructors, one for each possible combination of streams.

Constructors for Piping usually take Siphons as parameters. A Siphon specifies what to do with a particular standard stream.

Siphons can be built from each of the typical ways of consuming a Producer in the pipes ecosystem:

• Regular Consumers (with fromConsumer, fromConsumerM).
• Folds from the pipes Prelude or specialized folds from pipes-bytestring or pipes-text (with fromFold, fromFold').
• Parsers from pipes-parse (with fromParser and fromParserM).
• Applicative folds from the foldl package (with fromFoldl, fromFoldlIO and fromFoldlM).
• In general, any computation that does something with a Producer (with siphon and siphon').

Siphons have an explicit error type; when a Siphon reading one of the standard streams fails, the external program is immediately terminated and the error value is returned.

A Siphon reading a stream always consumes the whole stream. If the user wants to interrupt the computation early, he can return a failure (or throw an exception).

Siphons have an Applicative instance. pure creates a Siphon that drains a stream but does nothing with the data.

Collecting stdout as a lazy ByteString

We can do this with the toLazyM fold from pipes-bytestring:

>>> execute (pipeo (fromFold B.toLazyM)) (shell "echo ooo")
(ExitSuccess,"ooo\n")

But the intoLazyBytes Siphon is easier to use:

>>> execute (pipeo intoLazyBytes) (shell "echo ooo")
(ExitSuccess,"ooo\n")

Siphons are functors, so if we wanted to collect the output as a strict ByteString, we could do

>>> execute (pipeo (BL.toStrict <$> intoLazyBytes)) (shell "echo ooo")
(ExitSuccess,"ooo\n")

Of course, collecting the output in this way breaks streaming. But this is OK if the output is small.

Collecting stdout and stderr independently

We can use pipeoe collect stdout and stderr concurrently:

>>> execute (pipeoe intoLazyBytes intoLazyBytes) (shell "{ echo ooo ; echo eee 1>&2 ; }")
(ExitSuccess,("ooo\n","eee\n"))

Collecting stdout as a lazy Text

If we want to consume stdout as text, we need to use the encoded function. encoded takes as parameters a decoding function (here decodeUtf8) and a Siphon that specifies how to handle the leftovers. It returns a function that converts a Siphon for text into a Siphon for bytes.

In the example we pass pure id as the leftover-handling Siphon. This means "drain all the undecoded data remaining in the stream and return unchanged the result of intoLazyText". In other words: ignore any leftovers.

>>> execute (pipeo (encoded decodeUtf8 (pure id) intoLazyText)) (shell "echo ooo")
(ExitSuccess,"ooo\n")

But suppose we want to interrupt the execution of the program when we encounter a decoding error. In that case, we can pass unwanted id as the leftover-handling Siphon. unwanted constructs a Siphon that fails when the stream produces any output at all, meaning it will fail if any leftovers remain. unwanted uses the first leftovers that apear in the stream as the error value. So, in this example the error type will be ByteString:

>>> executeFallibly (pipeo (encoded decodeUtf8 (unwanted id) intoLazyText)) (shell "echo ooo")
Right (ExitSuccess,"ooo\n")

Notice also that we had to switch from execute to executeFallibly. This is because, for the first time in the tutorial, we actually have a need for the error type. execute only works when the error type unifies with Void.

Beware: even if the error type is Void, exceptions can still be thrown.

Consuming stdout and stderr combined as Text

Sometimes we want to consume both stdout and stderr, not independently, but combined into a single stream. We can use pipeoec for that.

pipeoec takes as parameter a Siphon for text, and two Lines values that know how to decode the bytes coming from stdout and stderr into lines of text.

>>> :{
   let 
      lin = toLines decodeUtf8 (pure id) 
      program = shell "{ echo ooo ; sleep 1 ; echo eee 1>&2 ; }"
   in execute (pipeoec lin lin intoLazyText) program
   :}
(ExitSuccess,"ooo\neee\n")

We may wish to tag each line in the combined stream with its provenance. This can be done by using tweakLines to modify each Lines argument.

>>> :{
   let 
      lin = toLines decodeUtf8 (pure id) 
      lin_stdout = tweakLines (\p -> P.yield "O" *> p) lin 
      lin_stderr = tweakLines (\p -> P.yield "E" *> p) lin 
      program = shell "{ echo ooo ; sleep 1 ; echo eee 1>&2 ; }"
   in execute (pipeoec lin_stdout lin_stderr intoLazyText) program
   :}
(ExitSuccess,"Oooo\nEeee\n")

Feeding stdin, consuming stdout

We can feed bytes to stdin while we read stdout or stderr. We use the Pump datatype for that.

>>> execute (pipeio (fromLazyBytes "iii") intoLazyBytes) (shell "cat")
(ExitSuccess,((),"iii"))

Early termination

An example of how returning a failure from a Siphon interrupts the whole computation and terminates the external program.

>>> executeFallibly (pipeo (siphon (\_ -> return (Left "oops")))) (shell "sleep infinity")
Left "oops"

Collecting stdout as lines of Text

>>> :{
   let 
      osiphon
          = getSiphonOp 
          $ contraencoded decodeUtf8 (pure id) 
          $ nest splitIntoLines intoLazyText 
          $ SiphonOp intoList
   in 
   execute (pipeo osiphon) (shell "{ echo aa ; echo bb ; }")
   :}
(ExitSuccess,["aa","bb"])