Pipes.Cliff

Spawn subprocesses and interact with them using Pipes

The interface in this module deliberately resembles the interface in System.Process. However, one consequence of this is that you will not want to have unqualified names from this module and from System.Process in scope at the same time.

As in System.Process, you create a subprocess by creating a CreateProcess record and then applying a function to that record. Unlike System.Process, you use functions such as pipeInput or pipeInputOutput to specify what streams you want to use a Proxy for and what streams you wish to be Inherited or if you want to UseHandle. You then send or receive information using one or more Proxy.

Use the -threaded GHC option when compiling your programs or when using GHCi. Internally, this module uses waitForProcess from the System.Process module. As the documentation for waitForProcess states, you must use the -threaded option to prevent every thread in the system from suspending when waitForProcess is used. So, if your program experiences deadlocks, be sure you used the -threaded option.

This module relies on the Pipes, Pipes.Safe, Control.Concurrent.Async, and System.Process modules. You will want to have basic familiarity with what all of those modules do before using this module.

All communcation with subprocesses is done with strict ByteStrings. If you are dealing with textual data, the text library has functions to convert a ByteString to a Text; you will want to look at Data.Text.Encoding.

Nobody would mistake this module for a shell; nothing beats the shell as a language for starting other programs, as the shell is designed for that. This module allows you to perform simple streaming with subprocesses without leaving the comfort of Haskell. Take a look at the README.md file, which is distributed with the tarball or is available at Github at

https://github.com/massysett/pipes-cliff

There you will find references to other libraries that you might find more useful than this one.

You will want to consult Pipes.Cliff.Examples for some examples before getting started. There are some important notes in there about how to run pipelines.

Specifying a subprocess's properties

data CmdSpec

Like CmdSpec in System.Process, but also has an instance for Show.

Constructors

ShellCommand String

RawCommand FilePath [String]

Instances

Eq CmdSpec

Ord CmdSpec

Show CmdSpec

data NonPipe

How will the subprocess get its information for this stream? A NonPipe is used for streams that will not be assigned to a Proxy but, instead, will be inherited from the parent or directed from an existing Handle.

Constructors

Inherit

Use whatever stream that the parent process has.

UseHandle Handle

Use the given handle for input or output

data CreateProcess

Like CreateProcess in System.Process, this gives the necessary information to create a subprocess. All but one of these fields is also present in CreateProcess, and they all have the same meaning; the only field that is different is the handler field.

Constructors

CreateProcess

Fields

cmdspec :: CmdSpec

Executable and arguments, or shell command

cwd :: Maybe FilePath

A new current working directory for the subprocess; if Nothing, use the calling process's working directory.

env :: Maybe [(String, String)]

The environment for the subprocess; if Nothing, use the calling process's working directory.

close_fds :: Bool

If True, close all file descriptors other than the standard descriptors. See the documentation for close_fds for details on how this works in Windows.

create_group :: Bool

If True, create a new process group.

delegate_ctlc :: Bool

See delegate_ctlc in the System.Process module for details.

handler :: Oopsie -> IO ()

Whenever an IO exception arises during the course of various IO actions, the exception is caught and placed into an Oopsie that indicates why and where the exception happened. The handler determines what happens when an Oopsie comes in. See Oopsie for details.

The default handler created by procSpec is defaultHandler, which will simply print the exceptions to standard error. You may not want to see the exceptions at all. For example, many exceptions come from broken pipes. A broken pipe might be entirely normal in your circumstance. For example, if you are streaming a large set of values to a pager such as less and you expect that the user will often quit the pager without viewing the whole result, a broken pipe will result, which will print a warning message. That can be a nuisance.

If you don't want to see the exceptions at all, just set handler to squelch, which simply discards the exceptions.

Conceivably you could rig up an elaborate mechanism that puts the Oopsies into a Pipes.Concurrent mailbox or something. Indeed, when using defaultHandler each thread will print its warnings to standard error at any time. If you are using multiple processes and each prints warnings at the same time, total gibberish can result as the text gets mixed in. You could solve this by putting the errors into a Pipes.Concurrent mailbox and having a single thread print the errors; this sort of thing could be built into the library but so far I haven't been motivated to do it.

procSpec

Arguments

:: String

The name of the program to run, such as less.

-> [String]

Command-line arguments

-> CreateProcess

Create a CreateProcess record with default settings. The default settings are:

• a raw command (as opposed to a shell command) is created
• the current working directory is not changed from the parent process
• the environment is not changed from the parent process
• the parent's file descriptors (other than standard input, standard output, and standard error) are inherited
• no new process group is created
• delegate_ctlc is False
• handler is defaultHandler

squelch :: Monad m => a -> m ()

Do not show or do anything with exceptions; useful to use as a handler.

Side effects: None.

Creating processes

Each of these functions creates a process. The process begins running immediately in a separate process while your Haskell program continues concurrently. A function is provided for each possible combination of standard input, standard output, and standard error. Use the NonPipe type to describe what you want to do with streams you do NOT want to create a stream for. For example, to create a subprocess that creates a Proxy for standard input and standard output, use pipeInputOutput. You must describe what you want done with standard error. A Producer is returned for standard output and a Consumer for standard input.

Each function also returns a ProcessHandle; this is not the same ProcessHandle that you will find in System.Process. You can use this ProcessHandle to obtain some information about the process that is created and to get the eventual ExitCode.

Every time you create a process with one of these functions, some additional behind-the-scenes resources are created, such as some threads to move data to and from the process. In normal usage, these threads will be cleaned up after the process exits. However, if exceptions are thrown, there could be resource leaks. Applying terminateProcess to a ProcessHandle makes a best effort to clean up all the resources that Cliff may create, including the process itself and any additional threads. To guard against resource leaks, use the functions found in Control.Exception or in Control.Monad.Catch. Control.Monad.Catch provides operations that are the same as those in Control.Exception, but they are not limited to IO.

I say that terminateProcess "makes a best effort" to release resources because in UNIX it merely sends a SIGTERM to the process. That should kill well-behaved processes, but terminateProcess does not send a SIGKILL. terminateProcess always closes all handles associated with the process and it kills all Haskell threads that were moving data to and from the process. (terminateProcess does not kill threads it does not know about, such as threads you created with conveyor.)

There is no function that will create a process that has no Proxy at all. For that, just use createProcess in System.Process.

pipeInput

Arguments

:: (MonadSafe mi, MonadCatch (Base mi))

=> NonPipe

Standard output

-> NonPipe

Standard error

-> CreateProcess

-> IO (Consumer ByteString mi ExitCode, ProcessHandle)

A Consumer for standard input

Create a Consumer for standard input.

pipeOutput

Arguments

:: (MonadSafe mo, MonadCatch (Base mo))

=> NonPipe

Standard input

-> NonPipe

Standard error

-> CreateProcess

-> IO (Producer ByteString mo ExitCode, ProcessHandle)

A Producer for standard output

Create a Producer for standard output.

pipeError

Arguments

:: (MonadSafe me, MonadCatch (Base me))

=> NonPipe

Standard input

-> NonPipe

Standard output

-> CreateProcess

-> IO (Producer ByteString me ExitCode, ProcessHandle)

A Producer for standard error

Create a Producer for standard error.

pipeInputOutput

Arguments

:: (MonadSafe mi, MonadCatch (Base mi), MonadSafe mo, MonadCatch (Base mo))

=> NonPipe

Standard error

-> CreateProcess

-> IO ((Consumer ByteString mi ExitCode, Producer ByteString mo ExitCode), ProcessHandle)

A Consumer for standard input, a Producer for standard output

Create a Consumer for standard input and a Producer for standard output.

pipeInputError

Arguments

:: (MonadSafe mi, MonadCatch (Base mi), MonadSafe me, MonadCatch (Base me))

=> NonPipe

Standard output

-> CreateProcess

-> IO ((Consumer ByteString mi ExitCode, Producer ByteString me ExitCode), ProcessHandle)

A Consumer for standard input, a Producer for standard error

Create a Consumer for standard input and a Producer for standard error.

pipeOutputError

Arguments

:: (MonadSafe mo, MonadCatch (Base mo), MonadSafe me, MonadCatch (Base me))

=> NonPipe

Standard input

-> CreateProcess

-> IO ((Producer ByteString mo ExitCode, Producer ByteString me ExitCode), ProcessHandle)

A Producer for standard output and a Producer for standard error

Create a Producer for standard output and a Producer for standard error.

pipeInputOutputError

Arguments

:: (MonadSafe mi, MonadCatch (Base mi), MonadSafe mo, MonadCatch (Base mo), MonadSafe me, MonadCatch (Base me))

=> CreateProcess

-> IO ((Consumer ByteString mi ExitCode, Producer ByteString mo ExitCode, Producer ByteString me ExitCode), ProcessHandle)

A Consumer for standard input, a Producer for standard output, and a Producer for standard error

Create a Consumer for standard input, a Producer for standard output, and a Producer for standard error.

Proxy combinators

conveyor :: Effect (SafeT IO) a -> IO (Async a)

Runs in the background an effect, typically one that is moving data from one process to another. For examples of its usage, see Pipes.Cliff.Examples.

safeEffect :: Effect (SafeT IO) a -> IO a

Runs in the foreground an effect in the SafeT monad.

Querying and terminating the process

data ProcessHandle

Allows you to terminate the process, as well as to obtain some information about the process.

originalCreateProcess :: ProcessHandle -> CreateProcess

Tells you the CreateProcess that was originally used to create the process associated with this ProcessHandle.

isStillRunning :: ProcessHandle -> IO Bool

Is this process still running?

Side effects: examines the process handle to see if it has yet returned a value. Does not block; should return immediately.

waitForProcess :: ProcessHandle -> IO ExitCode

Gets the exit code of the process that belongs to the ProcessHandle. Often you can get the exit code through more idiomatic pipes functions, as the various Proxy return the ExitCode. Sometimes though it can be difficult to use the pipes idioms to get the exit code, so this function is here.

Side effects: may block if process has not yet exited.

terminateProcess :: ProcessHandle -> IO ()

Terminates a process. Sends the process a SIGTERM, which does not absolutely guarantee that it will exit. Closes any Handles that were created for the process through Cliff, and terminates any associated background threads that were moving data to and from the process. Use this function this with bracket to ensure proper cleanup of resources.

Exception safety

These are some simple combinators built with bracket; feel free to use your own favorite idioms for exception safety.

withProcess

Arguments

:: IO (a, ProcessHandle)

Creates the process

-> (a -> IO b)

Uses the process

-> IO b

Creates a process, uses it, and terminates it when the last computation ends. Don't try to use any of the process resources after the last computation ends, because the process will already have been terminated. For an example of its use, see standardOutputAndErrorBracketed.

withConveyor

Arguments

:: Effect (SafeT IO) a

The Effect to run in another thread

-> IO b

The rest of the computation to run

-> IO b

Runs an Effect in the backgroud (typically one that is moving data from one process to another). If the background thread is still running when the second computation ends, the background thread is terminated. For an example of its use, see standardOutputAndErrorBracketed.

Errors and warnings

You will only need what's in this section if you want to examine errors more closely.

data Activity

When dealing with a Handle, errors can occur when reading from, writing to, or closing the handle.

Constructors

Reading

Writing

Closing

Instances

Eq Activity

Ord Activity

Show Activity

data Outbound

The two kinds of outbound handles.

Constructors

Output

Error

Instances

Eq Outbound

Ord Outbound

Show Outbound

data HandleDesc

Describes a handle. From the perspective of the subprocess; for example, Input means that this handle is connected to the process's standard input.

Constructors

Input

Outbound Outbound

Instances

Eq HandleDesc

Ord HandleDesc

Show HandleDesc

data Oopsie

Describes all IO exceptions. The Oopsie contains the IOException itself, along with the CmdSpec that was running when the exception occurred.

The exceptions that are caught and placed into an Oopsie may arise from reading data from or writing data to a Handle. In these errors, the associated Producer or Consumer will terminate (which may trigger various cleanup actions in the MonadSafe computation) but the exception itself is not re-thrown; rather, it is passed to the handler. Similarly, an exception may occur while closing a handle; these exceptions are caught, not rethrown, and are passed to the handler. If an exception arises when terminating a process (I'm not sure this is possible) then it is also caught, not rethrown, and passed to the handler.

If an exception arises when creating a process--such as a command not being found--the exception is not caught, handled, or passed to the handler. In addition, no exceptions are caught if they originated during a waitForProcess. (I can't conceive of how any synchronous exceptions could arise from waitForProcess, but if they do, Cliff does not handle them.) Also, an Oopsie is created only for an IOException; no other exceptions of any kind are caught or handled. However, exceptions of any kind will still trigger appropriate cleanup actions in the MonadSafe computation.

Constructors

Oopsie Activity HandleDesc CmdSpec IOException

Instances

Eq Oopsie

Show Oopsie

Re-exports

• Control.Concurrent.Async reexports all bindings
• Pipes reexports all bindings
• Pipes.Safe reexports runSafeT
• System.Exit reexports all bindings

module Control.Concurrent.Async

module Pipes

module Pipes.Safe

module System.Exit

Some design notes

Two overarching principles guided the design of this library. First, I wanted the interface to use simple ByteStrings. That most closely represents what a UNIX process sees. If the user wants to use Text or String, it's easy enough to convert between those types and a ByteString. Then the user has to pay explicit attention to encoding issues--as she should, because not all UNIX processes deal with encoded textual data.

Second, I paid meticulous attention to resource management. Resources are deterministically destroyed immediately after use. This eliminates many bugs. Even so, I decided to leave it up to the user to use something like bracket to ensure that all resources are cleaned up if there is an exception. Originally I tried to have the library do this, but that turned out not to be very composable. There are already many exception-handling mechanisms available in Control.Exception, Pipes.Safe, and Control.Monad.Catch, and it seems best to let the user choose how to handle this issue; she can just perform a bracket and may combine this with the ContT monad in transformers or mtl if she wishes, or perhaps with the managed library.

An earlier version of this library (see version 0.8.0.0) tried to use the return value of a Proxy to indicate the return value of both processes in the pipeline, not just one. I removed this because it interfered heavily with composability.

You might wonder why, if you are using an external process as a pipeline, why can't you create, well, a Pipe? Wouldn't that be an obvious choice? Well, if such an interface is possible using Pipes in its current incarnation, nobody has figured it out yet. I don't think it's possible. See also

https://groups.google.com/d/msg/haskell-pipes/JFfyquj5HAg/Lxz7p50JOh4J

for a discussion about this.