pipes-cliff-0.6.0.0: Streaming to and from subprocesses using Pipes

Safe HaskellSafe-Inferred
LanguageHaskell2010

Pipes.Cliff

Contents

Description

Spawn subprocesses and interact with them using Pipes

The interface in this module deliberately resembles the interface in System.Process. However, one consequence of this is that you will not want to have unqualified names from this module and from System.Process in scope at the same time.

As in System.Process, you create a subprocess by creating a CreateProcess record and then applying a function to that record. Unlike System.Process, you use functions such as pipeInput or pipeInputOutput to specify what streams you want to use a Proxy for and what streams you wish to be Inherited or if you want to UseHandle. You then send or receive information using one or more Proxy.

Use the -threaded GHC option when compiling your programs or when using GHCi. Internally, this module uses waitForProcess from the System.Process module; it's also quite likely that you will use this function when you write code using this library. As the documentation for waitForProcess states, you must use the -threaded option to prevent every thread in the system from suspending when you use waitForProcess. So, if your program experiences deadlocks, be sure you used the -threaded option.

This module relies on the Pipes, Pipes.Safe, and System.Process modules. You will want to have basic familiarity with what all of those modules do before using this module. It uses Control.Concurrent.Async and Pipes.Concurrent behind the scenes; you don't need to know how these work unless you're curious.

All communcation with subprocesses is done with strict ByteStrings. If you are dealing with textual data, the text library has functions to convert a ByteString to a Text; you will want to look at Data.Text.Encoding.

Nobody would mistake this module for a shell; nothing beats the shell as a language for starting other programs, as the shell is designed for that. This module allows you to perform simple streaming with subprocesses without leaving the comfort of Haskell. Take a look at the README.md file, which is distributed with the tarball or is available at Github at

https://github.com/massysett/pipes-cliff

There you will find references to other libraries that you might find more useful than this one.

You will want to consult Pipes.Cliff.Examples for some examples before getting started. There are some important notes in there about how to run pipelines.

Synopsis

Specifying a subprocess's properties

data CmdSpec Source

Like CmdSpec in System.Process, but also has an instance for Show.

data NonPipe Source

How will the subprocess get its information for this stream? A NonPipe is used for streams that will not be assigned to a Proxy but, instead, will be inherited from the parent or directed from an existing Handle.

Constructors

Inherit

Use whatever stream that the parent process has.

UseHandle Handle

Use the given handle for input or output

data CreateProcess Source

Like CreateProcess in System.Process, this gives the necessary information to create a subprocess. All but one of these fields is also present in CreateProcess, and they all have the same meaning; the only field that is different is the handler field.

Constructors

CreateProcess 

Fields

cmdspec :: CmdSpec

Executable and arguments, or shell command

cwd :: Maybe FilePath

A new current working directory for the subprocess; if Nothing, use the calling process's working directory.

env :: Maybe [(String, String)]

The environment for the subprocess; if Nothing, use the calling process's working directory.

close_fds :: Bool

If True, close all file descriptors other than the standard descriptors. See the documentation for close_fds for details on how this works in Windows.

create_group :: Bool

If True, create a new process group.

delegate_ctlc :: Bool

See delegate_ctlc in the System.Process module for details.

handler :: Oopsie -> IO ()

Whenever an IO exception arises during the course of various IO actios, the exception is caught and placed into an Oopsie that indicates why and where the exception happened. The handler determines what happens when an Oopsie comes in. See Oopsie for details.

The default handler created by procSpec is defaultHandler, which will simply print the exceptions to standard error. You may not want to see the exceptions at all. For example, many exceptions come from broken pipes. A broken pipe might be entirely normal in your circumstance. For example, if you are streaming a large set of values to a pager such as less and you expect that the user will often quit the pager without viewing the whole result, a broken pipe will result, which will print a warning message. That can be a nuisance.

If you don't want to see the exceptions at all, just set handler to squelch, which simply discards the exceptions.

Conceivably you could rig up an elaborate mechanism that puts the Oopsies into a Pipes.Concurrent mailbox or something. Indeed, when using defaultHandler each thread will print its warnings to standard error at any time. If you are using multiple processes and each prints warnings at the same time, total gibberish can result as the text gets mixed in. You could solve this by putting the errors into a Pipes.Concurrent mailbox and having a single thread print the errors; building this sort of functionality directly in to the library would clutter up the API somewhat so I have been reluctant to do it.

procSpec Source

Arguments

:: String

The name of the program to run, such as less.

-> [String]

Command-line arguments

-> CreateProcess 

Create a CreateProcess record with default settings. The default settings are:

  • a raw command (as opposed to a shell command) is created
  • the current working directory is not changed from the parent process
  • the environment is not changed from the parent process
  • the parent's file descriptors (other than standard input, standard output, and standard error) are inherited
  • no new process group is created
  • delegate_ctlc is False
  • storeProcessHandle is Nothing
  • handler is defaultHandler

squelch :: Monad m => a -> m () Source

Do not show or do anything with exceptions; useful to use as a handler.

Creating processes

Each of these functions creates a process. The process begins running immediately in a separate process while your Haskell program continues concurrently. A function is provided for each possible combination of standard input, standard output, and standard error. Use the NonPipe type to describe what you want to do with streams you do NOT want to create a stream for. For example, to create a subprocess that does not create a Pipe for any of the standard streams, use pipeNone. You must describe what you want done with standard input, standard output, and standard error. To create a subprocess that creates a Proxy for standard input and standard output, use pipeInputOutput. You must describe what you want done with standard error. A Producer is returned for standard output and a Consumer for standard input.

If you are creating a Proxy for only one stream (for instance, you're using pipeOutput) then a single Proxy is returned to you. That Proxy manages all the resources it creates; so, for example, when you ultimately run your Effect, the process is created and then destroyed when the MonadSafe computation completes.

If you are creating a Proxy for more than one stream (for instance, you're using pipeInputOutput) then the multiple Proxy are returned to you in a tuple in the MonadSafe computation. The MonadSafe computation will make sure that the resulting process and handles are destroyed when you exit the MonadSafe computation. In such a case, you must make sure that you don't try to use the streams outside of the MonadSafe computation, because the subprocess will already be destroyed. To make sure you are done using the streams before leaving the MonadSafe computation, you will want to use waitForProcess for one or more processes that you are most interested in.

Every function in this section (except for the pipeNone function) returns a value of type (a, h), where a is the set of Proxy, and h is the ProcessHandle; that means the functions that return multiple Proxy have a nested return type. That allows you to use fst and snd to pull out the part you are interested in. It would have been more consistent for pipeNone to return ((), ProcessHandle) but that just seemed silly.

pipeNone Source

Arguments

:: (MonadSafe m, MonadCatch (Base m)) 
=> NonPipe

Standard input

-> NonPipe

Standard output

-> NonPipe

Standard error

-> CreateProcess 
-> m ProcessHandle 

Do not create any Proxy to or from the process.

pipeInput Source

Arguments

:: (MonadSafe mi, MonadSafe m, MonadCatch (Base m)) 
=> NonPipe

Standard output

-> NonPipe

Standard error

-> CreateProcess 
-> m (Consumer ByteString mi (), ProcessHandle)

A Consumer for standard input

Create a Consumer for standard input.

pipeOutput Source

Arguments

:: (MonadSafe mo, MonadSafe m, MonadCatch (Base m)) 
=> NonPipe

Standard input

-> NonPipe

Standard error

-> CreateProcess 
-> m (Producer ByteString mo (), ProcessHandle)

A Producer for standard output

Create a Producer for standard output.

pipeError Source

Arguments

:: (MonadSafe m, MonadCatch (Base m)) 
=> NonPipe

Standard input

-> NonPipe

Standard output

-> CreateProcess 
-> m (Producer ByteString m (), ProcessHandle)

A Producer for standard error

Create a Producer for standard error.

pipeInputOutput Source

Arguments

:: (MonadSafe mi, MonadSafe mo, MonadSafe m, MonadCatch (Base m)) 
=> NonPipe

Standard error

-> CreateProcess 
-> m ((Consumer ByteString mi (), Producer ByteString mo ()), ProcessHandle)

A Consumer for standard input, a Producer for standard output

Create a Consumer for standard input and a Producer for standard output.

pipeInputError Source

Arguments

:: (MonadSafe mi, MonadSafe me, MonadSafe m, MonadCatch (Base m)) 
=> NonPipe

Standard output

-> CreateProcess 
-> m ((Consumer ByteString mi (), Producer ByteString me ()), ProcessHandle)

A Consumer for standard input, a Producer for standard error

Create a Consumer for standard input and a Producer for standard error.

pipeOutputError Source

Arguments

:: (MonadSafe mo, MonadSafe me, MonadSafe m, MonadCatch (Base m)) 
=> NonPipe

Standard input

-> CreateProcess 
-> m ((Producer ByteString mo (), Producer ByteString me ()), ProcessHandle)

A Producer for standard output, a Producer for standard error

Create a Producer for standard output and a Producer for standard error.

pipeInputOutputError Source

Arguments

:: (MonadSafe mi, MonadSafe mo, MonadSafe me, MonadSafe m, MonadCatch (Base m)) 
=> CreateProcess 
-> m ((Consumer ByteString mi (), Producer ByteString mo (), Producer ByteString me ()), ProcessHandle)

A Consumer for standard input, a Producer for standard output, a Producer for standard error

Create a Consumer for standard input, a Producer for standard output, and a Producer for standard error.

Background operations

Often it is necessary to run threads in the background; in addition, all subprocesses run in the background. These functions allow you to launch threads in the background and to wait on background threads and subprocesses.

conveyor :: MonadSafe m => Effect (SafeT IO) () -> m () Source

Runs in the background an effect, typically one that is moving data from one process to another. For examples of its usage, see Pipes.Cliff.Examples. The associated thread is killed when the MonadSafe computation completes.

background :: MonadSafe m => IO a -> m (Async a) Source

Runs a thread in the background. The thread is terminated when the MonadSafe computation completes.

waitForProcess :: MonadIO m => ProcessHandle -> m ExitCode Source

An overloaded version of the waitForProcess from System.Process.

waitForThread :: MonadIO m => Async a -> m a Source

A version of wait with an overloaded MonadIO return type. Allows you to wait for the return value of threads launched with background. If the thread throws an exception, waitForThread will throw that same exception.

Errors and warnings

You will only need what's in this section if you want to examine errors more closely.

data Activity Source

When dealing with a Handle, errors can occur when reading from, writing to, or closing the handle.

Constructors

Reading 
Writing 
Closing 

data HandleDesc Source

Describes a handle. From the perspective of the subprocess; for example, Input means that this handle is connected to the process's standard input.

Constructors

Input 
Output 
Error 

data HandleOopsie Source

Describes IO errors tha occur when dealing with a Handle.

data Oopsie Source

Describes all IO exceptions. The Oopsie contains the IOException itself, along with the CmdSpec that was running when the exception occurred. If the exception occurred while dealing with a Handle, there is also a HandleOopsie. If there is no HandleOopsie, this means that the exception arose when running terminateProcess.

The exceptions that are caught and placed into an Oopsie may arise from reading data from or writing data to a Handle. In these errors, the associated Producer or Consumer will terminate (which may trigger various cleanup actions in the MonadSafe computation) but the exception itself is not re-thrown; rather, it is passed to the handler. Similarly, an exception may occur while closing a handle; these exceptions are caught, not rethrown, and are passed to the handler. If an exception arises when terminating a process (I'm not sure this is possible) then it is also caught, not rethrown, and passed to the handler.

If an exception arises when creating a process--such as a command not being found--the exception is not caught, handled, or passed to the handler. Also, an Oopsie is created only for an IOException; no other exceptions of any kind are caught or handled. However, exceptions of any kind will still trigger appropriate cleanup actions in the MonadSafe computation.

Instances

Re-exports

module Pipes

module Pipes.Safe

Some design notes

Two overarching principles guided the design of this library. First, I wanted the interface to use simple ByteStrings. That most closely represents what a UNIX process sees. If the user wants to use Text or String, it's easy enough to convert between those types and a ByteString. Then the user has to pay explicit attention to encoding issues--as she should, because not all UNIX processes deal with encoded textual data.

Second, I paid meticulous attention to resource management. Resources are deterministically destroyed immediately after use. This eliminates many bugs.

You might wonder why, if you are using an external process as a pipeline, why can't you create, well, a Pipe? Wouldn't that be an obvious choice? Well, if such an interface is possible using Pipes in its current incarnation, nobody has figured it out yet. I don't think it's possible. See also

https://groups.google.com/d/msg/haskell-pipes/JFfyquj5HAg/Lxz7p50JOh4J

for a discussion about this.