Maintainer	Benjamin Surma <benjamin.surma@gmail.com>
Safe Haskell	None
Language	Haskell2010

Test.Sandbox

Contents

Introduction
- Features
- History
Usage examples
- Communication via TCP
- Communication via standard I/O
Initialization
Calling sandbox on IO
Registering processes
Managing sandboxed processes
Communication
Sandbox state management
Sandbox exception handling
Sandbox I/O handling

Description

Configuration and management of processes in a sandboxed environment for system testing.

This module contains extensive documentation. Please scroll down to the Introduction section to continue reading.

Synopsis

type Sandbox = ExceptT String (ReaderT SandboxStateRef IO)
data ProcessSettings
- = ProcessSettings {
  - psWait :: Maybe Int
  - psCapture :: Maybe Capture
  }
- | ProcessSettings2 {
  - psWait :: Maybe Int
  - psCapture :: Maybe Capture
  - psEnv :: Maybe [(String, String)]
  - psCwd :: Maybe FilePath
  }
def :: Default a => a
data Capture
- = CaptureStdout
- | CaptureStderr
- | CaptureBoth
- | CaptureStdoutWithFile FilePath
- | CaptureStderrWithFile FilePath
- | CaptureBothWithFile FilePath FilePath
sandbox :: String -> Sandbox a -> IO a
withSandbox :: (SandboxStateRef -> IO a) -> IO a
runSandbox :: Sandbox a -> SandboxStateRef -> IO (Either String a)
runSandbox' :: SandboxStateRef -> Sandbox a -> IO a
register :: String -> FilePath -> [String] -> ProcessSettings -> Sandbox String
run :: String -> Int -> Sandbox (ExitCode, Maybe String)
withProcess :: String -> Sandbox a -> Sandbox a
start :: String -> Sandbox ()
startAll :: Sandbox ()
stop :: String -> Sandbox ()
stopAll :: Sandbox ()
signal :: String -> Signal -> Sandbox ()
silently :: Sandbox a -> Sandbox a
interactWith :: String -> String -> Int -> Sandbox String
sendTo :: String -> String -> Int -> Sandbox String
readLastCapturedOutput :: String -> Sandbox String
getHandles :: String -> Sandbox (Handle, Handle)
getPort :: String -> Sandbox Port
getBinary :: String -> Sandbox FilePath
setPort :: String -> Int -> Sandbox Port
getFile :: String -> Sandbox FilePath
setFile :: String -> String -> Sandbox FilePath
getDataDir :: Sandbox FilePath
checkVariable :: String -> Sandbox Bool
getVariable :: Serialize a => String -> a -> Sandbox a
setVariable :: Serialize a => String -> a -> Sandbox a
unsetVariable :: String -> Sandbox ()
withVariable :: Serialize a => String -> a -> Sandbox b -> Sandbox b
bracket :: MonadBaseControl IO m => m a -> (a -> m b) -> (a -> m c) -> m c
catchError :: MonadError e m => m a -> (e -> m a) -> m a
finally :: MonadBaseControl IO m => m a -> m b -> m a
throwError :: MonadError e m => e -> m a
liftIO :: MonadIO m => IO a -> m a

Introduction

test-sandbox is a framework to manage external applications and communicate with them via TCP or standard I/O for system testing in a sandboxed environment. The Test.Sandbox monad can either be used stand-alone or in conjunction with HUnit, QuickCheck and the test-framework packages to build a complete test suite.

The API is meant to be simple to understand yet flexible enough to meet most of the needs of application testers.

Features

Register, start and stop programs in a sandboxed environment.
Automatic cleaning at shutdown: started processes are shutdown, temporary files are deleted.
Ask the framework to provide you with random, guaranteed not bound TCP ports for your tests: no more collisions when running 2 sets of tests at the same time.
Generate your temporary configuration files programatically in a secure manner.
Easily share variables between your tests and modify them at runtime.
Combine with the test-framework package for standardized output and XML test result generation.
Use the QuickCheck library to write property tests and generate automatic test cases for your external application; enjoy the full power of the Haskell test harness, even if the application to test is written in a different language!

History

At GREE, we spend lots of time meticulously testing our internally-developed middleware. We have solutions not only developed in Haskell, but also C++ and PHP, but wanted a simple and robust test framework to perform end-to-end testing, and this is how test-sandbox is born.

Usage examples

A basic test-sandbox usecase would be as follows:

Initialize a Test.Sandbox monad
Register one or several processes to test a. Ask the Sandbox to provide you with some free TCP ports if needed a. Prepare temporary configuration files if required by your application
Start some processes
Communicate with them via TCP or standard IO
Analyze the received answers and check whether they match an expected pattern
Error handling is done via the throwError and catchError functions.

Once all tests are done, the started processes are automatically killed, and all temporary files are deleted.

Communication via TCP

The following example shows a simple test for the "memcached" NoSQL key-value store.

First, the sandbox is initialized with the sandbox function; then, it is asked to provide a free TCP port, which will be used by the memcached process. Once the program is registered with the register function, it is started with the start function. Please note that the Sandbox monad keeps an internal state: processes registered in a function can be referenced in another without issues.

Communication via TCP is performed with the sendTo function: its arguments are the port name (given at the time of getPort), the input string, and a timeout in milli-seconds. The function returns the received TCP answer, if one was received in the correct timeframe, or fails by throwing an error (which can be caught by catchError).

The test is performed with the assertEqual function from the HUnit package. In case of matching failure, it will throw an exception, which, if uncaught (like it is) will cause the Sandbox to perform cleaning and rethrow the exception.

import Test.Sandbox
import Test.Sandbox.HUnit

setup :: Sandbox ()
setup = do
  port <- getPort "memcached"
  register "memcached" "memcached" [ "-p", show port ] def

main :: IO ()
main = sandbox $ do
  setup
  start "memcached"
  output <- sendTo "memcached" "set key 0 0 5\r\nvalue\r\n" 1
  assertEqual "item is stored" "STORED\r\n" output

Communication via standard I/O

The next example is a hypothetic system test for the popular "sed", the popular Unix stream editor.

Please note that at registration time, the psCapture parameter is set to CaptureStdout. This is required by the interactWith function, used for communication on the standard input, which will return the captured output on each request.

import Test.Sandbox
import Test.Sandbox.HUnit

main :: IO ()
main = sandbox $ do
  start =<< register "sed_regex" "sed" [ "-u", "s/a/b/" ] def { psCapture = CaptureStdout }
  assertEqual "a->b" "b\n" =<< interactWith "sed_regex_ "a\n" 5

type Sandbox = ExceptT String (ReaderT SandboxStateRef IO) Source #

data ProcessSettings Source #

Optional parameters when registering a process in the Sandbox monad.

Constructors

ProcessSettings
Fields psWait :: Maybe Int Time to wait (in s.) before checking that the process is still up psCapture :: Maybe Capture Which outputs to capture (if any)
ProcessSettings2
Fields psWait :: Maybe Int Time to wait (in s.) before checking that the process is still up psCapture :: Maybe Capture Which outputs to capture (if any) psEnv :: Maybe [(String, String)] Environment variables psCwd :: Maybe FilePath Working directory for the new process

Instances

Instances details

Default ProcessSettings Source #
Instance details Defined in Test.Sandbox Methods def :: ProcessSettings #

def :: Default a => a #

The default value for this type.

data Capture Source #

Constructors

CaptureStdout
CaptureStderr
CaptureBoth
CaptureStdoutWithFile FilePath
CaptureStderrWithFile FilePath
CaptureBothWithFile FilePath FilePath

Initialization

sandbox Source #

Arguments

:: String	Name of the sandbox environment
-> Sandbox a	Action to perform
-> IO a

Creates a sandbox and execute the given actions in the IO monad.

withSandbox :: (SandboxStateRef -> IO a) -> IO a Source #

Calling sandbox on IO

runSandbox :: Sandbox a -> SandboxStateRef -> IO (Either String a) Source #

runSandbox' :: SandboxStateRef -> Sandbox a -> IO a Source #

Registering processes

register Source #

Arguments

:: String	Process name
-> FilePath	Path to the application binary
-> [String]	Arguments to pass on the command-line
-> ProcessSettings	Process settings
-> Sandbox String

Registers a process in the Sandbox monad.

Managing sandboxed processes

run :: String -> Int -> Sandbox (ExitCode, Maybe String) Source #

Helper function: starts a process, wait for it to terminate and return its captured output.

withProcess Source #

Arguments

:: String	Process name
-> Sandbox a	Action to run
-> Sandbox a

Starts the given process, runs the action, then stops the process. The process is managed by the functions start and stop respectively.

start Source #

Arguments

:: String	Process name
-> Sandbox ()

Starts a previously registered process (verbose)

startAll :: Sandbox () Source #

Starts all registered processes (in their registration order)

stop Source #

Arguments

:: String	Process name
-> Sandbox ()

Gracefully stops a previously started process (verbose)

stopAll :: Sandbox () Source #

Gracefully stops all registered processes (in their reverse registration order)

signal Source #

Arguments

:: String	Process name
-> Signal	Signal to send
-> Sandbox ()

Sends a POSIX signal to a process

silently Source #

Arguments

:: Sandbox a	Action to execute
-> Sandbox a

Executes the given action silently.

Communication

interactWith Source #

Arguments

:: String	Process name
-> String	Input string
-> Int	Time to wait before timeout (in milli-seconds)
-> Sandbox String

Interacts with a sandboxed process via standard I/O.

sendTo Source #

Arguments

:: String	Name of the registered port
-> String	Input string
-> Int	Time to wait before timeout (in milli-seconds)
-> Sandbox String

Communicates with a sandboxed process via TCP and returns the answered message as a string.

readLastCapturedOutput Source #

Arguments

:: String	Process name
-> Sandbox String

Returns the last captured output of a started process.

getHandles Source #

Arguments

:: String	Process name
-> Sandbox (Handle, Handle)

Returns the handles used to communicate with a registered process using standard I/O.

getPort Source #

Arguments

:: String	Port name for future reference
-> Sandbox Port

Returns an unbound user TCP port and stores it for future reference.

Sandbox state management

getBinary Source #

Arguments

:: String	Process name
-> Sandbox FilePath

Returns the effective binary path of a registered process.

setPort Source #

Arguments

:: String	Port name for future reference
-> Int	TCP port number
-> Sandbox Port

Explicitely sets a port to be returned by getPort.

getFile Source #

Arguments

:: String	File name used during setFile
-> Sandbox FilePath

Returns the path of a file previously created by setFile.

setFile Source #

Arguments

:: String	File name for future reference
-> String	File contents
-> Sandbox FilePath

Creates a temporary file in the sandbox and returns its path.

getDataDir :: Sandbox FilePath Source #

Returns the temporary directory used to host the sandbox environment.

checkVariable Source #

Arguments

:: String	Variable key
-> Sandbox Bool

Checks that a custom sandbox variable is set.

getVariable Source #

Arguments

:: Serialize a
=> String	Variable key
-> a	Default value if not found
-> Sandbox a

Returns the value of a previously set sandbox variable (or a provided default value if unset)

setVariable Source #

Arguments

:: Serialize a
=> String	Variable key for future reference
-> a	Variable value
-> Sandbox a

Sets a custom variable in the sandbox monad.

unsetVariable Source #

Arguments

:: String	Variable key
-> Sandbox ()

Unsets a custom variable.

withVariable Source #

Arguments

:: Serialize a
=> String	Variable key
-> a	Variable value
-> Sandbox b	Action to run
-> Sandbox b

Temporarily sets a variable for the execution of the given action.

Sandbox exception handling

bracket #

Arguments

:: MonadBaseControl IO m
=> m a	computation to run first ("acquire resource")
-> (a -> m b)	computation to run last ("release resource")
-> (a -> m c)	computation to run in-between
-> m c

Generalized version of bracket.

Note:

When the "acquire" or "release" computations throw exceptions any monadic side effects in m will be discarded.
When the "in-between" computation throws an exception any monadic side effects in m produced by that computation will be discarded but the side effects of the "acquire" or "release" computations will be retained.
Also, any monadic side effects in m of the "release" computation will be discarded; it is run only for its side effects in IO.

Note that when your acquire and release computations are of type IO it will be more efficient to write:

liftBaseOp (bracket acquire release)

catchError :: MonadError e m => m a -> (e -> m a) -> m a #

A handler function to handle previous errors and return to normal execution. A common idiom is:

do { action1; action2; action3 } `catchError` handler

where the action functions can call throwError. Note that handler and the do-block must have the same return type.

finally #

Arguments

:: MonadBaseControl IO m
=> m a	computation to run first
-> m b	computation to run afterward (even if an exception was raised)
-> m a

Generalized version of finally.

Note, any monadic side effects in m of the "afterward" computation will be discarded.

throwError :: MonadError e m => e -> m a #

Is used within a monadic computation to begin exception processing.

Sandbox I/O handling

liftIO :: MonadIO m => IO a -> m a #

Lift a computation from the IO monad.