Maintainer | Benjamin Surma <benjamin.surma@gree.net> |
---|---|
Safe Haskell | None |
Configuration and management of processes in a sandboxed environment for system testing.
This module contains extensive documentation. Please scroll down to the Introduction section to continue reading.
- data Sandbox a
- data ProcessSettings = ProcessSettings {}
- def :: Default a => a
- data Capture
- sandbox :: String -> Sandbox a -> IO a
- register :: String -> FilePath -> [String] -> ProcessSettings -> Sandbox String
- run :: String -> Int -> Sandbox (ExitCode, Maybe String)
- withProcess :: String -> Sandbox a -> Sandbox a
- start :: String -> Sandbox ()
- startAll :: Sandbox ()
- stop :: String -> Sandbox ()
- stopAll :: Sandbox ()
- signal :: String -> Signal -> Sandbox ()
- silently :: Sandbox a -> Sandbox a
- interactWith :: String -> String -> Int -> Sandbox String
- sendTo :: String -> String -> Int -> Sandbox String
- readLastCapturedOutput :: String -> Sandbox String
- getHandles :: String -> Sandbox (Handle, Handle)
- getPort :: String -> Sandbox PortNumber
- getBinary :: String -> Sandbox FilePath
- setPort :: String -> Int -> Sandbox PortNumber
- getFile :: String -> Sandbox FilePath
- setFile :: String -> String -> Sandbox FilePath
- getDataDir :: Sandbox FilePath
- bracket :: Sandbox a -> (a -> Sandbox b) -> (a -> Sandbox c) -> Sandbox c
- checkVariable :: String -> Sandbox Bool
- getVariable :: Serialize a => String -> a -> Sandbox a
- setVariable :: Serialize a => String -> a -> Sandbox a
- unsetVariable :: String -> Sandbox ()
- withVariable :: Serialize a => String -> a -> Sandbox b -> Sandbox b
- catchError :: MonadError e m => forall a. m a -> (e -> m a) -> m a
- throwError :: MonadError e m => forall a. e -> m a
- liftIO :: MonadIO m => forall a. IO a -> m a
Introduction
test-sandbox is a framework to manage external applications and communicate with them via TCP or standard I/O for system testing in a sandboxed environment. The Test.Sandbox monad can either be used stand-alone or in conjunction with HUnit, QuickCheck and the test-framework packages to build a complete test suite.
The API is meant to be simple to understand yet flexible enough to meet most of the needs of application testers.
Features
- Register, start and stop programs in a sandboxed environment.
- Automatic cleaning at shutdown: started processes are shutdown, temporary files are deleted.
- Ask the framework to provide you with random, guaranteed not bound TCP ports for your tests: no more collisions when running 2 sets of tests at the same time.
- Generate your temporary configuration files programatically in a secure manner.
- Easily share variables between your tests and modify them at runtime.
- Combine with the test-framework package for standardized output and XML test result generation.
- Use the QuickCheck library to write property tests and generate automatic test cases for your external application; enjoy the full power of the Haskell test harness, even if the application to test is written in a different language!
History
At GREE, we spend lots of time meticulously testing our internally-developed middleware. We have solutions not only developed in Haskell, but also C++ and PHP, but wanted a simple and robust test framework to perform end-to-end testing, and this is how test-sandbox is born.
Usage examples
A basic test-sandbox usecase would be as follows:
- Initialize a Test.Sandbox monad
- Register one or several processes to test a. Ask the Sandbox to provide you with some free TCP ports if needed a. Prepare temporary configuration files if required by your application
- Start some processes
- Communicate with them via TCP or standard IO
- Analyze the received answers and check whether they match an expected pattern
- Error handling is done via the
throwError
andcatchError
functions.
Once all tests are done, the started processes are automatically killed, and all temporary files are deleted.
Communication via TCP
The following example shows a simple test for the memcached NoSQL key-value store.
First, the sandbox is initialized with the sandbox
function;
then, it is asked to provide a free TCP port, which will be used
by the memcached process.
Once the program is registered with the register
function,
it is started with the start
function.
Please note that the Sandbox monad keeps an internal state: processes
registered in a function can be referenced in another without issues.
Communication via TCP is performed with the sendTo
function:
its arguments are the port name (given at the time of getPort
),
the input string, and a timeout in milli-seconds. The function
returns the received TCP answer, if one was received in the correct
timeframe, or fails by throwing an error (which can be caught by
catchError
).
The test is performed with the assertEqual
function from the HUnit
package. In case of matching failure, it will throw an exception,
which, if uncaught (like it is) will cause the Sandbox to perform
cleaning and rethrow the exception.
import Test.Sandbox import Test.Sandbox.HUnit setup :: Sandbox () setup = do port <- getPort "memcached" register "memcached" "memcached" [ "-p", show port ] def main :: IO () main = sandbox $ do setup start "memcached" output <- sendTo "memcached" "set key 0 0 5\r\nvalue\r\n" 1 assertEqual "item is stored" "STORED\r\n" output
Communication via standard I/O
The next example is a hypothetic system test for the popular sed, the popular Unix stream editor.
Please note that at registration time, the psCapture
parameter is
set to CaptureStdout
. This is required by the interactWith
function, used for communication on the standard input, which will
return the captured output on each request.
import Test.Sandbox import Test.Sandbox.HUnit main :: IO () main = sandbox $ do start =<< register "sed_regex" "sed" [ "-u", "s/a/b/" ] def { psCapture = CaptureStdout } assertEqual "a->b" "b\n" =<< interactWith "sed_regex_ "a\n" 5
data ProcessSettings Source
Optional parameters when registering a process in the Sandbox monad.
Initialization
Creates a sandbox and execute the given actions in the IO monad.
Registering processes
:: String | Process name |
-> FilePath | Path to the application binary |
-> [String] | Arguments to pass on the command-line |
-> ProcessSettings | Process settings |
-> Sandbox String |
Registers a process in the Sandbox monad.
Managing sandboxed processes
run :: String -> Int -> Sandbox (ExitCode, Maybe String)Source
Helper function: starts a process, wait for it to terminate and return its captured output.
Starts the given process, runs the action, then stops the process. The process is managed by the functions start and stop respectively.
Gracefully stops a previously started process (verbose)
Gracefully stops all registered processes (in their reverse registration order)
Sends a POSIX signal to a process
Communication
:: String | Process name |
-> String | Input string |
-> Int | Time to wait before timeout (in milli-seconds) |
-> Sandbox String |
Interacts with a sandboxed process via standard I/O.
:: String | Name of the registered port |
-> String | Input string |
-> Int | Time to wait before timeout (in milli-seconds) |
-> Sandbox String |
Communicates with a sandboxed process via TCP and returns the answered message as a string.
Returns the last captured output of a started process.
Returns the handles used to communicate with a registered process using standard I/O.
Returns an unbound user TCP port and stores it for future reference.
Sandbox state management
Returns the effective binary path of a registered process.
Explicitely sets a port to be returned by getPort.
Returns the path of a file previously created by setFile.
Creates a temporary file in the sandbox and returns its path.
getDataDir :: Sandbox FilePathSource
Returns the temporary directory used to host the sandbox environment.
:: Sandbox a | Computation to run first (acquire resource) |
-> (a -> Sandbox b) | Computation to run last (release resource) |
-> (a -> Sandbox c) | Computation to run in-between |
-> Sandbox c |
A variant of bracket from Control.Exception which works in the Sandbox monad.
Checks that a custom sandbox variable is set.
Returns the value of a previously set sandbox variable (or a provided default value if unset)
Sets a custom variable in the sandbox monad.
Temporarily sets a variable for the execution of the given action.
Sandbox exception handling
catchError :: MonadError e m => forall a. m a -> (e -> m a) -> m a
throwError :: MonadError e m => forall a. e -> m a