Test.Framework.BlackBoxTest

A black box test in the terminology of the HTF consists of a driver program that is run in various input files. For each input file, the HTF checks that the driver program exits with the correct exit code and that it produces the expected output. The samples directory of the HTF source tree shows an example for a black box test, see https://github.com/skogsbaer/HTF/tree/master/sample.

NOTE: If you use black box tests, you have to compile your program with the -threaded option. Otherwise, your program just blocks indefinitely!

Documentation

data BBTArgs

Use a value of this datatype to customize various aspects of your black box tests.

Constructors

BBTArgs

Fields

• bbtArgs_stdinSuffix :: String

  File extension for the file used as stdin.

• bbtArgs_stdoutSuffix :: String

  File extension for the file specifying expected output on stdout.

• bbtArgs_stderrSuffix :: String

  File extension for the file specifying expected output on stderr.

• bbtArgs_dynArgsName :: String

  Name of a file defining various arguments for executing the tests contained in a subdirectory of the test hierarchy. If a directory contains a such-named file, the arguments apply to all testfiles directly contained in this directory. See the documentation of blackBoxTests for a specification of the argument file format. Default: BBTArgs

• bbtArgs_verbose :: Bool

  Be verbose or not.

• bbtArgs_stdoutDiff :: Diff

  Diff program for comparing output on stdout with the expected value.

• bbtArgs_stderrDiff :: Diff

  Diff program for comparing output on stderr with the expected value.

defaultBBTArgs :: BBTArgs

Sensible default values for BBTArgs:

defaultBBTArgs = BBTArgs { bbtArgs_stdinSuffix    = ".in"
                         , bbtArgs_stdoutSuffix   = ".out"
                         , bbtArgs_stderrSuffix   = ".err"
                         , bbtArgs_dynArgsName    = "BBTArgs"
                         , bbtArgs_stdoutDiff     = defaultDiff
                         , bbtArgs_stderrDiff     = defaultDiff
                         , bbtArgs_verbose        = False }

blackBoxTests

Arguments

:: FilePath

root directory of the test hierarchy

-> String

name of program under test

-> String

filename suffix for input file

-> BBTArgs

configuration

-> IO [Test]

Collects all black box tests with the given file extension stored in a specific directory. For example, the invocation

blackBoxTests "bbt-dir" "dist/build/sample/sample" ".num" defaultBBTArgs

returns a list of Test values, one Test for each .num file found in bbt-dir and its subdirectories. (The samples directory of the HTF source tree contains the example shown here, see https://github.com/skogsbaer/HTF/tree/master/sample.)

Suppose that one of the .num files is bbt-dir/should-pass/x.num. Running the corresponding Test invokes dist/build/sample/sample (the program under test) with bbt-dir/should-pass/x.num as the last commandline argument. The other commandline arguments are taken from the flags specification given in the file whose name is stored in the bbtArgs_dynArgsName field of the BBTArgs record (see below, default is BBTArgs).

If bbt-dir/should-pass/x.in existed, its content would be used as stdin. The tests succeeds if the exit code of the program is zero and the output on stdout and stderr matches the contents of bbt-dir/should-pass/x.out and bbt-dir/should-pass/x.err, respectively. If bbt-dir/should-pass/x.out and bbt-dir/should-pass/x.err do not exist, then output is not checked.

The bbtArgs_dynArgsName field of the BBTArgs record specifies a filename that contains some more configuration flags for the tests. The following flags (separated by newlines) are supported:

Skip

Skips all tests in the same directory as the argument file.

Fail

Specify that the test should succeed if it exits with a non-zero exit code.

Flags: flags

Passes the given flags to the program under test.

type Diff = Maybe FilePath -> String -> IO (Maybe String)

The type of a function comparing the content of a file against a string, similar to the unix tool diff. The first parameter is the name of the file containing the expected output. If this parameter is Nothing, then output should not be checked. The second parameter is the actual output produced. If the result is Nothing then no difference was found. Otherwise, a Just value contains a string explaining the difference.

defaultDiff :: Diff

A default value for the Diff datatype that simple resorts to the diff commandline utility.