tasty: Modern and extensible testing framework
Tasty is a modern testing framework for Haskell. It lets you combine your unit tests, golden tests, QuickCheck/SmallCheck properties, and any other types of tests into a single test suite.
[Skip to Readme]
|Versions [faq]||0.1, 0.1.1, 0.2, 0.3, 0.3.1, 0.4, 0.4.0.1, 0.4.1.1, 0.4.2, 0.5, 0.5.1, 0.5.2, 0.5.2.1, 0.6, 0.7, 0.8, 0.8.0.2, 0.8.0.4, 0.8.1.1, 0.8.1.2, 0.8.1.3, 0.9.0.1, 0.10, 0.10.0.1, 0.10.0.2, 0.10.0.3, 0.10.0.4, 0.10.1, 0.10.1.1, 0.10.1.2, 0.11, 0.11.0.1, 0.11.0.2, 0.11.0.3, 0.11.0.4, 0.11.1, 0.11.2, 0.11.2.1, 0.11.2.2, 0.11.2.3, 0.11.2.4, 0.11.2.5, 0.11.3, 0.12, 0.12.0.1, 1.0, 220.127.116.11, 1.0.1, 18.104.22.168, 1.1, 22.214.171.124, 126.96.36.199, 188.8.131.52, 184.108.40.206, 1.2, 1.2.1|
|Dependencies||ansi-terminal (>=0.9), async (>=2.0), base (>=4.5 && <5), clock (>=0.4.4.0), containers, ghc-prim, mtl (>=220.127.116.11), optparse-applicative (>=0.14), stm (>=2.3), tagged (>=0.5), time (>=1.4), unbounded-delays (>=0.1), unix, wcwidth [details]|
|Author||Roman Cheplyaka <firstname.lastname@example.org>|
|Maintainer||Roman Cheplyaka <email@example.com>|
|Source repo||head: git clone git://github.com/feuerbach/tasty.git(core)|
|Uploaded||by RomanCheplyaka at Thu Jan 31 21:50:10 UTC 2019|
|Distributions||Arch:1.2, Debian:18.104.22.168, Fedora:22.214.171.124, LTSHaskell:1.2, NixOS:1.2.1, Stackage:1.2.1|
|Downloads||89840 total (684 in the last 30 days)|
|Rating||2.5 (votes: 8) [estimated by rule of succession]|
Docs available [build log]
Last success reported on 2019-01-31 [all 1 reports]
Depend on the clock package for more accurate time measurement
Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info
For package maintainers and hackage trustees
Readme for tasty-1.2.1[back to package description]
Tasty is a modern testing framework for Haskell.
It lets you combine your unit tests, golden tests, QuickCheck/SmallCheck properties, and any other types of tests into a single test suite.
- Run tests in parallel but report results in a deterministic order
- Filter the tests to be run using patterns specified on the command line
- Hierarchical, colored display of test results
- Reporting of test statistics
- Acquire and release resources (sockets, temporary files etc.) that can be shared among several tests
- Extensibility: add your own test providers and ingredients (runners) above and beyond those provided
To find out what's new, read the change log.
Here's how your
test.hs might look like:
import Test.Tasty import Test.Tasty.SmallCheck as SC import Test.Tasty.QuickCheck as QC import Test.Tasty.HUnit import Data.List import Data.Ord main = defaultMain tests tests :: TestTree tests = testGroup "Tests" [properties, unitTests] properties :: TestTree properties = testGroup "Properties" [scProps, qcProps] scProps = testGroup "(checked by SmallCheck)" [ SC.testProperty "sort == sort . reverse" $ \list -> sort (list :: [Int]) == sort (reverse list) , SC.testProperty "Fermat's little theorem" $ \x -> ((x :: Integer)^7 - x) `mod` 7 == 0 -- the following property does not hold , SC.testProperty "Fermat's last theorem" $ \x y z n -> (n :: Integer) >= 3 SC.==> x^n + y^n /= (z^n :: Integer) ] qcProps = testGroup "(checked by QuickCheck)" [ QC.testProperty "sort == sort . reverse" $ \list -> sort (list :: [Int]) == sort (reverse list) , QC.testProperty "Fermat's little theorem" $ \x -> ((x :: Integer)^7 - x) `mod` 7 == 0 -- the following property does not hold , QC.testProperty "Fermat's last theorem" $ \x y z n -> (n :: Integer) >= 3 QC.==> x^n + y^n /= (z^n :: Integer) ] unitTests = testGroup "Unit tests" [ testCase "List comparison (different length)" $ [1, 2, 3] `compare` [1,2] @?= GT -- the following test does not hold , testCase "List comparison (same length)" $ [1, 2, 3] `compare` [1,2,2] @?= LT ]
And here is the output of the above program:
(Note that whether QuickCheck finds a counterexample to the third property is determined by chance.)
tasty is the core package. It contains basic definitions and APIs and a console runner.
In order to create a test suite, you also need to install one or more «providers» (see below).
The following providers exist:
- tasty-hunit — for unit tests (based on HUnit)
- tasty-golden — for golden tests, which are unit tests whose results are kept in files
- tasty-smallcheck — exhaustive property-based testing (based on smallcheck)
- tasty-quickcheck — for randomized property-based testing (based on QuickCheck)
- tasty-hedgehog — for randomized property-based testing (based on Hedgehog)
- tasty-hspec — for Hspec tests
- tasty-leancheck — for enumerative property-based testing (based on LeanCheck)
- tasty-program — run external program and test whether it terminates successfully
- tasty-wai - for testing wai endpoints.
It's easy to create custom providers using the API from
Ingredients represent different actions that you can perform on your test suite. One obvious ingredient that you want to include is one that runs tests and reports the progress and results.
Another standard ingredient is one that simply prints the names of all tests.
It is possible to write custom ingredients using the API from
Some ingredients that can enhance your test suite are:
- tasty-ant-xml adds a possibility to write the test results in a machine-readable XML format, which is understood by various CI systems and IDEs
- tasty-rerun adds support for minimal test reruns by recording previous test runs and using this information to filter the test tree. For example, you can use this ingredient to only run failed tests, or only run tests that threw an exception.
- tasty-html adds the possibility to write the test results as a HTML file
- tasty-stats adds the possibility to collect statistics of the test suite in a CSV file.
- tasty-th automatically discovers tests based on the function names and generate the boilerplate code for you
- tasty-hunit-adapter converts existing HUnit test suites into tasty test suites
- tasty-discover automatically discovers your tests.
- tasty-expected-failure provides test markers for when you expect failures or wish to ignore tests.
Options allow one to customize the run-time behavior of the test suite, such as:
- mode of operation (run tests, list tests, run tests quietly etc.)
- which tests are run (see «Patterns» below)
- parameters of individual providers (like depth of search for SmallCheck)
There are two main ways to set options:
When using the standard console runner, the options can be passed on the
command line or via environment variables. To see the available options, run
your test suite with the
--help flag. The output will look something like this
(depending on which ingredients and providers the test suite uses):
% ./test --help Mmm... tasty test suite Usage: test [-p|--pattern PATTERN] [-t|--timeout DURATION] [-l|--list-tests] [-j|--num-threads NUMBER] [-q|--quiet] [--hide-successes] [--color never|always|auto] [--quickcheck-tests NUMBER] [--quickcheck-replay SEED] [--quickcheck-show-replay] [--quickcheck-max-size NUMBER] [--quickcheck-max-ratio NUMBER] [--quickcheck-verbose] [--smallcheck-depth NUMBER] Available options: -h,--help Show this help text -p,--pattern PATTERN Select only tests which satisfy a pattern or awk expression -t,--timeout DURATION Timeout for individual tests (suffixes: ms,s,m,h; default: s) -l,--list-tests Do not run the tests; just print their names -j,--num-threads NUMBER Number of threads to use for tests execution -q,--quiet Do not produce any output; indicate success only by the exit code --hide-successes Do not print tests that passed successfully --color never|always|auto When to use colored output (default: 'auto') --quickcheck-tests NUMBER Number of test cases for QuickCheck to generate --quickcheck-replay SEED Random seed to use for replaying a previous test run (use same --quickcheck-max-size) --quickcheck-show-replay Show a replay token for replaying tests --quickcheck-max-size NUMBER Size of the biggest test cases quickcheck generates --quickcheck-max-ratio NUMBER Maximum number of discared tests per successful test before giving up --quickcheck-verbose Show the generated test cases --smallcheck-depth NUMBER Depth to use for smallcheck tests
Every option can be passed via environment. To obtain the environment variable
name from the option name, replace hyphens
- with underscores
all letters, and prepend
TASTY_. For example, the environment equivalent of
Note on boolean options: by convention, boolean ("on/off") options are specified
using a switch on the command line, for example
--quickcheck-show-replay=true. However, when
passed via the environment, the option value needs to be
If you're using a non-console runner, please refer to its documentation to find out how to configure options during the run time.
You can also specify options in the test suite itself, using
localOption. It can be applied not only to the whole test tree, but also to
individual tests or subgroups, so that different tests can be run with
It is possible to combine run-time and compile-time options, too, by using
adjustOption. For example, make the overall testing depth configurable
during the run time, but increase or decrease it slightly for individual
This method currently doesn't work for ingredient options, such as
--num-threads. You can set them by setting the corresponding environment
variable before calling
import Test.Tasty import System.Environment main = do setEnv "TASTY_NUM_THREADS" "1" defaultMain _
It is possible to restrict the set of executed tests using the
Tasty patterns are very powerful, but if you just want to quickly run tests containing
somewhere in their name or in the name of an enclosing test group, you can just
-p foo. If you need more power, or if that didn't work as expected, read
A pattern is an awk expression. When the expression is evaluated, the field
is set to the outermost test group name,
$2 is set to the next test group
name, and so on up to
$NF, which is set to the test's own name. The field
is set to all other fields concatenated using
. as a separator.
As an example, consider a test inside two test groups:
testGroup "One" [ testGroup "Two" [ testCase "Three" _ ] ]
When a pattern is evaluated for the above test case, the available fields and variables are:
$0 = "One.Two.Three" $1 = "One" $2 = "Two" $3 = "Three" NF = 3
Here are some examples of awk expressions accepted as patterns:
$2 == "Two"— select the subgroup
$2 == "Two" && $3 == "Three"— select the test or subgroup named
Threein the subgroup named
$2 == "Two" || $2 == "Twenty-two"— select two subgroups
$0 !~ /skip/or
! /skip/— select tests whose full names (including group names) do not contain the word
$NF !~ /skip/— select tests whose own names (but not group names) do not contain the word
$(NF-1) ~ /QuickCheck/— select tests whose immediate parent group name contains
As an extension to the awk expression language, if a pattern
pat contains only
letters, digits, and characters from the set
._ - (period, underscore, space, hyphen),
it is treated like
/pat/ (and therefore matched against
This is so that we can use
-p foo as a shortcut for
The only deviation from awk that you will likely notice is that Tasty
does not implement regular expression matching.
$1 ~ /foo/ means that the string
foo occurs somewhere in
case-sensitively. We want to avoid a heavy dependency of
similar libraries; however, if there is demand, regular expression support could
be added under a cabal flag.
The following operators are supported (in the order of decreasing precedence):
<center> <table> <tr> <th> <p><b>Syntax</b></p> </th> <th> <p><b>Name</b></p> </th> <th> <p><b>Type of Result</b></p> </th> <th> <p><b>Associativity</b></p> </th> </tr><tr> <td> <p><code>(expr)</code></p> </td> <td> <p>Grouping</p> </td> <td> <p>Type of <code>expr</code></p> </td> <td> <p>N/A</p> </td> </tr> <tr> <td> <p><code>$expr</code></p> </td> <td> <p>Field reference</p> </td> <td> <p>String</p> </td> <td> <p>N/A</p> </td> </tr> <tr> <td> <p><code>!expr</code></p> <p><code>-expr</code></p> </td> <td> <p>Logical not</p> <p>Unary minus</p> </td> <td> <p>Numeric</p> <p>Numeric</p> </td> <td> <p>N/A</p> <p>N/A</p> </td> </tr> <tr> <td> <p><code>expr + expr</code></p> <p><code>expr - expr</code></p> </td> <td> <p>Addition</p> <p>Subtraction</p> </td> <td> <p>Numeric</p> <p>Numeric</p> </td> <td> <p>Left</p> <p>Left</p> </td> </tr> <tr> <td> <p><code>expr expr</code></p> </td> <td> <p>String concatenation</p> </td> <td> <p>String</p> </td> <td> <p>Right</p> </td> </tr> <tr> <td> <p><code>expr < expr</code></p> <p><code>expr <= expr</code></p> <p><code>expr != expr</code></p> <p><code>expr == expr</code></p> <p><code>expr > expr</code></p> <p><code>expr >= expr</code></p> </td> <td> <p>Less than</p> <p>Less than or equal to</p> <p>Not equal to</p> <p>Equal to</p> <p>Greater than</p> <p>Greater than or equal to</p> </td> <td> <p>Numeric</p> <p>Numeric</p> <p>Numeric</p> <p>Numeric</p> <p>Numeric</p> <p>Numeric</p> </td> <td> <p>None</p> <p>None</p> <p>None</p> <p>None</p> <p>None</p> <p>None</p> </td> </tr> <tr> <td> <p><code>expr ~ pat</code></p> <p><code>expr !~ pat</code></p> <p>(<code>pat</code> must be a literal, not an expression, e.g. <code>/foo/</code>)</p> </td> <td> <p>Substring match</p> <p>No substring match</p> </td> <td> <p>Numeric</p> <p>Numeric</p> </td> <td> <p>None</p> <p>None</p> </td> </tr> <tr> <td> <p><code>expr && expr</code></p> </td> <td> <p>Logical AND</p> </td> <td> <p>Numeric</p> </td> <td> <p>Left</p> </td> </tr> <tr> <td> <p><code>expr || expr</code></p> </td> <td> <p>Logical OR</p> </td> <td> <p>Numeric</p> </td> <td> <p>Left</p> </td> </tr> <tr> <td> <p><code>expr1 ? expr2 : expr3</code></p> </td> <td> <p>Conditional expression</p> </td> <td> <p>Type of selected<br><code>expr2</code> or <code>expr3</code></p> </td> <td> <p>Right</p> </td> </tr> </table> </center>
The following built-in functions are supported:
substr(s, m[, n])
Return the at most
n-character substring of
s that begins at
m, numbering from 1. If
n is omitted, or if
more characters than are left in the string, the length of the substring
will be limited by the length of the string
Convert the string
s to lower case.
Convert the string
s to upper case.
Return the position, in characters, numbering from 1, in string
s where the
pat occurs, or zero if it does not occur at all.
pat must be a literal, not an expression, e.g.
Return the length, in characters, of its argument taken as a string, or of the whole record,
$0, if there is no argument.
Running tests in parallel
In order to run tests in parallel, you have to do the following:
- Compile (or, more precisely, link) your test program with the
- Launch the program with
+RTS -N -RTS.
To apply timeout to individual tests, use the
option, or set the option in your test suite using the
Timeouts can be fractional, and can be optionally followed by a suffix
m (minutes), or
h (hours). When there's no
suffix, seconds are assumed.
sets a 30 seconds timeout for each individual test.
Options controlling console output
The following options control behavior of the standard console interface:<dl> <dt><code>-q,--quiet</code></dt> <dd> Run the tests but don't output anything. The result is indicated only by the exit code, which is 1 if at least one test has failed, and 0 if all tests have passed. Execution stops when the first failure is detected, so not all tests are necessarily run. This may be useful for various batch systems, such as commit hooks. </dd> <dt><code>--hide-successes</code></dt> <dd>Report only the tests that has failed. Especially useful when the number of tests is large.</dd> <dt><code>-l,--list-tests</code></dt> <dd>Don't run the tests; only list their names, in the format accepted by <code>--pattern</code>.</dd> <dt><code>--color</code></dt> <dd>Whether to produce colorful output. Accepted values: <code>never</code>, <code>always</code>, <code>auto</code>. <code>auto</code> means that colors will only be enabled when output goes to a terminal and is the default value.</dd> </dl>
It is possible to add custom options, too.
To do that,
- Define a datatype to represent the option, and make it an instance of
- Register the options with the
- To query the option value, use
See the Custom options in Tasty article for some examples.
Project organization and integration with Cabal
There may be several ways to organize your project. What follows is not Tasty's requirements but my recommendations.
Tests for a library
Place your test suite sources in a dedicated subdirectory (called
here) instead of putting them among the main library sources.
The directory structure will be as follows:
my-project/ my-project.cabal src/ ... tests/ test.hs Mod1.hs Mod2.hs ...
test.hs is where your
main function is defined. The tests may be
test.hs or spread across multiple modules (
...) which are then imported by
Add the following section to the cabal file (
test-suite test default-language: Haskell2010 type: exitcode-stdio-1.0 hs-source-dirs: tests main-is: test.hs build-depends: base >= 4 && < 5 , tasty >= 0.7 -- insert the current version here , my-project -- depend on the library we're testing , ...
Tests for a program
All the above applies, except you can't depend on the library if there's no library. You have two options:
- Re-organize the project into a library and a program, so that both the program and the test suite depend on this new library. The library can be declared in the same cabal file.
- Add your program sources directory to the
Hs-source-dirs. Note that this will lead to double compilation (once for the program and once for the test suite).
Tasty executes tests in parallel to make them finish faster. If this parallelism is not desirable, you can declare dependencies between tests, so that one test will not start until certain other tests finish.
Dependencies are declared using the
after AllFinish "pattern" my_testswill execute the test tree
my_testsonly after all tests that match the pattern finish.
after AllSucceed "pattern" my_testswill execute the test tree
my_testsonly after all tests that match the pattern finish and only if they all succeed. If at least one dependency fails, then
my_testswill be skipped.
The relevant types are:
after :: DependencyType -- ^ whether to run the tests even if some of the dependencies fail -> String -- ^ the pattern -> TestTree -- ^ the subtree that depends on other tests -> TestTree -- ^ the subtree annotated with dependency information data DependencyType = AllSucceed | AllFinish
The pattern follows the same AWK-like syntax and semantics as described in
Patterns. There is also a variant named
after_ that accepts the
AST of the pattern instead of a textual representation.
Let's consider some typical examples. (A note about terminology: here
by "resource" I mean anything stateful and external to the test: it could be a file,
a database record, or even a value stored in an
IORef that's shared among
tests. The resource may or may not be managed by
Two tests, Test A and Test B, access the same shared resource and cannot be run concurrently. To achieve this, make Test A a dependency of Test B:
testGroup "Tests accessing the same resource" [ testCase "Test A" $ ... , after AllFinish "Test A" $ testCase "Test B" $ ... ]
Test A creates a resource and Test B uses that resource. Like above, we make Test A a dependency of Test B, except now we don't want to run Test B if Test A failed because the resource may not have been set up properly. So we use
testGroup "Tests creating and using a resource" [ testCase "Test A" $ ... , after AllSucceed "Test A" $ testCase "Test B" $ ... ]
Here are some caveats to keep in mind regarding dependencies in Tasty:
If Test B depends on Test A, remember that either of the may be filtered out using the
--patternoption. Collecting the dependency info happens after filtering. Therefore, if Test A is filtered out, Test B will run unconditionally, and if Test B is filtered out, it simply won't run.
Tasty does not currently check whether the pattern in a dependency matches anything at all, so make sure your patterns are correct and do not contain typos. Fortunately, misspecified dependencies usually lead to test failures and so can be detected that way.
Dependencies shouldn't form a cycle, otherwise Tasty with fail with the message "Test dependencies form a loop." A common cause of this is a test matching its own dependency pattern.
Using dependencies may introduce quadratic complexity. Specifically, resolving dependencies is O(number_of_tests × number_of_dependencies), since each pattern has to be matched against each test name. As a guideline, if you have up to 1000 tests, the overhead will be negligible, but if you have thousands of tests or more, then you probably shouldn't have more than a few dependencies.
Additionally, it is recommended that the dependencies follow the natural order of tests, i.e. that the later tests in the test tree depend on the earlier ones and not vice versa. If the execution order mandated by the dependencies is sufficiently different from the natural order of tests in the test tree, searching for the next test to execute may also have an overhead quadratic in the number of tests.
When my tests write to stdout/stderr, the output is garbled. Why is that and what do I do?
It is not recommended that you print anything to the console when using the console test reporter (which is the default one). See #103 for the discussion.
Some ideas on how to work around this:
- Use testCaseSteps (for tasty-hunit only).
- Use a test reporter that does not print to the console (like tasty-ant-xml).
- Write your output to files instead.
Blog posts and other publications related to tasty. If you wrote or just found something not mentioned here, send a pull request!
- Holy Haskell Project Starter
- First time testing, also with FP Complete (tasty has been added to stackage since then)
- 24 Days of Hackage: tasty
- Resources in Tasty
- Custom options in Tasty
- Resources in Tasty (update)
- Announcing tasty-rerun
- Code testing in Haskell revisited (with Tasty)
- New patterns in tasty
- Screencast: Dynamic Test Suites in Haskell using Hspec and Tasty
- Automatically generated directories for tasty tests
Roman Cheplyaka is the primary maintainer.
Oliver Charles is the backup maintainer. Please get in touch with him if the primary maintainer cannot be reached.