GHC Test framework

GHC includes a comprehensive testsuite for catching any regressions.

The testsuite relies primarily on GNU Make and Python. Any version >= 2.5.2 will do although avoid Python 2.6.1 as the testsuite tickles a bug in one of the included libraries.

If you have not checked out the testsuite, first run:

$ ./sync-all --testsuite get

If you just want to run the whole testsuite, then in the root of the GHC tree, typing:

$ make test

will do a run in "fast" mode (which gives an idea whether there are major problems), or

$ make fulltest

will do a full testsuite run (more thorough, but takes a lot longer). You should expect that there are no test case failures for the "fast" mode as as that is a quality level that all GHC developers are expected to maintain when they check in code. There will usually be some test case failures for the full testsuite run though.

Using the Testsuite

• Running the testsuite
• Testsuite Settings and WAYS
• Updating test case results
• Adding new test cases
• Testsuite implementation details

Problems running the testsuite

1. If the testsuite fails mysteriously, make sure that the timeout utility is working properly. This Haskell utility is compiled with the stage 1 compiler and invoked by the python driver, which does not print a nice error report if the utility fails. This can happen if, for example, the compiler produces bogus binaries. A workaround is to compile timeout with a stable ghc.

The testsuite and version control branches

It is not clear what to do with the testsuite when branching a compiler; should the testsuite also be branched?

If it is not branched then we have the problem that, given a set of tests

test(tc1, ...)
test(tc2, ...)
test(tc3, ...)

if we add first one test, and then another to the HEAD

test(tc1, ...)
test(tc2, ...)
test(tc3, ...)
test(tc4, ...)
test(tc5, ...)

and we want to merge tc5 but not tc4 to the branch then the merge has to be done by hand, as the patch for tc5 depends on the patch for tc4, although most of the files in the patches (tc5.hs etc) are disjoint.

On the other hand, if it is not branched then any changes in test output mean we need to add extra logic to the test definitions, e.g.

test(tc5, namebase_if_compiler_lt('ghc','6.9', 'tc5-6.8'), ...)