id	summary	reporter	owner	description	type	status	priority	resolution	keywords	cc	topic	difficulty	mentor
1581	Improve Cabal's test support	tibbe		"= Motivation =

Proper test support is essential for good software quality. By improving Cabal's test support we can test all Cabal packages on continuous build machines which should help us detect breakages earlier. Making it easier to run the tests means that more people will run them and those who already do will run the more often.

= Note =

The task is described in detail in the Cabal ticket [http://hackage.haskell.org/trac/hackage/ticket/215 Overhaul support for package's tests]. The description below is copied from the ticket.

= Description =

The current support for running package's testsuites is almost uniformly unused and packages instead implement ad-hoc solutions which makes it hard for users to run the testsuites and impossible to automate on hackage.

The current support is so trivial anyone who takes this task on can ignore it and start with a clean slate.

A [http://www.haskell.org/pipermail/cabal-devel/2008-January/001661.html thread on the issue].

Here are some ideas for possible requirements:

 * Support tests even in the Simple build system. Do not require using the build-type: Custom just to be able to use tests.

 * Define a standard interface for returning test results from a testsuite to Cabal so they can be included into a build report and could be published on the hackage website or some other custom in-house test server.

 * Support for above interface in QuickCheck and HUnit.

 * Support for HPC when running testsuite programs, and gathering coverage results, again for reporting purposes.

 * We want to be able to run tests on a library that is not installed yet. This will rely on support in Cabal for registering a package inplace in a local package db, and building test code against that.

 * What about performance tests? What format should that use for reporting?

The emphasis is on automation. A lot of interesting information becomes available with enough automation. For example we could compare on hackage which packages have more comprehensive testsuites (measured by HPC coverage). Or we could track performance over time. Once the data is available there is a lot we could do with it. It's not just for a public hackage server, developers could use this privately too.

= Interested Mentors =

 * Johan Tibell
 * Rogan Creswick
 * Duncan Coutts (as an adviser rather than primary mentor)
"	proposed-project	closed	good	fixed		michal.palka@… creswick@…	Cabal	unknown	not-accepted