Copyright  (c) 20152017 Rudy Matela 

License  3Clause BSD (see the file LICENSE) 
Maintainer  Rudy Matela <rudy@matela.com.br> 
Safe Haskell  None 
Language  Haskell2010 
FitSpec: refining propertysets for functional testing
FitSpec provides automated assistance in the task of refining test properties for Haskell functions. FitSpec tests mutant variations of functions under test against a given property set, recording any surviving mutants that pass all tests. FitSpec then reports:
 surviving mutants: indicating incompleteness of properties, prompting the user to amend a property or to add a new one;
 conjectures: indicating redundancy in the property set, prompting the user to remove properties so to reduce the cost of testing.
Example, refining a sort
specification:
import Test.FitSpec import Data.List (sort) properties sort = [ property $ \xs > ordered (sort xs) , property $ \xs > length (sort xs) == length xs , property $ \x xs > elem x (sort xs) == elem x xs , property $ \x xs > notElem x (sort xs) == notElem x xs , property $ \x xs > minimum (x:xs) == head (sort (x:xs)) ] where ordered (x:y:xs) = x <= y && ordered (y:xs) ordered _ = True main = mainWith args { names = ["sort xs"] , nMutants = 4000 , nTests = 4000 , timeout = 0 } (sort::[Word2]>[Word2]) properties
The above program reports the following:
Apparent incomplete and nonminimal specification based on 4000 test cases for each of properties 1, 2, 3, 4 and 5 for each of 4000 mutant variations. 3 survivors (99% killed), smallest: \xs > case xs of [0,0,1] > [0,1,1] _ > sort xs apparent minimal property subsets: {1,2,3} {1,2,4} conjectures: {3} = {4} 96% killed (weak) {1,3} ==> {5} 98% killed (weak)
Synopsis
 type Property = [([String], Bool)]
 property :: Testable a => a > Property
 data Args = Args {}
 data ShowMutantAs
 args :: Args
 fixargs :: Int > Int > Args
 report :: (Mutable a, ShowMutable a) => a > (a > [Property]) > IO ()
 reportWith :: (Mutable a, ShowMutable a) => Args > a > (a > [Property]) > IO ()
 reportWithExtra :: (Mutable a, ShowMutable a) => [a] > Args > a > (a > [Property]) > IO ()
 mainWith :: (Mutable a, ShowMutable a) => Args > a > (a > [Property]) > IO ()
 defaultMain :: (Mutable a, ShowMutable a) => a > (a > [Property]) > IO ()
 getArgs :: IO Args
 getArgsWith :: Args > IO Args
 class Mutable a where
 mutiersEq :: (Listable a, Eq a) => a > [[a]]
 class ShowMutable a where
 mutantSEq :: (Eq a, Show a) => a > a > MutantS
 showMutantAsTuple :: ShowMutable a => [String] > a > a > String
 showMutantDefinition :: ShowMutable a => [String] > a > a > String
 showMutantNested :: ShowMutable a => [String] > a > a > String
 showMutantBindings :: ShowMutable a => [String] > a > a > String
 deriveMutable :: Name > DecsQ
 deriveMutableE :: [Name] > Name > DecsQ
 deriveMutableCascading :: Name > DecsQ
 deriveMutableCascadingE :: [Name] > Name > DecsQ
 module Test.FitSpec.TestTypes
 module Test.LeanCheck
Encoding properties
type Property = [([String], Bool)] Source #
An encoded representation of a property suitable for use by FitSpec.
Each list of strings is a printable representation of one possible choice of argument values for the property. Each boolean indicate whether the property holds for this choice.
property :: Testable a => a > Property Source #
Given a Testable
type (as defined by Test.LeanCheck), returns a Property
.
This function should be used on every property to create a property list to
be passed to report
, reportWith
, mainDefault
or mainWith
.
property $ \x y > x + y < y + (x::Int)
Configuring reports
Extra arguments / configuration for reportWith
.
See args
for default values.
Args  

Instances
Data Args #  
Defined in Test.FitSpec.Main gfoldl :: (forall d b. Data d => c (d > b) > d > c b) > (forall g. g > c g) > Args > c Args # gunfold :: (forall b r. Data b => c (b > r) > c r) > (forall r. r > c r) > Constr > c Args # dataTypeOf :: Args > DataType # dataCast1 :: Typeable t => (forall d. Data d => c (t d)) > Maybe (c Args) # dataCast2 :: Typeable t => (forall d e. (Data d, Data e) => c (t d e)) > Maybe (c Args) # gmapT :: (forall b. Data b => b > b) > Args > Args # gmapQl :: (r > r' > r) > r > (forall d. Data d => d > r') > Args > r # gmapQr :: (r' > r > r) > r > (forall d. Data d => d > r') > Args > r # gmapQ :: (forall d. Data d => d > u) > Args > [u] # gmapQi :: Int > (forall d. Data d => d > u) > Args > u # gmapM :: Monad m => (forall d. Data d => d > m d) > Args > m Args # gmapMp :: MonadPlus m => (forall d. Data d => d > m d) > Args > m Args # gmapMo :: MonadPlus m => (forall d. Data d => d > m d) > Args > m Args # 
data ShowMutantAs Source #
How to show mutants. Use this to fill showMutantAs
.
Instances
Data ShowMutantAs #  
Defined in Test.FitSpec.Main gfoldl :: (forall d b. Data d => c (d > b) > d > c b) > (forall g. g > c g) > ShowMutantAs > c ShowMutantAs # gunfold :: (forall b r. Data b => c (b > r) > c r) > (forall r. r > c r) > Constr > c ShowMutantAs # toConstr :: ShowMutantAs > Constr # dataTypeOf :: ShowMutantAs > DataType # dataCast1 :: Typeable t => (forall d. Data d => c (t d)) > Maybe (c ShowMutantAs) # dataCast2 :: Typeable t => (forall d e. (Data d, Data e) => c (t d e)) > Maybe (c ShowMutantAs) # gmapT :: (forall b. Data b => b > b) > ShowMutantAs > ShowMutantAs # gmapQl :: (r > r' > r) > r > (forall d. Data d => d > r') > ShowMutantAs > r # gmapQr :: (r' > r > r) > r > (forall d. Data d => d > r') > ShowMutantAs > r # gmapQ :: (forall d. Data d => d > u) > ShowMutantAs > [u] # gmapQi :: Int > (forall d. Data d => d > u) > ShowMutantAs > u # gmapM :: Monad m => (forall d. Data d => d > m d) > ShowMutantAs > m ShowMutantAs # gmapMp :: MonadPlus m => (forall d. Data d => d > m d) > ShowMutantAs > m ShowMutantAs # gmapMo :: MonadPlus m => (forall d. Data d => d > m d) > ShowMutantAs > m ShowMutantAs # 
Default arguments for reportWith
:
nMutants = 500
, start with 500 mutantsnTests = 1000
, start with 1000 test valuestimeout = 5
, keep incresing the number of mutants until 5 seconds elapsenames = []
, default function call template:
["f x y z", "g x y z", "h x y z", ...]
fixargs :: Int > Int > Args Source #
Non timedout default arguments. Make conjectures based on a fixed number of mutants and tests, e.g.:
reportWith (fixargs 100 200) f pmap
This is just a shorthand, see:
fixargs nm nt = args { nMutants = nm, nTests = nt, timeout = 0 }
(fixargs nm nt) { nMutants = 500, nTests = 1000, timeout = 5 } = args
Reporting results
report :: (Mutable a, ShowMutable a) => a > (a > [Property]) > IO () Source #
Report results generated by FitSpec.
Uses standard configuration (see args
).
Needs a function to be mutated and a property map.
Example (specification of boolean negation):
properties not = [ property $ \p > not (not p) == p , property $ \p > not (not (not p)) == not p ] main = report not properties
reportWith :: (Mutable a, ShowMutable a) => Args > a > (a > [Property]) > IO () Source #
reportWithExtra :: (Mutable a, ShowMutable a) => [a] > Args > a > (a > [Property]) > IO () Source #
Same as reportWith
, but accepts a list of manually defined (extra)
mutants to be tested alongside those automatically generated.
Parsing command line arguments
mainWith :: (Mutable a, ShowMutable a) => Args > a > (a > [Property]) > IO () Source #
Same as reportWith
, but allow overriding of configuration via command
line arguments.
defaultMain :: (Mutable a, ShowMutable a) => a > (a > [Property]) > IO () Source #
Same as report
, but allow configuration via command line arguments.
Mutable types
class Mutable a where Source #
This typeclass is similar to Listable
.
A type is Mutable
when there exists a function that
is able to list mutations of a value.
Ideally: list all possible values without repetitions.
Instances are usually defined by a mutiers
function that
given a value, returns tiers of mutants of that value:
the first tier contains the equivalent mutant, of size 0,
the second tier contains mutants of size 1,
the third tier contains mutants of size 2,
and so on.
The equivalent mutant is the actual function without mutations.
The size of a mutant is given by the sum of: the number of mutated points (relations) and the sizes of mutated arguments and results.
To get only inequivalent mutants,
just take the tail
of either mutants
or mutiers
:
tail mutants
tail mutiers
Given that the underlying Listable
enumeration has no repetitions,
parametric instances defined in this file will have no repeated mutants.
Instances
mutiersEq :: (Listable a, Eq a) => a > [[a]] Source #
Implementation of mutiers
for nonfunctional data types.
Use this to create instances for userdefined data types, e.g.:
instance MyData where mutiers = mutiersEq
and for parametric datatypes:
instance (Eq a, Eq b) => MyDt a b where mutiers = mutiersEq
Examples:
mutiersEq True = [[True], [False]] mutiersEq 2 = [[2], [0], [1], [], [3], [4], [5], [6], [7], [8], [9], ...] mutiersEq [1] = [[[1]], [[]], [[0]], [[0,0]], [[0,0,0],[0,1],[1,0],[1]], ...]
class ShowMutable a where Source #
Types that can have their mutation shown.
Has only one function mutantS
that returns a simple AST (MutantS
)
representing the mutant. A standard implementation of mutantS
for Eq
types is given by mutantSEq
.
Instances
mutantSEq :: (Eq a, Show a) => a > a > MutantS Source #
For a given type Type
instance of Eq
and Show
,
define the ShowMutable
instance as:
instance ShowMutable Type where mutantS = mutantSEq
showMutantAsTuple :: ShowMutable a => [String] > a > a > String Source #
Show a Mutant as a tuple of lambdas.
> putStrLn $ showMutantAsTuple ["p && q","not p"] ((&&),not) ((),id) ( \p q > case (p,q) of (False,False) > True _ > p && q , \p > case p of False > False True > True _ > not p )
Can be easily copy pasted into an interactive session for manipulation.
On GHCi, use :{
and :}
to allow multiline expressions and definitions.
showMutantDefinition :: ShowMutable a => [String] > a > a > String Source #
Show a Mutant as a new complete toplevel definition, with a prime appended to the name of the mutant.
> putStrLn $ showMutantDefinition ["p && q","not p"] ((&&),not) ((==),id) False && False = True p && q = p && q not' False = False not' True = True not' p = not p
showMutantNested :: ShowMutable a => [String] > a > a > String Source #
Show a Mutant as a tuple of nested lambdas.
Very similar to showMutantAsTuple
, but the underlying data structure is
not flatten: so the output is as close as possible to the underlying
representation.
showMutantBindings :: ShowMutable a => [String] > a > a > String Source #
Show a Mutant as the list of bindings that differ from the original function(s).
> putStrLn $ showMutantBindings ["p && q","not p"] ((&&),not) ((==),id) False && False = True not False = False not True = True
Can possibly be copied into the source of the original function for manipulation.
Automatic derivation
deriveMutable :: Name > DecsQ Source #
Derives Mutable
, ShowMutable
and (optionally) Listable
instances
for a given type Name
.
Consider the following Stack
datatype:
data Stack a = Stack a (Stack a)  Empty
Writing
deriveMutable ''Stack
will automatically derive the following
Listable
, Mutable
and ShowMutable
instances:
instance Listable a => Listable (Stack a) where tiers = cons2 Stack \/ cons0 Empty instance (Eq a, Listable a) => Mutable a where mutiers = mutiersEq instance (Eq a, Show a) => ShowMutable a where mutantS = mutantSEq
If a Listable
instance already exists, it is not derived.
(cf.: deriveListable
)
Needs the TemplateHaskell
extension.
deriveMutableE :: [Name] > Name > DecsQ Source #
Derives a Mutable instance for a given type Name
using a given context for all type variables.
deriveMutableCascading :: Name > DecsQ Source #
Reexport modules
module Test.FitSpec.TestTypes
module Test.LeanCheck