LeanCheck ========= [![LeanCheck's Build Status][build-status]][build-log] [![LeanCheck on Hackage][hackage-version]][leancheck-on-hackage] [![LeanCheck on Stackage LTS][stackage-lts-badge]][leancheck-on-stackage-lts] [![LeanCheck on Stackage Nightly][stackage-nightly-badge]][leancheck-on-stackage-nightly] ![LeanCheck logo][leancheck-logo] LeanCheck is a simple enumerative [property-based testing] library. Properties are defined as Haskell functions returning a boolean value which should be `True` for all possible choices of argument values. LeanCheck applies enumerated argument values to these properties in search for a counterexample. Properties can be viewed as parameterized unit tests. LeanCheck works by producing *tiers* of test values: a possibly infinite list of finite sublists of same-and-increasingly-sized values. This enumeration is similar to [Feat]'s. However, the ranking and ordering of values are defined differently. The interface is also different. Throughout this README lines that begin with the [symbol `>`] indicate a line entered into an interactive interpreter (`ghci`). The result of evaluating the expression is then printed on the following line. LeanCheck implementation is easy to understand. [LeanCheck's core] is under 190 lines of code. Installing ---------- To install the latest LeanCheck version from Hackage using Cabal v3.0 or later, just run: $ cabal v1-update $ cabal v1-install leancheck If you are using a cabal earlier than v3.0, just run: $ cabal update $ cabal install leancheck With Cabal v3.0 or later, [using `v1-install` is preferred for the time being.](https://github.com/haskell/cabal/issues/7373) If all fails, you can start over by resetting your user's cabal installation with: rm -rf ~/.cabal/{bin,lib,logs,share,store} ~/.ghc/*/ WARNING: the above command will erase all user-local packages. LeanCheck has (official) packages available on [Stackage](https://www.stackage.org/package/leancheck), [OpenSUSE](https://packagehub.suse.com/packages/ghc-leancheck/), [Gentoo](https://packages.gentoo.org/packages/dev-haskell/leancheck), [Arch Linux](https://archlinux.org/packages/community/x86_64/haskell-leancheck/) and [NixOS](https://hydra.nixos.org/job/nixpkgs/trunk/haskellPackages.leancheck.x86_64-linux). Checking if properties are True ------------------------------- To check if properties are True, just use the function [`holds`] `:: Testable a => Int -> a -> Bool`. It takes _two arguments_: the _number of values_ to test and a _property_ (function returning Bool), then, it returns a boolean indicating whether the property holds. See (ghci): > import Test.LeanCheck > import Data.List > holds 100 $ \xs -> sort (sort xs) == sort (xs::[Int]) True > holds 100 $ \xs -> [] `union` xs == (xs::[Int]) False As a rule-of-thumb, you should run holds for 500, 1 000, or 10 000 tests. With more than that you may run out-of-memory depending on the types being tested. Finding counter examples ------------------------ To find counter examples to properties, you can use the function [`counterExample`] `:: Testable a => Int -> a -> Maybe [String]`. It takes _two arguments_: the _number of values_ to test and a _property_ (function returning Bool). Then, it returns Nothing if no results are found or Just a list of Strings representing the offending arguments to the property. See (ghci): > import Test.LeanCheck > import Data.List > counterExample 100 $ \xs -> sort (sort xs) == sort (xs::[Int]) Nothing > counterExample 100 $ \xs -> [] `union` xs == (xs::[Int]) Just ["[0,0]"] > counterExample 100 $ \xs ys -> xs `union` ys == ys `union` (xs::[Int]) Just ["[]","[0,0]"] Checking properties like in SmallCheck/QuickCheck ------------------------------------------------- To "check" properties like in [SmallCheck] and [QuickCheck] automatically printing results on standard output, you can use the function [`check`] `:: Testable a => a -> IO ()`. > import Test.LeanCheck > import Data.List > check $ \xs -> sort (sort xs) == sort (xs::[Int]) +++ OK, passed 200 tests. > check $ \xs ys -> xs `union` ys == ys `union` (xs::[Int]) *** Failed! Falsifiable (after 4 tests): [] [0,0] The function [`check`] tests for a maximum of 200 tests. To check for a maximum of `n` tests, use [`checkFor`] `n`. To get a boolean result wrapped in `IO`, use [`checkResult`] or [`checkResultFor`]. There is no "quiet" option, just use [`holds`] or [`counterExample`] in that case. Testing user-defined types -------------------------- LeanCheck works on properties with [`Listable`] argument types. [`Listable`] instances are declared similarly to SmallCheck: data MyType = MyConsA | MyConsB Int | MyConsC Int Char | MyConsD String instance Listable MyType where tiers = cons0 MyConsA \/ cons1 MyConsB \/ cons2 MyConsC \/ cons1 MyConsD For types that do not have a constraning data invariant, instances can be automatically derived with [Template Haskell] by using [`deriveListable`] like so: deriveListable ''MyType The [`tiers`] function return a potentially infinite list of finite sub-lists (tiers). Each successive tier has values of increasing size. tiers :: Listable a => [[a]] For convenience, the function [`list`] returns a potentially infinite list of values of the bound type: list :: Listable a => [a] So, for example: > take 5 (list :: [(Int,Int)]) [(0,0),(0,1),(1,0),(0,-1),(1,1)] The `list` function can be used to debug your custom instances. [`Listable`] class instances are more customizable than what is described here: check source comments or haddock documentation for details. Standard Listable Instances --------------------------- LeanCheck comes out-of-the-box with [`Listable`] instances for all types in the [Haskell 2010 Language Report] with [the intentional exception of a few types]. The [leancheck-instances] package aims to support types in the [Haskell Platform] -- `$ cabal install leancheck-instances`. Providers for Tasty, test-framework and Hspec --------------------------------------------- The following providers allow including LeanCheck properties into [Tasty], [test-framework] or [Hspec] test suites. * [LeanCheck provider for Tasty] -- `$ cabal install tasty-leancheck` ; * [LeanCheck provider for test-framework] -- `$ cabal install test-framework-leancheck` ; * [LeanCheck provider for Hspec] -- `$ cabal install hspec-leancheck` . Memory usage ------------ Due to the way it is implemented (using lists of lists), LeanCheck can be quite memory intensive if we set the maximum number of tests of a property to millions of values (YMMV). For the default maximum number of tests (200) you should be safe on most cases. If you use 1 000 or 10 000 as the maximum number of tests for a property you're also generally safe. More than that, it is in a hit or miss basis. For more details, see [LeanCheck memory usage]. Beginner friendliness --------------------- LeanCheck strives to be beginner/student friendly both in the interface and its implementation. For instance, to understand [LeanCheck's core], one does not need to understand Monads as they aren't used at all there. In the name of keeping the implementation easy to understand, a compromise were made in terms of performance (cf. [LeanCheck memory usage]). LeanCheck is mostly [Haskell 98] compliant and almost [Haskell 2010] compliant. With the exception of [`Listable`] derivation modules ([TH] and [Generics]), the only extension used by LeanCheck is [CPP]. This is to maintain compatibility with different compilers. LeanCheck even compiles and runs on [Hugs98] from September 2006. LeanCheck has 100% Haddock coverage with most functions having examples. Further reading --------------- For a detailed documentation of each function, see [LeanCheck's Haddock documentation]. For an introduction to property-based testing and a step-by-step guide to LeanCheck, see the [tutorial on property-based testing with LeanCheck] \(`doc/tutorial.md` in the source repository). LeanCheck is subject to a chapter in a [PhD Thesis (2017)]. LeanCheck has a list of [frequently asked questions] and answers. [LeanCheck's Haddock documentation]: https://hackage.haskell.org/package/leancheck/docs/Test-LeanCheck.html [tutorial on property-based testing with LeanCheck]: https://github.com/rudymatela/leancheck/blob/master/doc/tutorial.md [LeanCheck memory usage]: https://github.com/rudymatela/leancheck/blob/master/doc/memory-usage.md [frequently asked questions]: https://github.com/rudymatela/leancheck/blob/master/doc/faq.md [`Listable`]: https://hackage.haskell.org/package/leancheck/docs/Test-LeanCheck.html#t:Listable [`holds`]: https://hackage.haskell.org/package/leancheck/docs/Test-LeanCheck.html#v:holds [`counterExample`]: https://hackage.haskell.org/package/leancheck/docs/Test-LeanCheck.html#v:counterExample [`check`]: https://hackage.haskell.org/package/leancheck/docs/Test-LeanCheck.html#v:check [`checkFor`]: https://hackage.haskell.org/package/leancheck/docs/Test-LeanCheck.html#v:checkFor [`checkResult`]: https://hackage.haskell.org/package/leancheck/docs/Test-LeanCheck.html#v:checkResult [`checkResultFor`]: https://hackage.haskell.org/package/leancheck/docs/Test-LeanCheck.html#v:checkResultFor [`tiers`]: https://hackage.haskell.org/package/leancheck/docs/Test-LeanCheck.html#v:tiers [`list`]: https://hackage.haskell.org/package/leancheck/docs/Test-LeanCheck.html#v:list [`deriveListable`]: https://hackage.haskell.org/package/leancheck/docs/Test-LeanCheck.html#v:deriveListable [LeanCheck's core]: https://github.com/rudymatela/leancheck/blob/master/src/Test/LeanCheck/Core.hs [property-based testing]: https://github.com/rudymatela/leancheck/blob/master/doc/tutorial.md [Feat]: https://hackage.haskell.org/package/testing-feat [SmallCheck]: https://hackage.haskell.org/package/smallcheck [QuickCheck]: https://hackage.haskell.org/package/QuickCheck [PhD Thesis (2017)]: https://matela.com.br/paper/rudy-phd-thesis-2017.pdf [symbol `>`]: https://www.haskell.org/haddock/doc/html/ch03s08.html#idm140354810780208 [Template Haskell]: https://wiki.haskell.org/Template_Haskell [Tasty]: https://github.com/feuerbach/tasty#readme [test-framework]: https://haskell.github.io/test-framework/ [Hspec]: https://hspec.github.io/ [LeanCheck provider for Tasty]: https://hackage.haskell.org/package/tasty-leancheck [LeanCheck provider for test-framework]: https://hackage.haskell.org/package/test-framework-leancheck [LeanCheck provider for Hspec]: https://hackage.haskell.org/package/hspec-leancheck [leancheck-instances]: https://hackage.haskell.org/package/leancheck-instances [the intentional exception of a few types]: https://hackage.haskell.org/package/leancheck/docs/Test-LeanCheck-Basic.html [Haskell 2010 Language Report]: https://www.haskell.org/onlinereport/haskell2010/ [Haskell 2010]: https://www.haskell.org/onlinereport/haskell2010/ [Haskell 98]: https://www.haskell.org/onlinereport/ [Haskell Platform]: https://www.haskell.org/platform/ [Hugs98]: https://www.haskell.org/hugs/ [TH]: https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/glasgow_exts.html#template-haskell [CPP]: https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/phases.html#extension-CPP [Generics]: https://hackage.haskell.org/package/base/docs/GHC-Generics.html [leancheck-logo]: https://github.com/rudymatela/leancheck/raw/master/doc/leancheck.svg?sanitize=true [build-log]: https://github.com/rudymatela/leancheck/actions/workflows/build.yml [build-status]: https://github.com/rudymatela/leancheck/actions/workflows/build.yml/badge.svg [hackage-version]: https://img.shields.io/hackage/v/leancheck.svg [leancheck-on-hackage]: https://hackage.haskell.org/package/leancheck [stackage-lts-badge]: https://stackage.org/package/leancheck/badge/lts [stackage-nightly-badge]: https://stackage.org/package/leancheck/badge/nightly [leancheck-on-stackage]: https://stackage.org/package/leancheck [leancheck-on-stackage-lts]: https://stackage.org/lts/package/leancheck [leancheck-on-stackage-nightly]: https://stackage.org/nightly/package/leancheck