# Skeletest

[![GitHub Actions](https://img.shields.io/github/actions/workflow/status/brandonchinn178/skeletest/ci.yml?branch=main)](https://github.com/brandonchinn178/skeletest/actions?query=branch%3Amain)
[![](https://img.shields.io/hackage/v/skeletest)](https://hackage.haskell.org/package/skeletest)

Skeletest is a batteries-included, opinionated test framework heavily inspired by [pytest](https://pytest.org) and [jest](https://jestjs.io). It's the built-in test framework for [Skelly](https://github.com/brandonchinn178/skelly), but it can be used as a standalone library as well.

Features:
* Seamless experience writing unit tests, property tests, and snapshot tests
* Descriptive failure messages
* Easy test selection from CLI
* Automatic fixtures management
* Rich plugin + hooks functionality

## Example

```haskell
import Skeletest
import qualified Skeletest.Predicate as P
import qualified Skeletest.Prop.Gen as Gen
import qualified Skeletest.Prop.Range as Range

spec :: Spec
spec = do
  describe "myFunc" $ do
    it "returns the correct list" $
      myFunc 1 2 `shouldBe` ["a", "b", "c"]

    it "returns a list containing an element" $
      myFunc 1 2 `shouldSatisfy` P.any (P.eq "a")

    it "returns a list matching the given predicates" $
      myFunc 1 2 `shouldSatisfy` P.list [P.eq "a", P.anything, P.anything]

    prop "myFunc 0 x == []" $ do
      x <- forAll $ Gen.int (Range.linear 0 100)
      myFunc 0 x `shouldBe` ""

    prop "myFunc x y == myFunc y x" $ do
      x <- forAll $ Gen.int (Range.linear 0 100)
      y <- forAll $ Gen.int (Range.linear 0 100)
      myFunc x y `shouldBe` myFunc y x

  -- top-level property that's not grouped under
  -- either myFunc nor otherFunc
  prop "myFunc x . otherFunc === id" $ do
    x <- forAll $ Gen.int (Range.linear 0 100)
    let input = 
          Gen.list (Range.linear 0 10) $
            Gen.string (Range.linear 0 100) Gen.unicode
    (myFunc x . otherFunc) P.=== id `shouldSatisfy` P.isoWith input

  describe "ioFunc" $ do
    it "returns the correct string" $ do
      DbConnFixture conn <- getFixture
      ioFunc conn 1 `shouldSatisfy` P.returns (P.eq "hello world")

    it "errors on bad input" $ do
      DbConnFixture conn <- getFixture
      ioFunc conn (-1) `shouldSatisfy` P.throws (P.eq MyException)

    it "returns the expected result" $ do
      DbConnFixture conn <- getFixture
      ioFunc conn 100 `shouldSatisfy` P.matchesSnapshot

  describe "getUser" $ do
    it "returns a matching user" $ do
      getUser "user1" `shouldSatisfy` P.con User{name = P.eq "user1", email = P.contains "@"}

newtype DbConnFixture = DbConnFixture Connection

instance Fixture DbConnFixture where
  fixtureAction = do
    conn <- initDBConn
    setupTestTables conn
    pure . withCleanup (DbConnFixture conn) $ do
      destroyTestTables conn
      closeConn conn
```

## Quickstart

1. If you're using Skeletest as a standalone library, add the following to your cabal file:

    ```cabal
    test-suite my-tests
      ghc-options: -F -pgmF=skeletest-preprocessor
      build-tool-depends: skeletest:skeletest-preprocessor
    ```

1. Add `Main.hs`:

    ```haskell
    import Skeletest.Main
    ```

1. To test some module `MyLib.Foo`, add a new file `MyLib/FooSpec.hs`:

    ```haskell
    module MyLib.FooSpec (spec) where

    import Skeletest
    import qualified Skeletest.Predicate as P

    spec :: Spec
    spec = do
      describe "myFunc" $ do
        it "does a thing" $ do
          myFunc 1 `shouldBe` 2
    ```

## Guide

### Defining tests

Tests should be defined in a `spec` identifier with the type `Spec`. A `Spec` is defined as a tree of tests, written using do-notation. The entire `Spec` is wrapped in an implicit `describe` containing the name of the module without the `Spec` suffix. The `describe` and `it` functions are intended to read nicely if you use types or function names as `describe` groups.

```haskell
spec :: Spec
spec = do
  -- A property test, grouped under the implicit module group.
  -- See the "Property tests" section.
  prop "encodeUser . decodeUser === id" $ do
    ...

  describe "encodeUser" $ do
    -- A unit test testing a particular aspect of the encodeUser function.
    -- See the "Unit tests" section.
    it "encodes a user with a name" $ do
      ...

    it "encodes an empty user" $ do
      ...
```

Tests can also be marked as `xfail` or `skip`. `xfail` tests will succeed if the test fails, or fail if the test unexpectedly passes. `skip` tests will skip running the test entirely. Both `xfail` and `skip` require a message explaining the reason it; this is a good place to put links to the relevant ticket or issue.

```haskell
-- xfail a single test
xfail "https://github.com/my-company/my-repo/issues/123" . it "does a thing" $ do
  ...

-- xfail multiple tests
xfail "https://github.com/my-company/my-repo/issues/123" $ do
  it "does a thing" $ do
    ...

  it "does another thing" $ do
    ...

-- skip a whole describe
skip "https://github.com/my-company/my-repo/issues/123" . describe "myFunc" $ do
  it "does a thing" $ do
    ...
```

`markManual` marks tests in the given section as manual tests, which means they won't be run when no tests are selected (see the "Test selection" section).

```haskell
markManual $ do
  ...
```

### Test selection

Test targets are specified as plain positional arguments, with the following syntax:

| Target | Explanation |
|--------|-------------|
| `*`                     | Selects all tests (useful to include manual tests) |
| `[myFooFunc]`           | Tests including substring      |
| `@fast`                 | Tests tagged with marker       |
| `test/MyLib/FooSpec.hs` | Tests in file, relative to CWD |
| `test/MyLib/FooSpec.hs[myFooFunc]` | Syntax sugar for `(test/MyLib/FooSpec.hs and [myFooFunc])` |
| `[func1] and @fast`     | Tests matching both targets    |
| `[func1] or @fast`      | Tests matching either target   |
| `not [func1]`           | Tests not matching target      |

Some more examples:
* `test/MySpec.hs and ([myFooFunc] or [myBarFunc]) and @fast`
* `[myFooFunc] or test/MySpec.hs[myBarFunc]`
 
When multiple targets are specified, they are joined with `or`.

### Assertions and Predicates

All assertions in Skeletest use the following functions:

* `shouldSatisfy`
* `shouldNotSatisfy` - equivalent to `shouldSatisfy` with `P.not`
* `shouldBe` - equivalent to `shouldSatisfy` with `P.eq`
* `shouldNotBe` - equivalent to `shouldNotSatisfy` with `P.eq`

`shouldSatisfy` is the most general function, but the others are provided for convenience. `shouldSatisfy` takes in the value being tested on the left, and a predicate on the right. Predicates should be imported from `Skeletest.Predicate`, qualified as `P`.

Some notable predicates are listed here. See the [Haddocks](https://hackage.haskell.org/package/skeletest/docs/Skeletest-Predicate.html) for a full list of available predicates.

* `P.eq 10`
    * Satisfied when the actual value is equal to `10`.

* `P.just (P.gt 10)`
    * Satisfied when the actual value is a `Just` containing a value greater than `10`.

* `P.tup (P.eq 10, P.anything)`
    * Satisfied when the actual value is a tuple where the first element is `10` and the second element is anything. `P.tup` works for any tuple up to 6 elements.

* `P.con User{name = P.hasPrefix "user_"}`
    * Satisfied when the actual value is a `User` whose `name` field starts with `user_`. Omitted fields are not checked and can contain anything. `P.con` also works for positional constructors, except all arguments must contain predicates.

* `P.eq 10 P.<<< f`
    * Satisfied when the actual value is equal to `10` after being applied to `f`.

* `P.approx P.tol 0.5`
    * Satisfied when the actual value is approximately equal to `0.5`. Useful for floating point values. See docs for an explanation of how to adjust the tolerance with `P.tol`.

* `P.and [P.gt 0, P.lt 10]`
    * Satisfied when the actual value satisfies all of the given predicates. For just two predicates, `P.&&` can be used. Also see: `P.or`, `P.||`

* `P.returns (P.gt 10)`
    * Satisfied when the left hand side is an `IO` action that returns a value greater than `10`.

* `P.throws (P.eq MyException)`
    * Satisfied when the left hand side is an `IO` action that throws the given exception.

### Unit and Integration tests

Unit and integration tests are written with `it`, and run in `IO`.

```haskell
describe "mkUser" $ do
  it "creates a user with the given name" $ do
    let x = mkUser (Just "alice")
    x `shouldBe` User{name = "alice"}

    mkUser (Just "alice") `shouldBe` User{name = "alice"}

  it "creates a user with a default name" $ do
    mkUser Nothing `shouldSatisfy` P.con User{name = P.hasPrefix "user_"}

describe "addService" $ do
  it "queries the addition service" $ do
    x <- addService 1 2
    x `shouldBe` 3

    addService 1 2 `shouldSatisfy` P.returns (P.eq 3)
```

### Snapshot tests

[Snapshot tests](https://ro-che.info/articles/2017-12-04-golden-tests) can be done in any kind of test, although it's usually done in unit tests. Snapshot tests are best suited for testing that behavior doesn't change; they aren't great for testing _correctness_.

To write a snapshot test, simply use the `P.matchesSnapshot` predicate.

```haskell
myFunc 1 `shouldSatisfy` P.matchesSnapshot

-- can also snapshot within a nested predicate
fetchUserFromDb "alice" `shouldSatisfy` P.returns (P.just P.matchesSnapshot)
```

When running for the first time, or when the snapshot is changing, use the `-u`/`--update` flag. For a given test file `MyLib/FooSpec.hs`, snapshots are stored at `MyLib/__snapshots__/FooSpec.snap.md`. Snapshots are stored in a Markdown file that's easy to visually inspect in an editor or on GitHub.

Values will be rendered via their internal heap representation; even if the type has a `Show` instance, it won't be used. To use the show instance, add the following to `Main.hs`:

```haskell
snapshotRenderers =
  [ renderWithShow @User
  ]
```

You can also specify a custom renderer by implementing a `SnapshotRenderer` yourself, probably using `plainRenderer`.

Currently, old snapshots are not cleaned up, so you'll have to manually clean up snapshots if you rename or remove a test. ([Issue #24](https://github.com/brandonchinn178/skeletest/issues/24))

### Property tests

Property tests are written with `prop` and run in the `PropertyM` monad (`Property` is an alias for `PropertyM ()`). To write property tests, add the following imports:

```haskell
import qualified Skeletest.Prop as Prop
import qualified Skeletest.Prop.Gen as Gen
import qualified Skeletest.Prop.Range as Range
```

Property tests consist of two things: generating random data with `forAll` and checking properties using the usual `shouldSatisfy` assertions. See the [Haddocks](https://hackage.haskell.org/package/skeletest/docs/Skeletest-Prop-Gen.html) for the different ways to generate data.

```haskell
prop "reverse does not change the length" $ do
  xs <- forAll $ Gen.list (Range.linear 0 10) $ Gen.int (Range.linear 1 100)
  length (reverse xs) `shouldBe` length xs
```

One common usecase is to verify that two functions are isomorphic. This can be tested with the `P.===` operator:

```haskell
prop "decodeUser . encodeUser === pure" $ do
  let genUser = User <$> Gen.text (Range.linear 0 10) Gen.unicode
  (decodeUser . encodeUser) P.=== pure `shouldSatisfy` P.isoWith genUser
```

If a test fails, it'll say something like `Rerun with --seed=6430645105429331403:9929029875326664391 to reproduce`. Rerunning with that flag will generate the same random value for debugging.

To ignore certain values, use `discard`:

```haskell
x <- Gen.int (Range.linear (-10) 10)
when (x == 0) discard
...
```

Property tests can also be configured with the following functions. These must be called at the very beginning of the test, before any `forAll` calls. Values specified with CLI flags take precedence over the values in the code.

* `Prop.setDiscardLimit`
    * The max number of values to discard before reporting a failure
    * Default: `100`

* `Prop.setShrinkLimit`
    * The max number of shrinks before giving up
    * Default: `1000`

* `Prop.setShrinkRetries`
    * The number of times to re-run a test during shrinking. This is useful if you are testing something which fails non-deterministically and you want to increase the change of getting a good shrink. e.g. `10` means a test must pass 10 times before trying a different shrink
    * Default: `0`

* `Prop.setConfidence`
    * The acceptable occurrence of false positives. e.g. `10^9` means accepting a false positive for 1 in 10^9 tests
    * Default: don't check confidence

* `Prop.setVerifiedTermination`
    * Validate confidence is reached
    * Default: disabled

* `Prop.setTestLimit`
    * The number of tests to run before reporting success
    * Default: `100`
    * CLI flag: `--prop-test-limit`

Internally, Skeletest uses Hedgehog to run property tests, but the API is intended to stay the same, even if the underlying engine changes.

### Fixtures

Fixtures are a useful way to reuse setup logic between tests. They're commonly used to initialize a database connection, set up users, etc. Fixtures can also use other fixtures. Fixtures are cached for the given scope and cleaned up when that scope is exited.

```haskell
data DbConnFixture = DbConnFixture Connection

instance Fixture DbConnFixture where
  -- defaults to per-test
  fixtureScope = PerSessionFixture

  fixtureAction = do
    conn <- initDBConn
    pure . withCleanup (DbConnFixture conn) $ do
      closeConn conn

spec :: Spec
spec = do
  it "creates a user" $ do
    DbConnFixture conn <- getFixture
    createUser conn "alice" `shouldSatisfy` P.not (P.throws P.anything)

  it "fetches a user" $ do
    -- reuses the same connection initialized in the first test
    DbConnFixture conn <- getFixture
    getUser conn "alice" `shouldSatisfy` P.just (P.con User{name = P.eq "alice"})
```

#### Built-in fixtures

The following fixtures are available out of the box:

* `FixtureTmpDir` - Contains a temporary directory that is cleaned up between tests.

    ```haskell
    FixtureTmpDir tmpDir <- getFixture
    writeFile (tmpDir </> "myfile.txt") "test"
    ````

### Markers

Markers are a useful way to mark tests for selection (see "Test selection"). There are two ways to mark a test:

1. With anonymous markers:

    ```haskell
    withMarkers ["foo", "bar"] $ do
      ...
    ```

    All tests in the given section will be marked with anonymous markers named "foo" and "bar", which can be selected with `@foo` and `@bar`, respectively.

1. With typed markers:

    ```haskell
    data MyMarker = MyMarker Int
    instance IsMarker MyMarker where
      getMarkerName _ = "my-marker"

    withMarker (MyMarker 10) $ do
      ...
    ```

    All tests in the given section will be marked with the given marker, which can be selected with `@my-marker`. You can see if a test has a marker with `findMarkers` (see the "Hooks" section).

### Output capturing

By default, Skeletest will capture all output to stdout/stderr that occurs during a test and will only display it if the test fails. To disable this, pass `--capture-output=off`.

### Custom CLI flags

To register and use your own CLI flags, do the following:

1. Create an instance of `IsFlag`

1. In `Main.hs`, add the following:

    ```haskell
    import TestUtils.Flags (MyFlag)

    cliFlags =
      [ flag @MyFlag
      ]
    ```

1. In a fixture or test, do the following:

    ```haskell
    MyFlag flagVal <- getFlag
    ```

### Hooks

Skeletest lets you hook into specific parts of test execution. Skeletest currently supports the following hooks:

* `hookRunTest` - Modify how/if a test is run. Takes the `TestInfo` of the currently running test. `TestInfo` contains `testInfoMarkers`, which you can query with `findMarker` or `hasMarkerNamed`.

### Plugins

Skeletest is fully pluggable; any configuration specified in `Main.hs` (e.g. `cliFlags` or `snapshotRenderer`) can be defined in a `Plugin` that you can import from another module or even another package.

```haskell
module TestUtils.Plugins (myPlugin) where

import Skeletest.Plugin

myPlugin :: Plugin
myPlugin =
  defaultPlugin
    { hooks =
        defaultHooks
          { hookRunTest = \testInfo runTest -> do
              putStrLn "before test"
              result <- runTest
              putStrLn "after test"
              pure result
          }
    }
```

```haskell
import TestUtils.Plugins (myPlugin)

plugins =
  [ myPlugin
  ]
```