goldplate: A lightweight golden test runner

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.

[maintain] [Publish]

Warnings:


[Skip to Readme]

Properties

Versions 0.1.0, 0.1.1, 0.1.2, 0.1.2, 0.1.3, 0.2.0
Change log CHANGELOG.md
Dependencies aeson (==1.4.*), aeson-pretty (==0.8.*), async (==2.2.*), base (>=4.9 && <5), bytestring (==0.10.*), Diff (>=0.3 && <0.5), directory (==1.3.*), filepath (==1.4.*), Glob (==0.10.*), optparse-applicative (>=0.14 && <0.16), process (==1.6.*), regex-pcre-builtin (>=0.95.1.3 && <0.96), text (==1.2.*), time (>=1.8 && <1.10), unordered-containers (==0.2.*) [details]
License Apache-2.0
Copyright 2019-2020 Fugue, Inc
Author Jasper Van der Jeugt <jasper@fugue.co>
Maintainer Jasper Van der Jeugt <jasper@fugue.co>
Category Language
Uploaded by JasperVanDerJeugt at 2020-11-05T14:33:04Z

Downloads

Maintainer's Corner

For package maintainers and hackage trustees


Readme for goldplate-0.1.2

[back to package description]

goldplate 🏅

$ goldplate -j2 tests/
Found 32 specs
Running 49 executions in 2 jobs
Finished in 0.84s
Ran 32 specs, 49 executions, 146 asserts, all A-OK!

goldplate is a cute and simple opaque golden test runner for CLI applications. You place your test cases in a directory, annotate them with .goldplate files, and that's it. It is completely language agnostic. And perhaps its best feature is that it can automatically✨🪄 fix your tests outputs!

Give goldplate a try if:

At Fugue, we've been using internal versions of this tool since 2016, so it should be pretty stable.

Table of Contents

Tutorial

Creating a first test

Imagine we are up to testing the behaviour of echo command. In this very simple example, we run echo "Hello, world!" and expect it to print Hello, world! to the stdout stream as a result.

Create a new file echo.goldplate and add the following content:

{
    "command": "echo",
    "arguments": ["Hello, world!"],
    "asserts": [
        {"exit_code": 0},
        {"stdout": "hello-world.txt"}
    ]
}

Let's go through this bit by bit.

The test invocation is specified by the command and arguments fields. We are invoking the echo process with a single argument, "Hello, world!".

The expected results of our test live in the asserts field. This simple test has two asserts:

  1. We verify that the exit code is 0 (success).
  2. We check the stdout (output) of the command against the contents of the file hello-world.txt.

We haven't created hello-world.txt yet, but that's not a problem. We can invoke goldplate --fix to create it:

$ goldplate echo.goldplate --pretty-diff --fix
...
echo.goldplate: stdout: does not match
echo.goldplate: fixed ./hello-world.txt
...
Ran 1 specs, 1 executions, 2 asserts, 1 failed.

After hello-world.txt has been created with proper contents, subsequent goldplate invocations will pass:

$ goldplate echo.goldplate
...
Ran 1 specs, 1 executions, 2 asserts, all A-OK!

You can view the full example here:

Feeding input on stdin

View example:

You can pass one or multiple lines of input to the command by using the stdin field.

Setting environment variables

View example:

The environment field can be used to set environment variables for the program.

goldplate also sets a number of environment variables. You can use these directly within the configuration JSON. In this example, we use:

{"stdout": "${GOLDPLATE_NAME}.stdout"}

Rather than:

{"stdout": "env.stdout"}

We found this to be good practice, it makes mass-renaming of tests much easier.

Globbing input files

View example:

.goldplate files are fairly small but if you have a whole directory of files that you just want to run the same command on, they can get very repetitive. This is why goldplate provides a simple way to pull in multiple input files.

If the input_files field is set to a glob, all asserts will be ran for every matching input file. goldplate will set the following variables:

Post processing: find and replace

View example:

Sometimes you may want to do a find-and-replace on the actual output, for example to filter out timestamps or other information that you do not expect to match up against the expected output.

Post processing: prettify JSON

View example:

Many modern CLI tools output JSON. You can use the prettify_json post processor to make sure the JSON is converted to a normalized form with sorted keys and consistent indentation.

Created files and directories

View example:

goldplate is geared towards checking the stdout and stderr outputs of a program, but you can also check that files were created with specific contents. If you do this, goldplate will remove these files and directories afterwards to leave a clean repository behind.

Installation

Installation through source is done using standard Haskell tooling -- Cabal and stack both work well.

Using stack

  1. Install stack for your platform.
  2. Clone this repository and cd into it.
  3. Run stack install.
  4. Make sure $HOME/.local/bin is in your $PATH.

Using Cabal

  1. Install Cabal for your platform.
  2. Clone this repository and cd into it.
  3. Run cabal install.
  4. Make sure $HOME/.cabal/bin is in your $PATH.

Reference

Syntax

Environment variables can be spliced into the configuration using ${VAR} syntax within strings. To escape this syntax, use $${VAR} to get a literal ${VAR}, $$${VAR} to get a literal $${VAR}, and so on.

Environment variables

The test is always executed in the directory that holds the .goldplate file. goldplate will always set the following environment variables:

When dealing with multiple input files, the following additional variables are set:

Here is an example that outputs all of these environment variables:

Similar projects

A similar project is smoke. I think goldplate has two major advantages over smoke: