roboservant: Automatic session-aware servant testing

[ bsd3, library, web ] [ Propose Tags ]

Please see the README on GitHub at https://github.com/mwotton/roboservant#readme


[Skip to Readme]

Downloads

Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees

Candidates

Versions [RSS] 0.1.0.0, 0.1.0.1, 0.1.0.2, 0.1.0.3
Change log ChangeLog.md
Dependencies aeson, base (>=4.7 && <5), bytestring, containers, hedgehog, hspec, http-client, http-media, mtl, QuickCheck, quickcheck-state-machine (>=0.7), servant (>=0.17), servant-client (>=0.17), servant-flatten, servant-server (>=0.17), string-conversions [details]
License BSD-3-Clause
Copyright 2020 Mark Wotton, Samuel Schlesinger
Author Mark Wotton, Samuel Schlesinger
Maintainer mwotton@gmail.com
Category Web
Home page https://github.com/mwotton/roboservant#readme
Bug tracker https://github.com/mwotton/roboservant/issues
Source repo head: git clone https://github.com/mwotton/roboservant
Uploaded by MarkWotton at 2020-09-14T00:41:18Z
Distributions NixOS:0.1.0.3
Downloads 492 total (17 in the last 30 days)
Rating 2.0 (votes: 1) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs available [build log]
Last success reported on 2020-09-14 [all 1 reports]

Readme for roboservant-0.1.0.1

[back to package description]

roboservant

Automatically fuzz your servant apis in a contextually-aware way.

CI

why?

Servant gives us a lot of information about what a server can do. We use this information to generate arbitrarily long request/response sessions and verify properties that should hold over them.

example

Our api under test:

newtype Foo = Foo Int
  deriving (Generic, Eq, Show, Typeable)
  deriving newtype (FromHttpApiData, ToHttpApiData)

type FooApi =
  "item" :> Get '[JSON] Foo
    :<|> "itemAdd" :> Capture "one" Foo :> Capture "two" Foo :> Get '[JSON] Foo
    :<|> "item" :> Capture "itemId" Foo :> Get '[JSON] ()

From the tests:

  assert "should find an error in Foo" . not
    =<< checkSequential (Group "Foo" [("Foo", RS.prop_sequential @Foo.FooApi Foo.fooServer)])

We have a server that blows up if the value of the int in a Foo ever gets above 10. Note: there is no generator for Foo types: larger Foos can only be made only by combining existing Foos with itemAdd. This is an important distinction, because many APIs will return UUIDs or similar as keys, which make it impossible to cover a useful section of the state space without using the values returned by the API

why not servant-quickcheck?

servant-quickcheck is a great package and I've learned a lot from it. Unfortunately, as mentioned previously, there's a lot of the state space you just can't explore without context: modern webapps are full of pointer-like structures, whether they're URLs or database keys/uuids, and servant-quickcheck requires that you be able to generate these without context via Arbitrary.

extensions/todo

  • add some "starter" values to the store
    • there may be a JWT that's established outside the servant app, for instance.
  • class Extras a where extras :: Gen [a]
    • default implementation pure []
    • selectively allow some types to create values we haven't seen from the api. newtype FirstName = FirstName Text, say.
  • break down each response type into its components
    • if i have
      • data Foo = FBar Bar | FBaz Baz
      • an endpoint foo that returns a Foo
      • and an endpoint bar that takes a Bar
    • I should be able to call foo to get a Foo, and if it happens to be an FBar Bar, I should be able to use that Bar to call bar.
  • better handling of properties to be verified
    • some properties should always hold (no 500s): this already works.
    • to-do: there may be some other properties that hold contextually
      • healthcheck should be 200
      • test complex permissions/ownership/delegation logic - should never be able to get access to something you don't own or haven't been delegated access to.

other possible applications

  • coverage

    • if you run the checker for a while and hpc suggests you still have bad coverage, your api is designed in a way that requires external manipulation and may be improvable.
  • benchmarking

    • we can generate "big-enough" call sequences, then save the database & a sample call for each endpoint that takes long enough to be a reasonable test.
    • from this we can generate tests that a given call on that setup never gets slower.