.github/workflows | ||
app | ||
scripts | ||
src | ||
test | ||
.gitignore | ||
ChangeLog.md | ||
LICENSE | ||
Makefile | ||
package.yaml | ||
README.md | ||
roboservant.cabal | ||
Setup.hs | ||
stack.yaml | ||
TODO.md |
roboservant
Automatically fuzz your servant apis in a contextually-aware way.
why?
Servant gives us a lot of information about what a server can do. We use this information to generate arbitrarily long request/response sessions and verify properties that should hold over them.
example
Our api under test:
newtype Foo = Foo Int
deriving (Generic, Eq, Show, Typeable)
deriving newtype (FromHttpApiData, ToHttpApiData)
type FooApi =
"item" :> Get '[JSON] Foo
:<|> "itemAdd" :> Capture "one" Foo :> Capture "two" Foo :> Get '[JSON] Foo
:<|> "item" :> Capture "itemId" Foo :> Get '[JSON] ()
From the tests:
assert "should find an error in Foo" . not
=<< checkSequential (Group "Foo" [("Foo", RS.prop_sequential @Foo.FooApi Foo.fooServer)])
We have a server that blows up if the value of the int in a Foo
ever gets above 10. Note:
there is no generator for Foo
types: larger Foo
s can only be made only by combining existing
Foo
s with itemAdd
. This is an important distinction, because many APIs will return UUIDs or
similar as keys, which make it impossible to cover a useful section of the state space without
using the values returned by the API
why not servant-quickcheck?
servant-quickcheck is a great package and I've learned a lot from it. Unfortunately, as mentioned previously, there's a lot of the state space you just can't explore without context: modern webapps are full of pointer-like structures, whether they're URLs or database keys/uuids, and servant-quickcheck requires that you be able to generate these without context via Arbitrary.
extensions/todo
- add some "starter" values to the store
- there may be a JWT that's established outside the servant app, for instance.
class Extras a where extras :: Gen [a]
- default implementation
pure []
- selectively allow some types to create values we haven't seen from the api.
newtype FirstName = FirstName Text
, say.
- default implementation
- break down each response type into its components
- if i have
data Foo = FBar Bar | FBaz Baz
- an endpoint
foo
that returns aFoo
- and an endpoint
bar
that takes aBar
- I should be able to call
foo
to get aFoo
, and if it happens to be anFBar Bar
, I should be able to use thatBar
to callbar
.
- if i have
- better handling of properties to be verified
- some properties should always hold (no 500s): this already works.
- to-do: there may be some other properties that hold contextually
- healthcheck should be 200
- test complex permissions/ownership/delegation logic - should never be able to get access to something you don't own or haven't been delegated access to.
other possible applications
-
coverage
- if you run the checker for a while and
hpc
suggests you still have bad coverage, your api is designed in a way that requires external manipulation and may be improvable.
- if you run the checker for a while and
-
benchmarking
- we can generate "big-enough" call sequences, then save the database & a sample call for each endpoint that takes long enough to be a reasonable test.
- from this we can generate tests that a given call on that setup never gets slower.