@@ -4,40 +4,54 @@ Automatically fuzz your servant apis in a contextually-aware way.
44
55
66
7+
78# why?
89
910Servant gives us a lot of information about what a server can do. We
1011use this information to generate arbitrarily long request/response
1112sessions and verify properties that should hold over them.
1213
13−# why not servant-quickcheck?
14+# example
1415
15−[servant-quickcheck](https://hackage.haskell.org/package/servant-quickcheck)
16−is a great package and I've learned a lot from it. Unfortunately,
17−there's a lot of the state space it can't explore: modern webapps are
18−full of pointer-like structures, whether they're URLs or database
19−keys/uuids. servant-quickcheck demands that you be able to generate
20−these without context via Arbitrary: good luck exploring an API that
21−requires you to generate just the right UUID to hit non-trivial code.
16+Our api under test:
2217
23−roboservant avoids this by using
24−[quickcheck-state-machine](https://github.com/advancedtelematic/quickcheck-state-machine#readme),
25−which models the dynamic state in such a way that we can use results
26−of previous calls.
18+```
19+newtype Foo = Foo Int
20+ deriving (Generic, Eq, Show, Typeable)
21+ deriving newtype (FromHttpApiData, ToHttpApiData)
2722
28−# concept
23+type FooApi =
24+ "item" :> Get '[JSON] Foo
25+ :<|> "itemAdd" :> Capture "one" Foo :> Capture "two" Foo :> Get '[JSON] Foo
26+ :<|> "item" :> Capture "itemId" Foo :> Get '[JSON] ()
27+```
2928
30−we start with a servant api and a server that fulfills the type.
29+From the tests:
3130
32−From that api, we should be able to summon up an empty type-indexed store with a key
33−for each response type in the API.
31+```
32+ let reifiedApi = RS.toReifiedApi (RS.flattenServer @Foo.FooApi Foo.fooServer) (Proxy @(Endpoints Foo.FooApi))
33+ assert "should find an error in Foo" . not
34+ =<< checkSequential (Group "Foo" [("Foo", RS.prop_sequential reifiedApi)])
35+```
3436
35−We can then look at each callable endpoint, and eliminate any that require values that are
36−empty in the type-indexed store. this allows a generative process where we can extend a sequence
37−of calls indefinitely, by making a call to the concrete server and recording the result in the
38−type-indexed store.
37+We have a server that blows up if the value of the int in a `Foo` ever gets above 10. Note:
38+there is no generator for `Foo` types: larger `Foo`s can only be made only by combining existing
39+`Foo`s with `itemAdd`. This is an important distinction, because many APIs will return UUIDs or
40+similar as keys, which make it impossible to cover a useful section of the state space without
41+using the values returned by the API
42+
43+
44+# why not servant-quickcheck?
45+
46+[servant-quickcheck](https://hackage.haskell.org/package/servant-quickcheck)
47+is a great package and I've learned a lot from it. Unfortunately, as mentioned previously,
48+there's a lot of the state space you just can't explore without context: modern webapps are
49+full of pointer-like structures, whether they're URLs or database
50+keys/uuids, and servant-quickcheck requires that you be able to generate
51+these without context via Arbitrary.
52+
53+## extensions/todo
3954
40−## extensions
4155- add some "starter" values to the store
4256 - there may be a JWT that's established outside the servant app, for instance.
4357- `class Extras a where extras :: Gen [a]`
@@ -51,15 +65,15 @@ type-indexed store.
5165 - and an endpoint `bar` that takes a `Bar`
5266 - I should be able to call `foo` to get a `Foo`, and if it happens to be an `FBar Bar`, I
5367 should be able to use that `Bar` to call `bar`.
54−
55−## applications
56−- testing
57− - some properties should always hold (no 500s)
58− - there may be some other properties that hold contextually
68+- better handling of properties to be verified
69+ - some properties should always hold (no 500s): this already works.
70+ - to-do: there may be some other properties that hold contextually
5971 - healthcheck should be 200
6072 - test complex permissions/ownership/delegation logic - should never be able to
6173 get access to something you don't own or haven't been delegated access to.
6274
75+## other possible applications
76+
6377- coverage
6478 - if you run the checker for a while and `hpc` suggests you still have bad coverage,
6579 your api is designed in a way that requires external manipulation and may be improvable.