@@ -10,13 +10,62 @@ an API description.
1010
1111This is much closer to the traditional use of `QuickCheck`. The most obvious
1212use-case is checking that properties hold of an *entire* server rather than of
13−individual endpoints.
13+individual endpoints. (But there are other uses that you can skip to if they
14+sound more interesting.)
1415
1516## `serverSatisfies`
1617
17−There are a variety of best practices in writing web APIs that aren't always
18−obvious. As a running example, let's use a simple service that allows adding,
19−removing, and querying biological species. Our SQL schema is:
18+A useful guideline when writing and maintaing software is that, if there isn't
19+a test for a behaviour or property, sooner or later that property will be broken.
20+Another important perspective is that tests are a form of documentation - the
21+present developer telling future ones "this matters, and should be this way".
22+
23+The advantage of using tests for this form of documentation is that there's
24+simply too much information to convey, some of it only relevant to very specific
25+use cases, and rather than overload developers with an inexhaustible quantity of
26+details that would be hard to keep track of or remember, tests are a mechanism
27+of reminding developers of *only the relevant information, at the right time*.
28+<<EXAMPLE>>.
29+
30+We might hope that we could use tests to communicate the wide array of best
31+practices that have developed around APIs. About to return a top-level integer
32+in JSON? A test should say that's bad practice. About to not catch exceptions
33+and give a more meaningful HTTP status code? Another test there to stop you.
34+
35+Traditionally, in web services these things get done at the level of *individual*
36+endpoints. But this means that if a developer who hasn't had extensive experience with web
37+programming best practices writes a *new* endpoint which *does* return a top-level
38+integer literal, there's no test there to stop her. Code review might help, but
39+code review is much more error prone than tests, and really only meant for those
40+things that are too subtle to automate. (Indeed, if code review were such a reliable
41+defense mechanism against bugs and bad code, why have tests and linters at all?)
42+
43+The problem, then, with thinking about tests as only existing at the level of individual
44+endpoints is that there are no tests *for* tests - tests that check that new
45+behaviour and tests conforms to higher-level, more general best practices.
46+
47+`servant-quickcheck` aims to solve that. It allows describing properties that
48+*all* endpoints myst satisfy. If a new endpoint comes along, it too will be
49+tested for that property, without any further work.
50+
51+Why isn't this idea already popular? Well, most web frameworks don't have a
52+reified description of APIs. When you don't know what the endpoints of an
53+application are, and what request body they expect, trying to generate arbitrary
54+requests is almost entirely going to result in 404s (not found) and 400s (bad
55+request). Maybe one in a thousand requests will actually test a handler. Not
56+very useful.
57+
58+`servant` applications, on the other hand, have a machine-readable API description
59+already available. And they already associate "correct" requests with particular
60+types. It's a small step, therefore, to generate 'arbitrary' values for these
61+requests, and all of them will go through to your handlers. (Note: all of the
62+uses of `servant-quickcheck` work with applications *not* written with servant-server -
63+and indeed not *in Haskell - but the API must be described with the servant
64+DSL.)
65+
66+Let's see how this works in practice. As a running example, let's use a simple
67+service that allows adding, removing, and querying biological species. Our SQL
68+schema is:
2069
2170:d schema.sql
2271
@@ -150,6 +199,60 @@ instance Arbitrary Species where
150199 arbitrary = Species <$> arbitrary <*> arbitrary
151200:
152201
202+But this fails in quite a few ways.
203+
204+### Why best practices are good
205+
206+As a side note: you might have wondered "why bother with API best practices?".
207+It is, it would be said, a lot of extra (as in not only getting the feature done)
208+work to do, for dubious benefit. And indeed, the relevance of discoverability, for
209+example, unclear, since not that many tools use it.
210+
211+But `servant-quickcheck` both makes it *easier* to conform to best practices,
212+and exemplifies their advantage. If we pick 201 (Success, the 'resource' was
213+created), rather than the more generic 200 (Success), `servant-quickcheck` knows
214+this means there should be some representation of the rec°as a response
215+
216+
217+## `serversEqual`
218+
219+There's another very appealing application of the ability to generate "sensible"
220+arbitrary requests. It's testing that two applications are equal. Generate arbitrary
221+requests, send them to both servers (in the same order), and check that the responses
222+are equivalent. (This was, in fact, one of the first applications of
223+`servant-client`, albeit in a much more manual way, when we rewrote a microservice
224+originally in Python in Haskell.) Generally with rewrites, even if there's some
225+behaviour that isn't optimal, if a lot of things already depend on that service,
226+it makes sense to first mimick *exactly* the original behaviour, and only then
227+aim for improvements.
228+
229+`servant-quickcheck` provides a single function, `serversEqual`, that attempts
230+to verify the equivalence of servers. Since some aspects of responses might not
231+be relevant (for example, whether the the `Server` header is the same, or whether
232+two JSON responses have the same formatting), it allows you to provide a custom
233+equivalence function. Other than that, you need only provide an API type and two
234+URLs for testing, and the rest `serversEqual` handles.
235+
236+## Future directions: benchmarking
237+
238+What else could benefit from tooling that can automatically generate sensible
239+(*vis-a-vis* a particular application's expectations) requests?
240+
241+One area is extensive automatic benchmarking. Currently we use tools such as
242+`ab`, `wrk`, `httperf` in a very manual way - we pick a particular request that
243+we are interested in, and write a request that gets made thousands of times.
244+But now we can have a multiplicity of requests to benchmark with! This allows
245+*finding* slow endpoints, as well as (I would imagine, though I haven't actually
246+tried this yet) synchronization issues that make threads wait for too long (such
247+as waiting on an MVar that's not really needed), bad asymptotics with respect
248+to some other type of request.
249+
250+(On this last point, imagine not having an index in a database for "people",
251+ and having a tool that discovers that the latency on a search by first name
252+ grows linearly with the number of POST requests to a *different* endpoint! We'd
253+ need to do some to do this well, possibly involving some machine learning, but
254+ it's an interesting and probably useful idea.)
255+
153256
154257**Note**: This post is an anansi literate file that generates multiple source
155258files. They are: