@@ -45,15 +45,15 @@ endpoints is that there are no tests *for* tests - tests that check that new
4545behaviour and tests conforms to higher-level, more general best practices.
4646
4747`servant-quickcheck` aims to solve that. It allows describing properties that
48−*all* endpoints myst satisfy. If a new endpoint comes along, it too will be
48+*all* endpoints must satisfy. If a new endpoint comes along, it too will be
4949tested for that property, without any further work.
5050
5151Why isn't this idea already popular? Well, most web frameworks don't have a
52−reified description of APIs. When you don't know what the endpoints of an
53−application are, and what request body they expect, trying to generate arbitrary
54−requests is almost entirely going to result in 404s (not found) and 400s (bad
55−request). Maybe one in a thousand requests will actually test a handler. Not
56−very useful.
52+reified description of APIs (beyond perhaps the routes). When you don't know
53+what the endpoints of an application are, and what request body they expect,
54+trying to generate arbitrary requests is almost entirely going to result in
55+404s (not found) and 400s (bad request). Maybe one in a thousand requests will
56+actually test a handler. Not very useful.
5757
5858`servant` applications, on the other hand, have a machine-readable API description
5959already available. And they already associate "correct" requests with particular
@@ -178,11 +178,14 @@ import Test.QuickCheck (Arbitrary(..))
178178import Database.PostgreSQL.Simple (connectPostgreSQL)
179179
180180spec :: Spec
181−spec = describe "the species application" $ do
181+spec = describe "the species application" $ beforeAll check $ do
182182 let pserver = do
183183 conn <- connectPostgreSQL "dbname=servant-quickcheck"
184184 return $ server conn
185185
186+
187+ it "should not return 500s" $ do
188+
186189 it "should not return 500s" $ do
187190 withServantServer api pserver $ \url ->
188191 serverSatisfies api url defaultArgs (not500 <%> mempty)
@@ -191,6 +194,12 @@ spec = describe "the species application" $ do
191194 withServantServer api pserver $ \url ->
192195 serverSatisfies api url defaultArgs (onlyJsonObjects <%> mempty)
193196
197+ where
198+ check = do
199+ mvar <- newMVar []
200+ withServantServer api pserver $ \url ->
201+ serverSatisfies api url defaultArgs (onlyJsonObjects <%> mempty)
202+
194203main :: IO ()
195204main = do
196205 hspec spec
@@ -201,30 +210,51 @@ instance Arbitrary Species where
201210
202211But this fails in quite a few ways.
203212
213+
214+<<TODO>>
215+
216+This was an example created with the knowledge of what it was supposed to
217+exemplify. To try to get a more accurate assessment of the practical usefulness
218+of `servant-quickcheck`, I tried running `serverSatisfies` with a few
219+predicates over some of the open-source `servant` servers I could find, and
220+results were also promising.
221+
222+There are probably a lot of other interesting properties that one might to add
223+besides those I've included. As an example, we could have a property that
224+all HTML is checked against, which is sometimes tricky for HTML that's
225+generated dynamically. Or check that every page has a Portuguese translation.
226+
204227### Why best practices are good
205228
206229As a side note: you might have wondered "why bother with API best practices?".
207−It is, it would be said, a lot of extra (as in not only getting the feature done)
230+It is, it has to be said, a lot of extra (as in not only getting the feature done)
208231work to do, for dubious benefit. And indeed, the relevance of discoverability, for
209−example, unclear, since not that many tools use it.
232+example, unclear, since not that many tools use it as perhaps was anticipated.
210233
211234But `servant-quickcheck` both makes it *easier* to conform to best practices,
212−and exemplifies their advantage. If we pick 201 (Success, the 'resource' was
213−created), rather than the more generic 200 (Success), `servant-quickcheck` knows
214−this means there should be some representation of the rec°as a response
235+and exemplifies their advantage in enabling better tooling. If we pick 201 (Success, the 'resource' was
236+created), rather than the more generic 200 (Success), and do a *little* more work
237+by knowing to make this decision, `servant-quickcheck` knows this means there
238+should be some representation of the resource created. So it knows to ask you
239+for a link to it (the RFC creators thought to ask for this). And if you do (again,
240+a little more work), `servant-quickcheck` will know to try to look at that
241+resource by following the link, checking that it's not broken, and maybe even
242+returns a response that equivalent to the original POST request). And then it
243+finds a real bug - your application allows species with '/' in their name to
244+be created, but not queried with a 'GET' for! This, I think, is already a win.
215245
216246
217247## `serversEqual`
218248
219249There's another very appealing application of the ability to generate "sensible"
220−arbitrary requests. It's testing that two applications are equal. Generate arbitrary
250+arbitrary requests. It's for testing that two applications are equal. We can generate arbitrary
221251requests, send them to both servers (in the same order), and check that the responses
222−are equivalent. (This was, in fact, one of the first applications of
252+are equivalent. (This was, incidentally, one of the first applications of
223253`servant-client`, albeit in a much more manual way, when we rewrote a microservice
224254originally in Python in Haskell.) Generally with rewrites, even if there's some
225−behaviour that isn't optimal, if a lot of things already depend on that service,
226−it makes sense to first mimick *exactly* the original behaviour, and only then
227−aim for improvements.
255+behaviour that isn't optimal, perhaps a lot of things already depend on that service
256+and make interace poorly with "improvements", so it makes sense to first mimick
257+*exactly* the original behaviour, and only then aim for improvements.
228258
229259`servant-quickcheck` provides a single function, `serversEqual`, that attempts
230260to verify the equivalence of servers. Since some aspects of responses might not
@@ -243,16 +273,23 @@ One area is extensive automatic benchmarking. Currently we use tools such as
243273we are interested in, and write a request that gets made thousands of times.
244274But now we can have a multiplicity of requests to benchmark with! This allows
245275*finding* slow endpoints, as well as (I would imagine, though I haven't actually
246−tried this yet) synchronization issues that make threads wait for too long (such
247−as waiting on an MVar that's not really needed), bad asymptotics with respect
248−to some other type of request.
276+tried this yet) finding synchronization issues that make threads wait for too
277+long (such as waiting on an MVar that's not really needed), bad asymptotics
278+with respect to some other type of request.
249279
250280(On this last point, imagine not having an index in a database for "people",
251281 and having a tool that discovers that the latency on a search by first name
252282 grows linearly with the number of POST requests to a *different* endpoint! We'd
253− need to do some to do this well, possibly involving some machine learning, but
254− it's an interesting and probably useful idea.)
283+ need to do some work to do this well, possibly involving some machine
284+ learning, but it's an interesting and probably useful idea.)
285+
286+
287+# Conclusion
288+
289+I hope this library presents some useful functionality already, but I hope
290+you'll also think how it could be improved!
255291
292+There'll be a few more packages in the comings weeks - check back soon!
256293
257294**Note**: This post is an anansi literate file that generates multiple source
258295files. They are: