@@ -45,15 +45,15 @@ endpoints is that there are no tests *for* tests - tests that check that new
45behaviour and tests conforms to higher-level, more general best practices.
45behaviour and tests conforms to higher-level, more general best practices.
47`servant-quickcheck` aims to solve that. It allows describing properties that
47`servant-quickcheck` aims to solve that. It allows describing properties that
48−*all* endpoints myst satisfy. If a new endpoint comes along, it too will be
48+*all* endpoints must satisfy. If a new endpoint comes along, it too will be
49tested for that property, without any further work.
49tested for that property, without any further work.
51Why isn't this idea already popular? Well, most web frameworks don't have a
51Why isn't this idea already popular? Well, most web frameworks don't have a
52−reified description of APIs. When you don't know what the endpoints of an
52+reified description of APIs (beyond perhaps the routes). When you don't know
53−application are, and what request body they expect, trying to generate arbitrary
53+what the endpoints of an application are, and what request body they expect,
54−requests is almost entirely going to result in 404s (not found) and 400s (bad
54+trying to generate arbitrary requests is almost entirely going to result in
55−request). Maybe one in a thousand requests will actually test a handler. Not
55+404s (not found) and 400s (bad request). Maybe one in a thousand requests will
56−very useful.
56+actually test a handler. Not very useful.
58`servant` applications, on the other hand, have a machine-readable API description
58`servant` applications, on the other hand, have a machine-readable API description
59already available. And they already associate "correct" requests with particular
59already available. And they already associate "correct" requests with particular
@@ -178,11 +178,14 @@ import Test.QuickCheck (Arbitrary(..))
178import Database.PostgreSQL.Simple (connectPostgreSQL)
178import Database.PostgreSQL.Simple (connectPostgreSQL)
180spec :: Spec
180spec :: Spec
181−spec = describe "the species application" $ do
181+spec = describe "the species application" $ beforeAll check $ do
182 let pserver = do
182 let pserver = do
183 conn <- connectPostgreSQL "dbname=servant-quickcheck"
183 conn <- connectPostgreSQL "dbname=servant-quickcheck"
184 return $ server conn
184 return $ server conn
187+ it "should not return 500s" $ do
186 it "should not return 500s" $ do
189 it "should not return 500s" $ do
187 withServantServer api pserver $ \url ->
190 withServantServer api pserver $ \url ->
188 serverSatisfies api url defaultArgs (not500 <%> mempty)
191 serverSatisfies api url defaultArgs (not500 <%> mempty)
@@ -191,6 +194,12 @@ spec = describe "the species application" $ do
191 withServantServer api pserver $ \url ->
194 withServantServer api pserver $ \url ->
192 serverSatisfies api url defaultArgs (onlyJsonObjects <%> mempty)
195 serverSatisfies api url defaultArgs (onlyJsonObjects <%> mempty)
200+ withServantServer api pserver $ \url ->
201+ serverSatisfies api url defaultArgs (onlyJsonObjects <%> mempty)
194main :: IO ()
203main :: IO ()
196 hspec spec
205 hspec spec
@@ -201,30 +210,51 @@ instance Arbitrary Species where
202But this fails in quite a few ways.
211But this fails in quite a few ways.
216+This was an example created with the knowledge of what it was supposed to
217+exemplify. To try to get a more accurate assessment of the practical usefulness
218+of `servant-quickcheck`, I tried running `serverSatisfies` with a few
219+predicates over some of the open-source `servant` servers I could find, and
220+results were also promising.
222+There are probably a lot of other interesting properties that one might to add
223+besides those I've included. As an example, we could have a property that
224+all HTML is checked against, which is sometimes tricky for HTML that's
225+generated dynamically. Or check that every page has a Portuguese translation.
204### Why best practices are good
227### Why best practices are good
206As a side note: you might have wondered "why bother with API best practices?".
229As a side note: you might have wondered "why bother with API best practices?".
207−It is, it would be said, a lot of extra (as in not only getting the feature done)
230+It is, it has to be said, a lot of extra (as in not only getting the feature done)
208work to do, for dubious benefit. And indeed, the relevance of discoverability, for
231work to do, for dubious benefit. And indeed, the relevance of discoverability, for
209−example, unclear, since not that many tools use it.
232+example, unclear, since not that many tools use it as perhaps was anticipated.
211But `servant-quickcheck` both makes it *easier* to conform to best practices,
234But `servant-quickcheck` both makes it *easier* to conform to best practices,
212−and exemplifies their advantage. If we pick 201 (Success, the 'resource' was
235+and exemplifies their advantage in enabling better tooling. If we pick 201 (Success, the 'resource' was
213−created), rather than the more generic 200 (Success), `servant-quickcheck` knows
236+created), rather than the more generic 200 (Success), and do a *little* more work
237+by knowing to make this decision, `servant-quickcheck` knows this means there
214−this means there should be some representation of the rec°as a response
238+should be some representation of the resource created. So it knows to ask you
239+for a link to it (the RFC creators thought to ask for this). And if you do (again,
240+a little more work), `servant-quickcheck` will know to try to look at that
241+resource by following the link, checking that it's not broken, and maybe even
242+returns a response that equivalent to the original POST request). And then it
243+finds a real bug - your application allows species with '/' in their name to
244+be created, but not queried with a 'GET' for! This, I think, is already a win.
217## `serversEqual`
247## `serversEqual`
219There's another very appealing application of the ability to generate "sensible"
249There's another very appealing application of the ability to generate "sensible"
220−arbitrary requests. It's testing that two applications are equal. Generate arbitrary
250+arbitrary requests. It's for testing that two applications are equal. We can generate arbitrary
221requests, send them to both servers (in the same order), and check that the responses
251requests, send them to both servers (in the same order), and check that the responses
222−are equivalent. (This was, in fact, one of the first applications of
252+are equivalent. (This was, incidentally, one of the first applications of
223`servant-client`, albeit in a much more manual way, when we rewrote a microservice
253`servant-client`, albeit in a much more manual way, when we rewrote a microservice
224originally in Python in Haskell.) Generally with rewrites, even if there's some
254originally in Python in Haskell.) Generally with rewrites, even if there's some
225−behaviour that isn't optimal, if a lot of things already depend on that service,
255+behaviour that isn't optimal, perhaps a lot of things already depend on that service
256+and make interace poorly with "improvements", so it makes sense to first mimick
226−it makes sense to first mimick *exactly* the original behaviour, and only then
257+*exactly* the original behaviour, and only then aim for improvements.
227−aim for improvements.
229`servant-quickcheck` provides a single function, `serversEqual`, that attempts
259`servant-quickcheck` provides a single function, `serversEqual`, that attempts
230to verify the equivalence of servers. Since some aspects of responses might not
260to verify the equivalence of servers. Since some aspects of responses might not
@@ -243,16 +273,23 @@ One area is extensive automatic benchmarking. Currently we use tools such as
243we are interested in, and write a request that gets made thousands of times.
273we are interested in, and write a request that gets made thousands of times.
244But now we can have a multiplicity of requests to benchmark with! This allows
274But now we can have a multiplicity of requests to benchmark with! This allows
245*finding* slow endpoints, as well as (I would imagine, though I haven't actually
275*finding* slow endpoints, as well as (I would imagine, though I haven't actually
246−tried this yet) synchronization issues that make threads wait for too long (such
276+tried this yet) finding synchronization issues that make threads wait for too
247−as waiting on an MVar that's not really needed), bad asymptotics with respect
277+long (such as waiting on an MVar that's not really needed), bad asymptotics
248−to some other type of request.
278+with respect to some other type of request.
250(On this last point, imagine not having an index in a database for "people",
280(On this last point, imagine not having an index in a database for "people",
251 and having a tool that discovers that the latency on a search by first name
281 and having a tool that discovers that the latency on a search by first name
252 grows linearly with the number of POST requests to a *different* endpoint! We'd
282 grows linearly with the number of POST requests to a *different* endpoint! We'd
253− need to do some to do this well, possibly involving some machine learning, but
283+ need to do some work to do this well, possibly involving some machine
254− it's an interesting and probably useful idea.)
284+ learning, but it's an interesting and probably useful idea.)
289+I hope this library presents some useful functionality already, but I hope
290+you'll also think how it could be improved!
292+There'll be a few more packages in the comings weeks - check back soon!
257**Note**: This post is an anansi literate file that generates multiple source
294**Note**: This post is an anansi literate file that generates multiple source
258files. They are:
295files. They are: