@@ -9,22 +9,71 @@ an API description.
99
1010This is much closer to the traditional use of `QuickCheck`. The most obvious
1111use-case is checking that properties hold of an *entire* server rather than of
12−individual endpoints.
12+individual endpoints. (But there are other uses that you can skip to if they
13+sound more interesting.)
1314
1415## `serverSatisfies`
1516
16−There are a variety of best practices in writing web APIs that aren't always
17−obvious. As a running example, let's use a simple service that allows adding,
18−removing, and querying biological species. Our SQL schema is:
17+A useful guideline when writing and maintaing software is that, if there isn't
18+a test for a behaviour or property, sooner or later that property will be broken.
19+Another important perspective is that tests are a form of documentation - the
20+present developer telling future ones "this matters, and should be this way".
1921
22+The advantage of using tests for this form of documentation is that there's
23+simply too much information to convey, some of it only relevant to very specific
24+use cases, and rather than overload developers with an inexhaustible quantity of
25+details that would be hard to keep track of or remember, tests are a mechanism
26+of reminding developers of *only the relevant information, at the right time*.
27+<<EXAMPLE>>.
2028
21−> **«schema.sql»**
29+We might hope that we could use tests to communicate the wide array of best
30+practices that have developed around APIs. About to return a top-level integer
31+in JSON? A test should say that's bad practice. About to not catch exceptions
32+and give a more meaningful HTTP status code? Another test there to stop you.
33+
34+Traditionally, in web services these things get done at the level of *individual*
35+endpoints. But this means that if a developer who hasn't had extensive experience with web
36+programming best practices writes a *new* endpoint which *does* return a top-level
37+integer literal, there's no test there to stop her. Code review might help, but
38+code review is much more error prone than tests, and really only meant for those
39+things that are too subtle to automate. (Indeed, if code review were such a reliable
40+defense mechanism against bugs and bad code, why have tests and linters at all?)
41+
42+The problem, then, with thinking about tests as only existing at the level of individual
43+endpoints is that there are no tests *for* tests - tests that check that new
44+behaviour and tests conforms to higher-level, more general best practices.
45+
46+`servant-quickcheck` aims to solve that. It allows describing properties that
47+*all* endpoints must satisfy. If a new endpoint comes along, it too will be
48+tested for that property, without any further work.
49+
50+Why isn't this idea already popular? Well, most web frameworks don't have a
51+reified description of APIs (beyond perhaps the routes). When you don't know
52+what the endpoints of an application are, and what request body they expect,
53+trying to generate arbitrary requests is almost entirely going to result in
54+404s (not found) and 400s (bad request). Maybe one in a thousand requests will
55+actually test a handler. Not very useful.
56+
57+`servant` applications, on the other hand, have a machine-readable API description
58+already available. And they already associate "correct" requests with particular
59+types. It's a small step, therefore, to generate 'arbitrary' values for these
60+requests, and all of them will go through to your handlers. (Note: all of the
61+uses of `servant-quickcheck` work with applications *not* written with servant-server -
62+and indeed not *in Haskell - but the API must be described with the servant
63+DSL.)
64+
65+Let's see how this works in practice. As a running example, let's use a simple
66+service that allows adding, removing, and querying biological species. Our SQL
67+schema is:
68+
69+
70+> **schema.sql**
2271
2372>
2473> CREATE TABLE genus (
2574> genus_name text PRIMARY KEY,
2675> genus_family text NOT NULL
27−> )
76+> );
2877>
2978> CREATE TABLE species (
3079> species_name text PRIMARY KEY,
@@ -35,7 +84,7 @@ removing, and querying biological species. Our SQL schema is:
3584And our actual application:
3685
3786
38−> **«Main.hs»**
87+> **Main.hs**
3988
4089> {-# LANGUAGE DataKinds #-}
4190> {-# LANGUAGE DeriveAnyClass #-}
@@ -43,6 +92,8 @@ And our actual application:
4392> {-# LANGUAGE TypeOperators #-}
4493> {-# LANGUAGE OverloadedStrings #-}
4594> {-# LANGUAGE RecordWildCards #-}
95+> module Main where
96+>
4697> import Servant
4798> import Data.Aeson
4899> import Database.PostgreSQL.Simple
@@ -54,8 +105,9 @@ And our actual application:
54105> type API
55106> = "species" :> (Capture "species-name" Text :> ( Get '[JSON] Species
56107> :<|> Delete '[JSON] ())
57−> :<|> ReqBody '[JSON] Species :> Post '[JSON] ()
58−> :<|> "count" :> Get '[JSON] Int)
108+> :<|> ReqBody '[JSON] Species :> Post '[JSON] ())
109+> -- The plural of 'species' is unfortunately also 'species'
110+> :<|> "speciess" :> Get '[JSON] [Species]
59111>
60112> api :: Proxy API
61113> api = Proxy
@@ -74,19 +126,19 @@ And our actual application:
74126> instance FromRow Species
75127>
76128> server :: Connection -> Server API
77−> server conn = (\sname -> liftIO (lookupSpecies conn sname)
78−> :<|> liftIO (deleteSpecies conn sname))
79−> :<|> (\species -> liftIO $ insertSpecies conn species)
80−> :<|> (liftIO $ countSpecies conn)
129+> server conn = ((\sname -> liftIO (lookupSpecies conn sname)
130+> :<|> liftIO (deleteSpecies conn sname))
131+> :<|> (\species -> liftIO $ insertSpecies conn species))
132+> :<|> (liftIO $ allSpecies conn)
81133>
82134> lookupSpecies :: Connection -> Text -> IO Species
83135> lookupSpecies conn name = do
84−> [s] <- query conn "SELECT * FROM species WHERE species_name == ?" (Only name)
136+> [s] <- query conn "SELECT * FROM species WHERE species_name = ?" (Only name)
85137> return s
86138>
87139> deleteSpecies :: Connection -> Text -> IO ()
88140> deleteSpecies conn name = do
89−> _ <- execute conn "DELETE FROM species WHERE species_name == ?" (Only name)
141+> _ <- execute conn "DELETE FROM species WHERE species_name = ?" (Only name)
90142> return ()
91143>
92144> insertSpecies :: Connection -> Species -> IO ()
@@ -94,25 +146,173 @@ And our actual application:
94146> _ <- execute conn "INSERT INTO species (species_name, species_genus) VALUES (?)" (speciesName, speciesGenus)
95147> return ()
96148>
97−> countSpecies :: Connection -> IO Int
98−> countSpecies conn = do
99−> [Only count] <- query_ conn "SELECT count(*) FROM species"
100−> return count
149+> allSpecies :: Connection -> IO [Species]
150+> allSpecies conn = do
151+> query_ conn "SELECT * FROM species"
101152>
102153> main :: IO ()
103154> main = do
104−> conn <- connectPostgreSQL ""
155+> conn <- connectPostgreSQL "dbname=servant-quickcheck"
105156> run 8090 (serve api $ server conn)
106157
107158
159+(You'll also also need to run:
160+
161+```
162+createdb servant-quickcheck
163+psql --file schema.sql -d servant-quickcheck
164+```
165+
166+If you want to run this example.)
167+
168+This is a plausible effort. You might want to spend a moment thinking about what
169+could be improved.
170+
171+
172+> **Spec.hs**
173+
174+>
175+> {-# LANGUAGE OverloadedStrings #-}
176+> module Spec (main) where
177+>
178+> import Main (server, api, Species(..))
179+> import Test.Hspec
180+> import Test.QuickCheck.Instances
181+> import Servant.QuickCheck
182+> import Test.QuickCheck (Arbitrary(..))
183+> import Database.PostgreSQL.Simple (connectPostgreSQL)
184+>
185+> spec :: Spec
186+> spec = describe "the species application" $ beforeAll check $ do
187+> let pserver = do
188+> conn <- connectPostgreSQL "dbname=servant-quickcheck"
189+> return $ server conn
190+>
191+>
192+> it "should not return 500s" $ do
193+>
194+> it "should not return 500s" $ do
195+> withServantServer api pserver $ \url ->
196+> serverSatisfies api url defaultArgs (not500 <%> mempty)
197+>
198+> it "should not return top-level json" $ do
199+> withServantServer api pserver $ \url ->
200+> serverSatisfies api url defaultArgs (onlyJsonObjects <%> mempty)
201+>
202+> where
203+> check = do
204+> mvar <- newMVar []
205+> withServantServer api pserver $ \url ->
206+> serverSatisfies api url defaultArgs (onlyJsonObjects <%> mempty)
207+>
208+> main :: IO ()
209+> main = do
210+> hspec spec
211+>
212+> instance Arbitrary Species where
213+> arbitrary = Species <$> arbitrary <*> arbitrary
214+
215+
216+But this fails in quite a few ways.
217+
218+
219+<<TODO>>
220+
221+This was an example created with the knowledge of what it was supposed to
222+exemplify. To try to get a more accurate assessment of the practical usefulness
223+of `servant-quickcheck`, I tried running `serverSatisfies` with a few
224+predicates over some of the open-source `servant` servers I could find, and
225+results were also promising.
226+
227+There are probably a lot of other interesting properties that one might to add
228+besides those I've included. As an example, we could have a property that
229+all HTML is checked against, which is sometimes tricky for HTML that's
230+generated dynamically. Or check that every page has a Portuguese translation.
231+
232+### Why best practices are good
233+
234+As a side note: you might have wondered "why bother with API best practices?".
235+It is, it has to be said, a lot of extra (as in not only getting the feature done)
236+work to do, for dubious benefit. And indeed, the relevance of discoverability, for
237+example, unclear, since not that many tools use it as perhaps was anticipated.
238+
239+But `servant-quickcheck` both makes it *easier* to conform to best practices,
240+and exemplifies their advantage in enabling better tooling. If we pick 201 (Success, the 'resource' was
241+created), rather than the more generic 200 (Success), and do a *little* more work
242+by knowing to make this decision, `servant-quickcheck` knows this means there
243+should be some representation of the resource created. So it knows to ask you
244+for a link to it (the RFC creators thought to ask for this). And if you do (again,
245+a little more work), `servant-quickcheck` will know to try to look at that
246+resource by following the link, checking that it's not broken, and maybe even
247+returns a response that equivalent to the original POST request). And then it
248+finds a real bug - your application allows species with '/' in their name to
249+be created, but not queried with a 'GET' for! This, I think, is already a win.
250+
251+
252+## `serversEqual`
253+
254+There's another very appealing application of the ability to generate "sensible"
255+arbitrary requests. It's for testing that two applications are equal. We can generate arbitrary
256+requests, send them to both servers (in the same order), and check that the responses
257+are equivalent. (This was, incidentally, one of the first applications of
258+`servant-client`, albeit in a much more manual way, when we rewrote a microservice
259+originally in Python in Haskell.) Generally with rewrites, even if there's some
260+behaviour that isn't optimal, perhaps a lot of things already depend on that service
261+and make interace poorly with "improvements", so it makes sense to first mimick
262+*exactly* the original behaviour, and only then aim for improvements.
263+
264+`servant-quickcheck` provides a single function, `serversEqual`, that attempts
265+to verify the equivalence of servers. Since some aspects of responses might not
266+be relevant (for example, whether the the `Server` header is the same, or whether
267+two JSON responses have the same formatting), it allows you to provide a custom
268+equivalence function. Other than that, you need only provide an API type and two
269+URLs for testing, and the rest `serversEqual` handles.
270+
271+## Future directions: benchmarking
272+
273+What else could benefit from tooling that can automatically generate sensible
274+(*vis-a-vis* a particular application's expectations) requests?
275+
276+One area is extensive automatic benchmarking. Currently we use tools such as
277+`ab`, `wrk`, `httperf` in a very manual way - we pick a particular request that
278+we are interested in, and write a request that gets made thousands of times.
279+But now we can have a multiplicity of requests to benchmark with! This allows
280+*finding* slow endpoints, as well as (I would imagine, though I haven't actually
281+tried this yet) finding synchronization issues that make threads wait for too
282+long (such as waiting on an MVar that's not really needed), bad asymptotics
283+with respect to some other type of request.
284+
285+(On this last point, imagine not having an index in a database for "people",
286+ and having a tool that discovers that the latency on a search by first name
287+ grows linearly with the number of POST requests to a *different* endpoint! We'd
288+ need to do some work to do this well, possibly involving some machine
289+ learning, but it's an interesting and probably useful idea.)
290+
291+
292+# Conclusion
293+
294+I hope this library presents some useful functionality already, but I hope
295+you'll also think how it could be improved!
296+
297+There'll be a few more packages in the comings weeks - check back soon!
298+
299+**Note**: This post is an anansi literate file that generates multiple source
300+files. They are:
301+
302+
303+> ** Main.hs**
304+
305+> Main.hs
306+
307+
108308
109−> **» Main.hs**
309+> ** schema.sql**
110310
111−> «Main.hs»
311+> schema.sql
112312
113313
114314
115−> **» schema.sql**
315+> ** Spec.hs**
116316
117−> «schema.sql»
317+> Spec.hs
118318