@@ -9,22 +9,71 @@ an API description.
10This is much closer to the traditional use of `QuickCheck`. The most obvious
10This is much closer to the traditional use of `QuickCheck`. The most obvious
11use-case is checking that properties hold of an *entire* server rather than of
11use-case is checking that properties hold of an *entire* server rather than of
12−individual endpoints.
12+individual endpoints. (But there are other uses that you can skip to if they
13+sound more interesting.)
14## `serverSatisfies`
15## `serverSatisfies`
16−There are a variety of best practices in writing web APIs that aren't always
17+A useful guideline when writing and maintaing software is that, if there isn't
17−obvious. As a running example, let's use a simple service that allows adding,
18+a test for a behaviour or property, sooner or later that property will be broken.
18−removing, and querying biological species. Our SQL schema is:
19+Another important perspective is that tests are a form of documentation - the
20+present developer telling future ones "this matters, and should be this way".
22+The advantage of using tests for this form of documentation is that there's
23+simply too much information to convey, some of it only relevant to very specific
24+use cases, and rather than overload developers with an inexhaustible quantity of
25+details that would be hard to keep track of or remember, tests are a mechanism
26+of reminding developers of *only the relevant information, at the right time*.
29+We might hope that we could use tests to communicate the wide array of best
30+practices that have developed around APIs. About to return a top-level integer
31+in JSON? A test should say that's bad practice. About to not catch exceptions
32+and give a more meaningful HTTP status code? Another test there to stop you.
34+Traditionally, in web services these things get done at the level of *individual*
35+endpoints. But this means that if a developer who hasn't had extensive experience with web
36+programming best practices writes a *new* endpoint which *does* return a top-level
37+integer literal, there's no test there to stop her. Code review might help, but
38+code review is much more error prone than tests, and really only meant for those
39+things that are too subtle to automate. (Indeed, if code review were such a reliable
40+defense mechanism against bugs and bad code, why have tests and linters at all?)
42+The problem, then, with thinking about tests as only existing at the level of individual
43+endpoints is that there are no tests *for* tests - tests that check that new
44+behaviour and tests conforms to higher-level, more general best practices.
46+`servant-quickcheck` aims to solve that. It allows describing properties that
47+*all* endpoints must satisfy. If a new endpoint comes along, it too will be
48+tested for that property, without any further work.
50+Why isn't this idea already popular? Well, most web frameworks don't have a
51+reified description of APIs (beyond perhaps the routes). When you don't know
52+what the endpoints of an application are, and what request body they expect,
53+trying to generate arbitrary requests is almost entirely going to result in
54+404s (not found) and 400s (bad request). Maybe one in a thousand requests will
55+actually test a handler. Not very useful.
57+`servant` applications, on the other hand, have a machine-readable API description
58+already available. And they already associate "correct" requests with particular
59+types. It's a small step, therefore, to generate 'arbitrary' values for these
60+requests, and all of them will go through to your handlers. (Note: all of the
61+uses of `servant-quickcheck` work with applications *not* written with servant-server -
62+and indeed not *in Haskell - but the API must be described with the servant
65+Let's see how this works in practice. As a running example, let's use a simple
66+service that allows adding, removing, and querying biological species. Our SQL
21−> **«schema.sql»**
70+> **schema.sql**
24> CREATE TABLE genus (
73> CREATE TABLE genus (
25> genus_name text PRIMARY KEY,
74> genus_name text PRIMARY KEY,
26> genus_family text NOT NULL
75> genus_family text NOT NULL
29> CREATE TABLE species (
78> CREATE TABLE species (
30> species_name text PRIMARY KEY,
79> species_name text PRIMARY KEY,
@@ -35,7 +84,7 @@ removing, and querying biological species. Our SQL schema is:
35And our actual application:
84And our actual application:
38−> **«Main.hs»**
87+> **Main.hs**
40> {-# LANGUAGE DataKinds #-}
89> {-# LANGUAGE DataKinds #-}
41> {-# LANGUAGE DeriveAnyClass #-}
90> {-# LANGUAGE DeriveAnyClass #-}
@@ -43,6 +92,8 @@ And our actual application:
43> {-# LANGUAGE TypeOperators #-}
92> {-# LANGUAGE TypeOperators #-}
44> {-# LANGUAGE OverloadedStrings #-}
93> {-# LANGUAGE OverloadedStrings #-}
45> {-# LANGUAGE RecordWildCards #-}
94> {-# LANGUAGE RecordWildCards #-}
46> import Servant
97> import Servant
47> import Data.Aeson
98> import Data.Aeson
48> import Database.PostgreSQL.Simple
99> import Database.PostgreSQL.Simple
@@ -54,8 +105,9 @@ And our actual application:
54> type API
105> type API
55> = "species" :> (Capture "species-name" Text :> ( Get '[JSON] Species
106> = "species" :> (Capture "species-name" Text :> ( Get '[JSON] Species
56> :<|> Delete '[JSON] ())
107> :<|> Delete '[JSON] ())
57−> :<|> ReqBody '[JSON] Species :> Post '[JSON] ()
108+> :<|> ReqBody '[JSON] Species :> Post '[JSON] ())
58−> :<|> "count" :> Get '[JSON] Int)
109+> -- The plural of 'species' is unfortunately also 'species'
110+> :<|> "speciess" :> Get '[JSON] [Species]
60> api :: Proxy API
112> api :: Proxy API
61> api = Proxy
113> api = Proxy
@@ -74,19 +126,19 @@ And our actual application:
74> instance FromRow Species
126> instance FromRow Species
76> server :: Connection -> Server API
128> server :: Connection -> Server API
77−> server conn = (\sname -> liftIO (lookupSpecies conn sname)
129+> server conn = ((\sname -> liftIO (lookupSpecies conn sname)
78−> :<|> liftIO (deleteSpecies conn sname))
130+> :<|> liftIO (deleteSpecies conn sname))
79−> :<|> (\species -> liftIO $ insertSpecies conn species)
131+> :<|> (\species -> liftIO $ insertSpecies conn species))
80−> :<|> (liftIO $ countSpecies conn)
132+> :<|> (liftIO $ allSpecies conn)
82> lookupSpecies :: Connection -> Text -> IO Species
134> lookupSpecies :: Connection -> Text -> IO Species
83> lookupSpecies conn name = do
135> lookupSpecies conn name = do
84−> [s] <- query conn "SELECT * FROM species WHERE species_name == ?" (Only name)
136+> [s] <- query conn "SELECT * FROM species WHERE species_name = ?" (Only name)
85> return s
137> return s
87> deleteSpecies :: Connection -> Text -> IO ()
139> deleteSpecies :: Connection -> Text -> IO ()
88> deleteSpecies conn name = do
140> deleteSpecies conn name = do
89−> _ <- execute conn "DELETE FROM species WHERE species_name == ?" (Only name)
141+> _ <- execute conn "DELETE FROM species WHERE species_name = ?" (Only name)
90> return ()
142> return ()
92> insertSpecies :: Connection -> Species -> IO ()
144> insertSpecies :: Connection -> Species -> IO ()
@@ -94,25 +146,173 @@ And our actual application:
94> _ <- execute conn "INSERT INTO species (species_name, species_genus) VALUES (?)" (speciesName, speciesGenus)
146> _ <- execute conn "INSERT INTO species (species_name, species_genus) VALUES (?)" (speciesName, speciesGenus)
95> return ()
147> return ()
97−> countSpecies :: Connection -> IO Int
149+> allSpecies :: Connection -> IO [Species]
98−> countSpecies conn = do
150+> allSpecies conn = do
99−> [Only count] <- query_ conn "SELECT count(*) FROM species"
151+> query_ conn "SELECT * FROM species"
102> main :: IO ()
153> main :: IO ()
103> main = do
154> main = do
104−> conn <- connectPostgreSQL ""
155+> conn <- connectPostgreSQL "dbname=servant-quickcheck"
105> run 8090 (serve api $ server conn)
156> run 8090 (serve api $ server conn)
159+(You'll also also need to run:
162+createdb servant-quickcheck
163+psql --file schema.sql -d servant-quickcheck
166+If you want to run this example.)
168+This is a plausible effort. You might want to spend a moment thinking about what
175+> {-# LANGUAGE OverloadedStrings #-}
176+> module Spec (main) where
178+> import Main (server, api, Species(..))
180+> import Test.QuickCheck.Instances
181+> import Servant.QuickCheck
182+> import Test.QuickCheck (Arbitrary(..))
183+> import Database.PostgreSQL.Simple (connectPostgreSQL)
186+> spec = describe "the species application" $ beforeAll check $ do
188+> conn <- connectPostgreSQL "dbname=servant-quickcheck"
189+> return $ server conn
192+> it "should not return 500s" $ do
194+> it "should not return 500s" $ do
195+> withServantServer api pserver $ \url ->
196+> serverSatisfies api url defaultArgs (not500 <%> mempty)
198+> it "should not return top-level json" $ do
199+> withServantServer api pserver $ \url ->
200+> serverSatisfies api url defaultArgs (onlyJsonObjects <%> mempty)
205+> withServantServer api pserver $ \url ->
206+> serverSatisfies api url defaultArgs (onlyJsonObjects <%> mempty)
212+> instance Arbitrary Species where
213+> arbitrary = Species <$> arbitrary <*> arbitrary
216+But this fails in quite a few ways.
221+This was an example created with the knowledge of what it was supposed to
222+exemplify. To try to get a more accurate assessment of the practical usefulness
223+of `servant-quickcheck`, I tried running `serverSatisfies` with a few
224+predicates over some of the open-source `servant` servers I could find, and
225+results were also promising.
227+There are probably a lot of other interesting properties that one might to add
228+besides those I've included. As an example, we could have a property that
229+all HTML is checked against, which is sometimes tricky for HTML that's
230+generated dynamically. Or check that every page has a Portuguese translation.
232+### Why best practices are good
234+As a side note: you might have wondered "why bother with API best practices?".
235+It is, it has to be said, a lot of extra (as in not only getting the feature done)
236+work to do, for dubious benefit. And indeed, the relevance of discoverability, for
237+example, unclear, since not that many tools use it as perhaps was anticipated.
239+But `servant-quickcheck` both makes it *easier* to conform to best practices,
240+and exemplifies their advantage in enabling better tooling. If we pick 201 (Success, the 'resource' was
241+created), rather than the more generic 200 (Success), and do a *little* more work
242+by knowing to make this decision, `servant-quickcheck` knows this means there
243+should be some representation of the resource created. So it knows to ask you
244+for a link to it (the RFC creators thought to ask for this). And if you do (again,
245+a little more work), `servant-quickcheck` will know to try to look at that
246+resource by following the link, checking that it's not broken, and maybe even
247+returns a response that equivalent to the original POST request). And then it
248+finds a real bug - your application allows species with '/' in their name to
249+be created, but not queried with a 'GET' for! This, I think, is already a win.
254+There's another very appealing application of the ability to generate "sensible"
255+arbitrary requests. It's for testing that two applications are equal. We can generate arbitrary
256+requests, send them to both servers (in the same order), and check that the responses
257+are equivalent. (This was, incidentally, one of the first applications of
258+`servant-client`, albeit in a much more manual way, when we rewrote a microservice
259+originally in Python in Haskell.) Generally with rewrites, even if there's some
260+behaviour that isn't optimal, perhaps a lot of things already depend on that service
261+and make interace poorly with "improvements", so it makes sense to first mimick
262+*exactly* the original behaviour, and only then aim for improvements.
264+`servant-quickcheck` provides a single function, `serversEqual`, that attempts
265+to verify the equivalence of servers. Since some aspects of responses might not
266+be relevant (for example, whether the the `Server` header is the same, or whether
267+two JSON responses have the same formatting), it allows you to provide a custom
268+equivalence function. Other than that, you need only provide an API type and two
269+URLs for testing, and the rest `serversEqual` handles.
271+## Future directions: benchmarking
273+What else could benefit from tooling that can automatically generate sensible
274+(*vis-a-vis* a particular application's expectations) requests?
276+One area is extensive automatic benchmarking. Currently we use tools such as
277+`ab`, `wrk`, `httperf` in a very manual way - we pick a particular request that
278+we are interested in, and write a request that gets made thousands of times.
279+But now we can have a multiplicity of requests to benchmark with! This allows
280+*finding* slow endpoints, as well as (I would imagine, though I haven't actually
281+tried this yet) finding synchronization issues that make threads wait for too
282+long (such as waiting on an MVar that's not really needed), bad asymptotics
283+with respect to some other type of request.
285+(On this last point, imagine not having an index in a database for "people",
286+ and having a tool that discovers that the latency on a search by first name
287+ grows linearly with the number of POST requests to a *different* endpoint! We'd
288+ need to do some work to do this well, possibly involving some machine
289+ learning, but it's an interesting and probably useful idea.)
294+I hope this library presents some useful functionality already, but I hope
295+you'll also think how it could be improved!
297+There'll be a few more packages in the comings weeks - check back soon!
299+**Note**: This post is an anansi literate file that generates multiple source
109−> **» Main.hs**
309+> ** schema.sql**
111−> «Main.hs»
311+> schema.sql
115−> **» schema.sql**
315+> ** Spec.hs**
117−> «schema.sql»
317+> Spec.hs