@@ -0,0 +1,505 @@
1+# How To Test Servant Applications
2+
3+Even with a nicely structured API that passes Haskell's strict type checker,
4+it's a good idea to write some tests for your application.
5+
6+In this recipe we'll work through some common testing strategies and provide
7+examples of utlizing these testing strategies in order to test Servant
8+applications.
9+
10+## Testing strategies
11+
12+There are many testing strategies you may wish to employ when testing your
13+Servant application, but included below are three common testing patterns:
14+
15+- We'll use `servant-client` to derive client functions and then send valid
16+requests to our API, running in another thread. This is great for testing
17+that our **business logic** is correctly implemented with only valid HTTP
18+requests.
19+
20+- We'll also use `hspec-wai` to make **arbitrary HTTP requests**, in order to
21+test how our application may respond to invalid or otherwise unexpected
22+requests.
23+
24+- Finally, we can also use `servant-quickcheck` for **whole-API tests**, in order
25+to assert that our entire application conforms to **best practices**.
26+
27+## Useful Libraries
28+
29+The following libraries will often come in handy when we decide to test our
30+Servant applications:
31+
32+- [hspec](https://hspec.github.io/)
33+- [hspec-wai](http://hackage.haskell.org/package/hspec-wai)
34+- [QuickCheck](http://hackage.haskell.org/package/QuickCheck)
35+- [servant-quickcheck](https://hackage.haskell.org/package/servant-quickcheck)
36+
37+## Imports and Our Testing Module
38+
39+This recipe starts with the following ingredients:
40+
41+```haskell
42+{-# LANGUAGE OverloadedStrings, TypeFamilies, DataKinds,
43+ DeriveGeneric, TypeOperators #-}
44+import qualified Control.Concurrent as C
45+import Control.Concurrent.MVar
46+import Control.Exception (bracket)
47+import Control.Lens hiding (Context)
48+import Data.Aeson
49+import Data.Aeson.Lens
50+import qualified Data.HashMap.Strict as HM
51+import Data.Text
52+import GHC.Generics
53+import Network.HTTP.Client hiding (Proxy)
54+import Network.HTTP.Types
55+import Network.Wai
56+import qualified Network.Wai.Handler.Warp as Warp
57+
58+import Servant
59+import Servant.Client
60+import Servant.Server
61+import Servant.QuickCheck
62+import Servant.QuickCheck.Internal (serverDoesntSatisfy)
63+
64+import Test.Hspec
65+import Test.Hspec.Wai
66+import Test.Hspec.Wai.Matcher
67+```
68+
69+We're going to produce different `Spec`s that represent different
70+aspects of our application, and we'll ask `hspec` to run all of our different
71+`Spec`s. This is a common organizational method for testing modules:
72+
73+```haskell
74+spec :: Spec
75+spec = do
76+ businessLogicSpec
77+ thirdPartyResourcesSpec
78+ servantQuickcheckSpec
79+```
80+
81+Often, codebases will use `hspec`'s
82+[autodiscover pragma](http://hspec.github.io/hspec-discover.html)
83+to find all testing modules and `Spec`s inside, but we're going to
84+explicitly make a `main` function to run our tests because we have only one
85+`spec` defined above:
86+
87+```haskell
88+main :: IO ()
89+main = hspec spec
90+```
91+
92+## Testing Your Business Logic
93+
94+Let's say we have an API that looks something like this:
95+
96+```haskell
97+data User = User {
98+ name :: Text
99+ , user_id :: Integer
100+ } deriving (Eq, Show, Generic)
101+
102+instance FromJSON User
103+instance ToJSON User
104+
105+type UserApi =
106+ -- One endpoint: create a user
107+ "user" :> Capture "userId" Integer :> Post '[JSON] User
108+```
109+
110+A real server would likely use a database to store, retrieve, and validate
111+users, but we're going to do something really simple merely to have something
112+to test. With that said, here's a sample handler, server, and `Application`
113+for the endpoint described above:
114+
115+```haskell
116+userApp :: Application
117+userApp = serve (Proxy :: Proxy UserApi) userServer
118+
119+userServer :: Server UserApi
120+userServer = createUser
121+
122+createUser :: Integer -> Handler User
123+createUser userId = do
124+ if userId > 5000
125+ then pure $ User { name = "some user", user_id = userId }
126+ else throwError $ err400 { errBody = "userId is too small" }
127+```
128+
129+### Strategy 1: Spin Up a Server, Create a Client, Make Some Requests
130+
131+One of the benefits of Servant's type-level DSL for describing APIs is that
132+once you have provided a type-level description of your API, you can create
133+clients, documentation, or other tools for it somewhat magically.
134+
135+In this case, we'd like to *test* our server, so we can use `servant-client`
136+to create a client, after which we'll run our server, and then make requests
137+of it and see how it responds.
138+
139+Let's write some tests:
140+
141+```haskell
142+withUserApp :: IO () -> IO ()
143+withUserApp action =
144+ -- we can spin up a server in another thread and kill that thread when done
145+ -- in an exception-safe way
146+ bracket (liftIO $ C.forkIO $ Warp.run 8888 userApp)
147+ C.killThread
148+ (const action)
149+
150+
151+businessLogicSpec :: Spec
152+businessLogicSpec =
153+ -- `around` will start our Server before the tests and turn it off after
154+ around_ withUserApp $ do
155+ -- create a test client function
156+ let createUser = client (Proxy :: Proxy UserApi)
157+ -- create a servant-client ClientEnv
158+ baseUrl <- runIO $ parseBaseUrl "http://localhost:8888"
159+ manager <- runIO $ newManager defaultManagerSettings
160+ let clientEnv = mkClientEnv manager baseUrl
161+
162+ -- testing scenarios start here
163+ describe "POST /user" $ do
164+ it "should create a user with a high enough ID" $ do
165+ result <- runClientM (createUser 50001) clientEnv
166+ result `shouldBe` (Right $ User { name = "some user", user_id = 50001})
167+ it "will it fail with a too-small ID?" $ do
168+ result <- runClientM (createUser 4999) clientEnv
169+ result `shouldBe` (Right $ User { name = "some user", user_id = 50001})
170+```
171+
172+### Running These Tests
173+
174+Let's run our tests and see what happens:
175+
176+```
177+$ cabal new-test all
178+POST /user
179+ should create a user with a high enough ID
180+ should fail with a too-small ID FAILED [1]
181+
182+ Failures:
183+
184+ Testing.lhs:129:7:
185+ 1) POST /user should fail with a too-small ID
186+ expected: Right (User {name = "some user", user_id = 50001})
187+ but got: Left (FailureResponse (Response {responseStatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, responseHeaders = fromList [("Transfer-Encoding","chunked"),("Date","Fri, 12 Oct 2018 04:36:22 GMT"),("Server","Warp/3.2.25")], responseHttpVersion = HTTP/1.1, responseBody = "userId is too small"}))
188+
189+ To rerun use: --match "/POST /user/should fail with a too-small ID/"
190+```
191+
192+Hmm. One passed and one failed! It looks like I *was* expecting a success
193+response in the second test, but I actually got a failure. We should fix that,
194+but first I'd like to introduce `hspec-wai`, which will give us different
195+mechanisms for making requests of our application and validating the responses
196+we get. We're also going to spin up a fake Elasticsearch server, so that our
197+server can think it's talking to a real database.
198+
199+
200+## *Mocking* 3rd Party Resources
201+
202+Often our web applications will need to make their own web
203+requests to other 3rd-party applications. These requests provide a lot
204+of opportunity for failure and so we'd like to test that the right
205+messages and failure values (in addition to success values) are returned
206+from our application.
207+
208+### Define the 3rd-Party Resource
209+
210+With Servant's type-level API definitions, assuming you've already defined the
211+API you want to mock, it's relatively trivial to create a simple server for
212+the purposes of running tests. For instance, consider an API server that needs
213+to get data out of Elasticsearch. Let's first define the Elasticsearch server
214+and client using Servant API descriptions:
215+
216+```haskell
217+type SearchAPI =
218+ -- We're using Aeson's Generic JSON `Value` to make things easier on
219+ -- ourselves. We're also representing only one Elasticsearch endpoint:
220+ -- get item by id
221+ "myIndex" :> "myDocType" :> Capture "docId" Integer :> Get '[JSON] Value
222+
223+-- Here's our Servant Client function
224+getDocument = client (Proxy :: Proxy SearchAPI)
225+
226+-- We can use these helpers when we want to make requests
227+-- using our client function
228+clientEnv :: Text -> Text -> IO ClientEnv
229+clientEnv esHost esPort = do
230+ baseUrl <- parseBaseUrl $ unpack $ esHost <> ":" <> esPort
231+ manager <- newManager defaultManagerSettings
232+ pure $ mkClientEnv manager baseUrl
233+
234+runSearchClient :: Text -> Text -> ClientM a -> IO (Either ServantError a)
235+runSearchClient esHost esPort = (clientEnv esHost esPort >>=) . runClientM
236+```
237+
238+### Servant Server Example Using this 3rd-Party Resource
239+
240+So we've got an Elasticsearch server and a client to talk to it. Let's now
241+build a simple app server that uses this client to retrieve documents. This
242+is somewhat contrived, but hopefully it illustrates the typical three-tier
243+application architecture.
244+
245+One note: we're also going to take advantage of `lens-aeson` here, which may
246+look a bit foreign. The gist of it is that we're going to traverse a JSON
247+`Value` from Elasticsearch and try to extract some kind of document to
248+return.
249+
250+Imagine, then, that this is our real server implementation:
251+
252+```haskell
253+type DocApi =
254+ "docs" :> Capture "docId" Integer :> Get '[JSON] Value
255+
256+docsApp :: Text -> Text -> Application
257+docsApp esHost esPort = serve (Proxy :: Proxy DocApi) $ docServer esHost esPort
258+
259+docServer :: Text -> Text -> Server DocApi
260+docServer esHost esPort = getDocById esHost esPort
261+
262+-- Our Handler tries to get a doc from Elasticsearch and then tries to parse
263+-- it. Unfortunately, there's a lot of opportunity for failure in these
264+-- actions
265+getDocById :: Text -> Text -> Integer -> Handler Value
266+getDocById esHost esPort docId = do
267+ -- Our Servant Client function returns Either ServantError Value here:
268+ docRes <- liftIO $ runSearchClient esHost esPort (getDocument docId)
269+ case docRes of
270+ Left err -> throwError $ err404 { errBody = "Failed looking up content" }
271+ Right value -> do
272+ -- we'll either fail to parse our document or we'll return it
273+ case value ^? _Object . ix "_source" of
274+ Nothing -> throwError $ err400 { errBody = "Failed parsing content" }
275+ Just obj -> pure obj
276+```
277+
278+### Testing Our Backend
279+
280+So the above represents our application and is close to a server we may
281+actually deploy. How then shall we test this application?
282+
283+Ideally, we'd like it to make requests of a *real* Elasticsearch server, but
284+we certainly don't want our tests to trigger requests to a live, production
285+database. In addition, we don't want to depend on our real Elasticsearch
286+server having specific, consistent results for us to test against, because
287+that would make our tests flaky (and flaky tests are sometimes described as
288+worse than not having tests at all).
289+
290+One solution to this is to create a trivial Elasticsearch server as part of
291+our testing code. We can do this relatively easily because we already have
292+an API definition for it above. With a *real* server, we can then let our own
293+application make requests of it and we'll simulate different scenarios in
294+order to make sure our application responds the way we expect it to.
295+
296+Let's start with some helpers which will allow us to run a testing version
297+of our Elasticsearch server in another thread:
298+
299+```haskell
300+-- | We'll run the Elasticsearch server so we can test behaviors
301+withElasticsearch :: IO () -> IO ()
302+withElasticsearch action =
303+ bracket (liftIO $ C.forkIO $ Warp.run 9999 esTestApp)
304+ C.killThread
305+ (const action)
306+
307+esTestApp :: Application
308+esTestApp = serve (Proxy :: Proxy SearchAPI) esTestServer
309+
310+esTestServer :: Server SearchAPI
311+esTestServer = getESDocument
312+
313+-- This is the *mock* handler we're going to use. We create it
314+-- here specifically to trigger different behavior in our tests.
315+getESDocument :: Integer -> Handler Value
316+getESDocument docId
317+ -- arbitrary things we can use in our tests to simulate failure:
318+ -- we want to trigger different code paths.
319+ | docId > 1000 = throwError err500
320+ | docId > 500 = pure . Object $ HM.fromList [("bad", String "data")]
321+ | otherwise = pure $ Object $ HM.fromList [("_source", Object $ HM.fromList [("a", String "b")])]
322+```
323+
324+Now, we should be ready to write some tests.
325+
326+In this case, we're going to use `hspec-wai`, which will give us a simple way
327+to run our application, make requests, and make assertions against the
328+responses we receive.
329+
330+Hopefully, this will simplify our testing code:
331+
332+```haskell
333+thirdPartyResourcesSpec :: Spec
334+thirdPartyResourcesSpec = around_ withElasticsearch $ do
335+ -- we call `with` from `hspec-wai` and pass *real* `Application`
336+ with (pure $ docsApp "localhost" "9999") $ do
337+ describe "GET /docs" $ do
338+ it "should be able to get a document" $
339+ -- `get` is a function from hspec-wai`.
340+ get "/docs/1" `shouldRespondWith` 200
341+ it "should be able to handle connection failures" $
342+ get "/docs/1001" `shouldRespondWith` 404
343+ it "should be able to handle parsing failures" $
344+ get "/docs/501" `shouldRespondWith` 400
345+ it "should be able to handle odd HTTP requests" $
346+ -- we can also make all kinds of arbitrary custom requests to see how
347+ -- our server responds using the `request` function:
348+ -- request :: Method -> ByteString -> [Header]
349+ -- -> LB.ByteString -> WaiSession SResponse
350+ request methodPost "/docs/501" [] "{" `shouldRespondWith` 405
351+ it "we can also do more with the Response using hspec-wai's matchers" $
352+ -- see also `MatchHeader` and JSON-matching tools as well...
353+ get "/docs/1" `shouldRespondWith` 200 { matchBody = MatchBody bodyMatcher }
354+
355+bodyMatcher :: [Network.HTTP.Types.Header] -> Body -> Maybe String
356+bodyMatcher _ body = case (decode body :: Maybe Value) of
357+ -- success in this case means we return `Nothing`
358+ Just val | val == (Object $ HM.fromList [("a", String "b")]) -> Nothing
359+ _ -> Just "This is how we represent failure: this message will be printed"
360+```
361+
362+Out of the box, `hspec-wai` provides a lot of useful tools for us to run tests
363+against our application. What happens when we run these tests?
364+
365+```
366+$ cabal new-test all
367+...
368+
369+GET /docs
370+ should be able to get a document
371+ should be able to handle connection failures
372+ should be able to handle parsing failures
373+ should be able to handle odd HTTP requests
374+ we can also do more with the Response using hspec-wai's matchers
375+```
376+
377+Fortunately, they all passed! Let's move to another strategy: whole-API
378+testing.
379+
380+## Servant Quickcheck
381+
382+[`servant-quickcheck`](https://github.com/haskell-servant/servant-quickcheck)
383+is a project that allows users to write tests for whole Servant APIs using
384+quickcheck-style property-checking mechanisms.
385+
386+`servant-quickcheck` is great for asserting API-wide rules, such as "no
387+endpoint throws a 500" or "all 301 status codes also come with a Location
388+header". The project even comes with a number of predicates that reference
389+the [RFCs they originate from](https://github.com/haskell-servant/servant-quickcheck/blob/master/src/Servant/QuickCheck/Internal/Predicates.hs).
390+
391+In other words, it's one way to assert that your APIs conform to specs and
392+best practices.
393+
394+### Quickcheckable API
395+
396+Let's make an API and a server to demonstrate how to use `servant-quickcheck`:
397+
398+```haskell
399+type API = ReqBody '[JSON] String :> Post '[JSON] String
400+ :<|> Get '[JSON] Int
401+ :<|> BasicAuth "some-realm" () :> Get '[JSON] ()
402+
403+api :: Proxy API
404+api = Proxy
405+
406+server :: IO (Server API)
407+server = do
408+ mvar <- newMVar ""
409+ return $ (\x -> liftIO $ swapMVar mvar x)
410+ :<|> (liftIO $ readMVar mvar >>= return . Prelude.length)
411+ :<|> (const $ return ())
412+```
413+
414+### Using `servant-quickcheck`
415+
416+Let's build some tests for our API using `servant-quickcheck`.
417+
418+Similar to the above examples, we're going to create `Spec`s, but in this
419+case, we'll rely on a number of predicates available from `servant-quickcheck`
420+to see if our API server conforms to best practices:
421+
422+```haskell
423+-- Let's set some QuickCheck values
424+args :: Args
425+args = defaultArgs { maxSuccess = 500 }
426+
427+-- Here's a Servant Context object we'll use
428+ctx :: Context '[BasicAuthCheck ()]
429+ctx = BasicAuthCheck (const . return $ NoSuchUser) :. EmptyContext
430+
431+
432+servantQuickcheckSpec :: Spec
433+servantQuickcheckSpec = describe "" $ do
434+ it "API demonstrates best practices" $
435+ -- `withServerServer` and `withServantServerAndContext` come from `servant-quickcheck`
436+ withServantServerAndContext api ctx server $ \burl ->
437+ -- `serverSatisfies` and the predicates also come from `servant-quickcheck`
438+ serverSatisfies api burl args (unauthorizedContainsWWWAuthenticate
439+ <%> not500
440+ <%> onlyJsonObjects -- this one isn't true!
441+ <%> mempty)
442+
443+ it "API doesn't have these things implemented yet" $
444+ withServantServerAndContext api ctx server $ \burl -> do
445+ serverDoesntSatisfy api burl args (getsHaveCacheControlHeader
446+ <%> notAllowedContainsAllowHeader
447+ <%> mempty)
448+```
449+
450+Let's see what happens when we run these tests:
451+
452+
453+```
454+ API demonstrates best practices FAILED [2]
455++++ OK, passed 500 tests.
456+ API doesn't have these things implemented yet
457+
458+ src/Servant/QuickCheck/Internal/QuickCheck.hs:143:11:
459+ 2) Main[339:25] API demonstrates best practices
460+ Failed:
461+ Just Predicate failed
462+ Predicate: onlyJsonObjects
463+
464+ Response:
465+ Status code: 200
466+ Headers: "Transfer-Encoding": "chunked"
467+ "Date": "Fri, 12 Oct 2018 04:36:22 GMT"
468+ "Server": "Warp/3.2.25"
469+ "Content-Type": "application/json;charset=utf-8"
470+ Body: ""
471+
472+ To rerun use: --match "/Main[339:25]/API demonstrates best practices/"
473+
474+Randomized with seed 1046277487
475+
476+Finished in 0.4306 seconds
477+```
478+
479+Hmm. It looks like we *thought* our API only returned JSON objects, which is a
480+best practice, but in fact, we *did* have an endpoint that returned an empty
481+body, which you can see in the printed response above: `Body: ""`. We should
482+consider revising our API to only return top-level JSON Objects in the future!
483+
484+### Other Cool Things
485+
486+`servant-quickcheck` also has a cool mechanism where you can compare two API
487+servers to demonstrate that they respond identically to requests. This may be
488+useful if you are planning to rewrite one API in another language or with
489+another web framework. You have to specify whether you're looking for
490+`jsonEquality` vs regular `ByteString` equality, though.
491+
492+## Conclusion
493+
494+There are lots of techniques for testing and we only covered a few here.
495+
496+Useful libraries such as `hspec-wai` have ways of running Wai `Application`s
497+and sending requests to them, while Servant's type-level DSL for defining APIs
498+allows us to more easily mock out servers and to derive clients, which will
499+only craft valid requests.
500+
501+Lastly, if you want a broad overview of where your application fits in with
502+regard to best practices, consider using `servant-quickcheck`.
503+
504+This program is available as a cabal project
505+[here](https://github.com/haskell-servant/servant/tree/master/doc/cookbook/testing).