git.iko.soy/servant
on d64c232 help sign in
servant/doc/cookbook/testing/Testing.lhs
508 lines 19 kB history raw
1# How To Test Servant Applications
3Even with a nicely structured API that passes Haskell's strict type checker,
4it's a good idea to write some tests for your application.
6In this recipe we'll work through some common testing strategies and provide
7examples of utlizing these testing strategies in order to test Servant
8applications.
10## Testing strategies
12There are many testing strategies you may wish to employ when testing your
13Servant application, but included below are three common testing patterns:
15- We'll use `servant-client` to derive client functions and then send valid
16requests to our API, running in another thread. This is great for testing
17that our **business logic** is correctly implemented with only valid HTTP
18requests.
20- We'll also use `hspec-wai` to make **arbitrary HTTP requests**, in order to
21test how our application may respond to invalid or otherwise unexpected
22requests.
24- Finally, we can also use `servant-quickcheck` for **whole-API tests**, in order
25to assert that our entire application conforms to **best practices**.
27## Useful Libraries
29The following libraries will often come in handy when we decide to test our
30Servant applications:
32- [hspec](https://hspec.github.io/)
33- [hspec-wai](http://hackage.haskell.org/package/hspec-wai)
34- [QuickCheck](http://hackage.haskell.org/package/QuickCheck)
35- [servant-quickcheck](https://hackage.haskell.org/package/servant-quickcheck)
37## Imports and Our Testing Module
39This recipe starts with the following ingredients:
41```haskell
42{-# LANGUAGE OverloadedStrings, TypeFamilies, DataKinds,
43 DeriveGeneric, TypeOperators #-}
44import Prelude ()
45import Prelude.Compat
47import qualified Control.Concurrent as C
48import Control.Concurrent.MVar
49import Control.Exception (bracket)
50import Control.Lens hiding (Context)
51import Data.Aeson
52import Data.Aeson.Lens
53import qualified Data.HashMap.Strict as HM
54import Data.Text (Text, unpack)
55import GHC.Generics
56import Network.HTTP.Client hiding (Proxy)
57import Network.HTTP.Types
58import Network.Wai
59import qualified Network.Wai.Handler.Warp as Warp
61import Servant
62import Servant.Client
63import Servant.Server
64import Servant.QuickCheck
65import Servant.QuickCheck.Internal (serverDoesntSatisfy)
67import Test.Hspec
68import Test.Hspec.Wai
69import Test.Hspec.Wai.Matcher
70```
72We're going to produce different `Spec`s that represent different
73aspects of our application, and we'll ask `hspec` to run all of our different
74`Spec`s. This is a common organizational method for testing modules:
76```haskell
77spec :: Spec
78spec = do
79 businessLogicSpec
80 thirdPartyResourcesSpec
81 servantQuickcheckSpec
82```
84Often, codebases will use `hspec`'s
85[autodiscover pragma](http://hspec.github.io/hspec-discover.html)
86to find all testing modules and `Spec`s inside, but we're going to
87explicitly make a `main` function to run our tests because we have only one
88`spec` defined above:
90```haskell
91main :: IO ()
92main = hspec spec
93```
95## Testing Your Business Logic
97Let's say we have an API that looks something like this:
99```haskell
100data User = User {
101 name :: Text
102 , user_id :: Integer
103 } deriving (Eq, Show, Generic)
105instance FromJSON User
106instance ToJSON User
108type UserApi =
109 -- One endpoint: create a user
110 "user" :> Capture "userId" Integer :> Post '[JSON] User
111```
113A real server would likely use a database to store, retrieve, and validate
114users, but we're going to do something really simple merely to have something
115to test. With that said, here's a sample handler, server, and `Application`
116for the endpoint described above:
118```haskell
119userApp :: Application
120userApp = serve (Proxy :: Proxy UserApi) userServer
122userServer :: Server UserApi
123userServer = createUser
125createUser :: Integer -> Handler User
126createUser userId = do
127 if userId > 5000
128 then pure $ User { name = "some user", user_id = userId }
129 else throwError $ err400 { errBody = "userId is too small" }
130```
132### Strategy 1: Spin Up a Server, Create a Client, Make Some Requests
134One of the benefits of Servant's type-level DSL for describing APIs is that
135once you have provided a type-level description of your API, you can create
136clients, documentation, or other tools for it somewhat magically.
138In this case, we'd like to *test* our server, so we can use `servant-client`
139to create a client, after which we'll run our server, and then make requests
140of it and see how it responds.
142Let's write some tests:
144```haskell
145withUserApp :: IO () -> IO ()
146withUserApp action =
147 -- we can spin up a server in another thread and kill that thread when done
148 -- in an exception-safe way
149 bracket (liftIO $ C.forkIO $ Warp.run 8888 userApp)
150 C.killThread
151 (const action)
154businessLogicSpec :: Spec
155businessLogicSpec =
156 -- `around` will start our Server before the tests and turn it off after
157 around_ withUserApp $ do
158 -- create a test client function
159 let createUser = client (Proxy :: Proxy UserApi)
160 -- create a servant-client ClientEnv
161 baseUrl <- runIO $ parseBaseUrl "http://localhost:8888"
162 manager <- runIO $ newManager defaultManagerSettings
163 let clientEnv = mkClientEnv manager baseUrl
165 -- testing scenarios start here
166 describe "POST /user" $ do
167 it "should create a user with a high enough ID" $ do
168 result <- runClientM (createUser 50001) clientEnv
169 result `shouldBe` (Right $ User { name = "some user", user_id = 50001})
170 it "will it fail with a too-small ID?" $ do
171 result <- runClientM (createUser 4999) clientEnv
172 result `shouldBe` (Right $ User { name = "some user", user_id = 50001})
173```
175### Running These Tests
177Let's run our tests and see what happens:
179```
180$ cabal new-test all
181POST /user
182 should create a user with a high enough ID
183 should fail with a too-small ID FAILED [1]
185 Failures:
187 Testing.lhs:129:7:
188 1) POST /user should fail with a too-small ID
189 expected: Right (User {name = "some user", user_id = 50001})
190 but got: Left (FailureResponse (Response {responseStatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, responseHeaders = fromList [("Transfer-Encoding","chunked"),("Date","Fri, 12 Oct 2018 04:36:22 GMT"),("Server","Warp/3.2.25")], responseHttpVersion = HTTP/1.1, responseBody = "userId is too small"}))
192 To rerun use: --match "/POST /user/should fail with a too-small ID/"
193```
195Hmm. One passed and one failed! It looks like I *was* expecting a success
196response in the second test, but I actually got a failure. We should fix that,
197but first I'd like to introduce `hspec-wai`, which will give us different
198mechanisms for making requests of our application and validating the responses
199we get. We're also going to spin up a fake Elasticsearch server, so that our
200server can think it's talking to a real database.
203## *Mocking* 3rd Party Resources
205Often our web applications will need to make their own web
206requests to other 3rd-party applications. These requests provide a lot
207of opportunity for failure and so we'd like to test that the right
208messages and failure values (in addition to success values) are returned
209from our application.
211### Define the 3rd-Party Resource
213With Servant's type-level API definitions, assuming you've already defined the
214API you want to mock, it's relatively trivial to create a simple server for
215the purposes of running tests. For instance, consider an API server that needs
216to get data out of Elasticsearch. Let's first define the Elasticsearch server
217and client using Servant API descriptions:
219```haskell
220type SearchAPI =
221 -- We're using Aeson's Generic JSON `Value` to make things easier on
222 -- ourselves. We're also representing only one Elasticsearch endpoint:
223 -- get item by id
224 "myIndex" :> "myDocType" :> Capture "docId" Integer :> Get '[JSON] Value
226-- Here's our Servant Client function
227getDocument = client (Proxy :: Proxy SearchAPI)
229-- We can use these helpers when we want to make requests
230-- using our client function
231clientEnv :: Text -> Text -> IO ClientEnv
232clientEnv esHost esPort = do
233 baseUrl <- parseBaseUrl $ unpack $ esHost <> ":" <> esPort
234 manager <- newManager defaultManagerSettings
235 pure $ mkClientEnv manager baseUrl
237runSearchClient :: Text -> Text -> ClientM a -> IO (Either ServantError a)
238runSearchClient esHost esPort = (clientEnv esHost esPort >>=) . runClientM
239```
241### Servant Server Example Using this 3rd-Party Resource
243So we've got an Elasticsearch server and a client to talk to it. Let's now
244build a simple app server that uses this client to retrieve documents. This
245is somewhat contrived, but hopefully it illustrates the typical three-tier
246application architecture.
248One note: we're also going to take advantage of `lens-aeson` here, which may
249look a bit foreign. The gist of it is that we're going to traverse a JSON
250`Value` from Elasticsearch and try to extract some kind of document to
251return.
253Imagine, then, that this is our real server implementation:
255```haskell
256type DocApi =
257 "docs" :> Capture "docId" Integer :> Get '[JSON] Value
259docsApp :: Text -> Text -> Application
260docsApp esHost esPort = serve (Proxy :: Proxy DocApi) $ docServer esHost esPort
262docServer :: Text -> Text -> Server DocApi
263docServer esHost esPort = getDocById esHost esPort
265-- Our Handler tries to get a doc from Elasticsearch and then tries to parse
266-- it. Unfortunately, there's a lot of opportunity for failure in these
267-- actions
268getDocById :: Text -> Text -> Integer -> Handler Value
269getDocById esHost esPort docId = do
270 -- Our Servant Client function returns Either ServantError Value here:
271 docRes <- liftIO $ runSearchClient esHost esPort (getDocument docId)
272 case docRes of
273 Left err -> throwError $ err404 { errBody = "Failed looking up content" }
274 Right value -> do
275 -- we'll either fail to parse our document or we'll return it
276 case value ^? _Object . ix "_source" of
277 Nothing -> throwError $ err400 { errBody = "Failed parsing content" }
278 Just obj -> pure obj
279```
281### Testing Our Backend
283So the above represents our application and is close to a server we may
284actually deploy. How then shall we test this application?
286Ideally, we'd like it to make requests of a *real* Elasticsearch server, but
287we certainly don't want our tests to trigger requests to a live, production
288database. In addition, we don't want to depend on our real Elasticsearch
289server having specific, consistent results for us to test against, because
290that would make our tests flaky (and flaky tests are sometimes described as
291worse than not having tests at all).
293One solution to this is to create a trivial Elasticsearch server as part of
294our testing code. We can do this relatively easily because we already have
295an API definition for it above. With a *real* server, we can then let our own
296application make requests of it and we'll simulate different scenarios in
297order to make sure our application responds the way we expect it to.
299Let's start with some helpers which will allow us to run a testing version
300of our Elasticsearch server in another thread:
302```haskell
303-- | We'll run the Elasticsearch server so we can test behaviors
304withElasticsearch :: IO () -> IO ()
305withElasticsearch action =
306 bracket (liftIO $ C.forkIO $ Warp.run 9999 esTestApp)
307 C.killThread
308 (const action)
310esTestApp :: Application
311esTestApp = serve (Proxy :: Proxy SearchAPI) esTestServer
313esTestServer :: Server SearchAPI
314esTestServer = getESDocument
316-- This is the *mock* handler we're going to use. We create it
317-- here specifically to trigger different behavior in our tests.
318getESDocument :: Integer -> Handler Value
319getESDocument docId
320 -- arbitrary things we can use in our tests to simulate failure:
321 -- we want to trigger different code paths.
322 | docId > 1000 = throwError err500
323 | docId > 500 = pure . Object $ HM.fromList [("bad", String "data")]
324 | otherwise = pure $ Object $ HM.fromList [("_source", Object $ HM.fromList [("a", String "b")])]
325```
327Now, we should be ready to write some tests.
329In this case, we're going to use `hspec-wai`, which will give us a simple way
330to run our application, make requests, and make assertions against the
331responses we receive.
333Hopefully, this will simplify our testing code:
335```haskell
336thirdPartyResourcesSpec :: Spec
337thirdPartyResourcesSpec = around_ withElasticsearch $ do
338 -- we call `with` from `hspec-wai` and pass *real* `Application`
339 with (pure $ docsApp "localhost" "9999") $ do
340 describe "GET /docs" $ do
341 it "should be able to get a document" $
342 -- `get` is a function from hspec-wai`.
343 get "/docs/1" `shouldRespondWith` 200
344 it "should be able to handle connection failures" $
345 get "/docs/1001" `shouldRespondWith` 404
346 it "should be able to handle parsing failures" $
347 get "/docs/501" `shouldRespondWith` 400
348 it "should be able to handle odd HTTP requests" $
349 -- we can also make all kinds of arbitrary custom requests to see how
350 -- our server responds using the `request` function:
351 -- request :: Method -> ByteString -> [Header]
352 -- -> LB.ByteString -> WaiSession SResponse
353 request methodPost "/docs/501" [] "{" `shouldRespondWith` 405
354 it "we can also do more with the Response using hspec-wai's matchers" $
355 -- see also `MatchHeader` and JSON-matching tools as well...
356 get "/docs/1" `shouldRespondWith` 200 { matchBody = MatchBody bodyMatcher }
358bodyMatcher :: [Network.HTTP.Types.Header] -> Body -> Maybe String
359bodyMatcher _ body = case (decode body :: Maybe Value) of
360 -- success in this case means we return `Nothing`
361 Just val | val == (Object $ HM.fromList [("a", String "b")]) -> Nothing
362 _ -> Just "This is how we represent failure: this message will be printed"
363```
365Out of the box, `hspec-wai` provides a lot of useful tools for us to run tests
366against our application. What happens when we run these tests?
368```
369$ cabal new-test all
370...
372GET /docs
373 should be able to get a document
374 should be able to handle connection failures
375 should be able to handle parsing failures
376 should be able to handle odd HTTP requests
377 we can also do more with the Response using hspec-wai's matchers
378```
380Fortunately, they all passed! Let's move to another strategy: whole-API
381testing.
383## Servant Quickcheck
385[`servant-quickcheck`](https://github.com/haskell-servant/servant-quickcheck)
386is a project that allows users to write tests for whole Servant APIs using
387quickcheck-style property-checking mechanisms.
389`servant-quickcheck` is great for asserting API-wide rules, such as "no
390endpoint throws a 500" or "all 301 status codes also come with a Location
391header". The project even comes with a number of predicates that reference
392the [RFCs they originate from](https://github.com/haskell-servant/servant-quickcheck/blob/master/src/Servant/QuickCheck/Internal/Predicates.hs).
394In other words, it's one way to assert that your APIs conform to specs and
395best practices.
397### Quickcheckable API
399Let's make an API and a server to demonstrate how to use `servant-quickcheck`:
401```haskell
402type API = ReqBody '[JSON] String :> Post '[JSON] String
403 :<|> Get '[JSON] Int
404 :<|> BasicAuth "some-realm" () :> Get '[JSON] ()
406api :: Proxy API
407api = Proxy
409server :: IO (Server API)
410server = do
411 mvar <- newMVar ""
412 return $ (\x -> liftIO $ swapMVar mvar x)
413 :<|> (liftIO $ readMVar mvar >>= return . length)
414 :<|> (const $ return ())
415```
417### Using `servant-quickcheck`
419Let's build some tests for our API using `servant-quickcheck`.
421Similar to the above examples, we're going to create `Spec`s, but in this
422case, we'll rely on a number of predicates available from `servant-quickcheck`
423to see if our API server conforms to best practices:
425```haskell
426-- Let's set some QuickCheck values
427args :: Args
428args = defaultArgs { maxSuccess = 500 }
430-- Here's a Servant Context object we'll use
431ctx :: Context '[BasicAuthCheck ()]
432ctx = BasicAuthCheck (const . return $ NoSuchUser) :. EmptyContext
435servantQuickcheckSpec :: Spec
436servantQuickcheckSpec = describe "" $ do
437 it "API demonstrates best practices" $
438 -- `withServerServer` and `withServantServerAndContext` come from `servant-quickcheck`
439 withServantServerAndContext api ctx server $ \burl ->
440 -- `serverSatisfies` and the predicates also come from `servant-quickcheck`
441 serverSatisfies api burl args (unauthorizedContainsWWWAuthenticate
442 <%> not500
443 <%> onlyJsonObjects -- this one isn't true!
444 <%> mempty)
446 it "API doesn't have these things implemented yet" $
447 withServantServerAndContext api ctx server $ \burl -> do
448 serverDoesntSatisfy api burl args (getsHaveCacheControlHeader
449 <%> notAllowedContainsAllowHeader
450 <%> mempty)
451```
453Let's see what happens when we run these tests:
456```
457 API demonstrates best practices FAILED [2]
458+++ OK, passed 500 tests.
459 API doesn't have these things implemented yet
461 src/Servant/QuickCheck/Internal/QuickCheck.hs:143:11:
462 2) Main[339:25] API demonstrates best practices
463 Failed:
464 Just Predicate failed
465 Predicate: onlyJsonObjects
467 Response:
468 Status code: 200
469 Headers: "Transfer-Encoding": "chunked"
470 "Date": "Fri, 12 Oct 2018 04:36:22 GMT"
471 "Server": "Warp/3.2.25"
472 "Content-Type": "application/json;charset=utf-8"
473 Body: ""
475 To rerun use: --match "/Main[339:25]/API demonstrates best practices/"
477Randomized with seed 1046277487
479Finished in 0.4306 seconds
480```
482Hmm. It looks like we *thought* our API only returned JSON objects, which is a
483best practice, but in fact, we *did* have an endpoint that returned an empty
484body, which you can see in the printed response above: `Body: ""`. We should
485consider revising our API to only return top-level JSON Objects in the future!
487### Other Cool Things
489`servant-quickcheck` also has a cool mechanism where you can compare two API
490servers to demonstrate that they respond identically to requests. This may be
491useful if you are planning to rewrite one API in another language or with
492another web framework. You have to specify whether you're looking for
493`jsonEquality` vs regular `ByteString` equality, though.
495## Conclusion
497There are lots of techniques for testing and we only covered a few here.
499Useful libraries such as `hspec-wai` have ways of running Wai `Application`s
500and sending requests to them, while Servant's type-level DSL for defining APIs
501allows us to more easily mock out servers and to derive clients, which will
502only craft valid requests.
504Lastly, if you want a broad overview of where your application fits in with
505regard to best practices, consider using `servant-quickcheck`.
507This program is available as a cabal project
508[here](https://github.com/haskell-servant/servant/tree/master/doc/cookbook/testing).