@@ -1,7 +1,7 @@
11# How To Test Servant Applications
22
33Even with a nicely structured API that passes Haskell's strict type checker,
4−it's a good idea to write some tests for your application.
4+it's a good idea to write some tests for your application.
55
66In this recipe we'll work through some common testing strategies and provide
77examples of utlizing these testing strategies in order to test Servant
@@ -13,17 +13,27 @@ This recipe uses the following ingredients:
1313{-# LANGUAGE OverloadedStrings, TypeFamilies, DataKinds,
1414 DeriveGeneric, TypeOperators #-}
1515import qualified Control.Concurrent as C
16+import Control.Concurrent.MVar
17+import Control.Exception (bracket)
18+import Control.Lens hiding (Context)
1619import Data.Aeson
1720import Data.Aeson.Lens
21+import qualified Data.HashMap.Strict as HM
1822import Data.Text
23+import GHC.Generics
24+import Network.HTTP.Client hiding (Proxy)
25+import Network.HTTP.Types
1926import Network.Wai
2027import qualified Network.Wai.Handler.Warp as Warp
28+
2129import Servant
2230import Servant.Client
2331import Servant.Server
32+import Servant.QuickCheck
33+import Servant.QuickCheck.Internal (serverDoesntSatisfy)
2434
2535import Test.Hspec
26−import Test.Hspec.Wai
36+import Test.Hspec.Wai
2737import Test.Hspec.Wai.Matcher
2838```
2939
@@ -33,13 +43,13 @@ aspects of our application, and we'll ask `hspec` to run all of our different
3343
3444```haskell
3545spec :: Spec
36−spec =
46+spec = do
3747 businessLogicSpec
3848 thirdPartyResourcesSpec
3949 servantQuickcheckSpec
4050```
4151
42−Often, codebases will use `hspec`'s
52+Often, codebases will use `hspec`'s
4353[autodiscover pragma](http://hspec.github.io/hspec-discover.html)
4454to find all testing modules and `Spec`s inside, but we're going to
4555explicitly make a `main` function to run our tests because we have only one
@@ -55,15 +65,15 @@ main = hspec spec
5565Let's say we have an API that looks something like this:
5666
5767```haskell
58−data User = User {
59− name :: Text
60− , user_id :: Integer
68+data User = User {
69+ name :: Text
70+ , user_id :: Integer
6171 } deriving (Eq, Show, Generic)
6272
6373instance FromJSON User
6474instance ToJSON User
6575
66−type UserApi =
76+type UserApi =
6777 -- One endpoint: create a user
6878 "user" :> Capture "userId" Integer :> Post '[JSON] User
6979```
@@ -74,11 +84,14 @@ to test. With that said, here's a sample handler for the endpoint described
7484above:
7585
7686```haskell
77−userServer :: Server UserApi Handler
87+userServer :: Server UserApi
7888userServer = createUser
79− where creatUser userId
80− | userId > 5000 = pure $ User { name = "some user", user_id = userId }
81− | otherwise = throwError $ err400 { errBody = "userId is too small" }
89+
90+createUser :: Integer -> Handler User
91+createUser userId = do
92+ if userId > 5000
93+ then pure $ User { name = "some user", user_id = userId }
94+ else throwError $ err400 { errBody = "userId is too small" }
8295```
8396
8497### Strategy 1: Spin Up a Server, Create a Client, Make Some Requests
@@ -97,23 +110,23 @@ Let's write some tests:
97110businessLogicSpec :: Spec
98111businessLogicSpec = do
99112 -- create a test client function
100− createUser = client (Proxy :: Proxy UserApi)
113+ let createUser = client (Proxy :: Proxy UserApi)
101114 -- create a servant-client ClientEnv
102− baseUrl <- parseBaseUrl "http://localhost:8888"
103− manager <- newManager defaultManagerSettings
115+ baseUrl <- runIO $ parseBaseUrl "http://localhost:8888"
116+ manager <- runIO $ newManager defaultManagerSettings
104117 let clientEnv = mkClientEnv manager baseUrl
105118
106− -- Run the server in another thread.
107− liftIO $ C.forkIO $ Warp.run 8888 (server userServer)
119+ -- Run the server in another thread (`runIO` is from `hspec`)
120+ runIO $ C.forkIO $ Warp.run 8888 (serve (Proxy :: Proxy UserApi) userServer)
108121
109122 -- testing scenarios start here
110123 describe "POST /user" $ do
111− it "should create a user with a high enough ID" $
112− result <- runClientM clientEnv (createUser 50001)
113− result `shouldEqual` Right $ User { name = "some_user", user_id = "5001 "}
114− it "should fail with a too-small ID" $
115− result <- runClientM clientEnv (createUser 4999)
116− result `shouldEqual` Right $ User { name = "some_user", user_id = "5001 "}
124+ it "should create a user with a high enough ID" $ do
125+ result <- runClientM (createUser 50001) clientEnv
126+ result `shouldBe` (Right $ User { name = "some user", user_id = 50001})
127+ it "will it fail with a too-small ID?" $ do
128+ result <- runClientM (createUser 4999) clientEnv
129+ result `shouldBe` (Right $ User { name = "some user", user_id = 50001})
117130```
118131
119132### Running These Tests
@@ -122,13 +135,26 @@ Let's run our tests and see what happens:
122135
123136```
124137$ cabal new-test all
138+POST /user
139+ should create a user with a high enough ID
140+ should fail with a too-small ID FAILED [1]
141+
142+ Failures:
143+
144+ Testing.lhs:129:7:
145+ 1) POST /user should fail with a too-small ID
146+ expected: Right (User {name = "some user", user_id = 50001})
147+ but got: Left (FailureResponse (Response {responseStatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, responseHeaders = fromList [("Transfer-Encoding","chunked"),("Date","Fri, 12 Oct 2018 04:36:22 GMT"),("Server","Warp/3.2.25")], responseHttpVersion = HTTP/1.1, responseBody = "userId is too small"}))
148+
149+ To rerun use: --match "/POST /user/should fail with a too-small ID/"
125150```
126151
127−Great: we passed! Servers obviously get more complex, though, and we may not
128−wish to create a whole suite of clients for our server every time. In our next
129−scenario we're going to mock out a 3rd-party resource that our server talks to
130−and we're going to be using `hspec-wai` to run our `Application` instance and
131−to make requests.
152+Hmm. One passed and one failed! It looks like I *was* expecting a success
153+response in the second test, but I actually got a failure. We should fix that,
154+but first I'd like to introduce `hspec-wai`, which will give us different
155+mechanisms for making requests of our application and validating the responses
156+we get.
157+
132158
133159## *Mocking* 3rd Party Resources
134160
@@ -147,7 +173,7 @@ to get data out of Elasticsearch. Let's first define the Elasticsearch server
147173and client using Servant API descriptions:
148174
149175```haskell
150−type SearchAPI =
176+type SearchAPI =
151177 -- We're using Aeson's Generic JSON `Value` to make things easier on
152178 -- ourselves. We're also representing only one Elasticsearch endpoint:
153179 -- get item by id
@@ -156,11 +182,11 @@ type SearchAPI =
156182-- Here's our Servant Client function
157183getDocument = client (Proxy :: Proxy SearchAPI)
158184
159−-- We can use these helpers when we want to make requests
185+-- We can use these helpers when we want to make requests
160186-- using our client function
161187clientEnv :: Text -> Text -> IO ClientEnv
162188clientEnv esHost esPort = do
163− baseUrl <- parseBaseUrl $ T.unpack $ esHost <> ":" <> esPort
189+ baseUrl <- parseBaseUrl $ unpack $ esHost <> ":" <> esPort
164190 manager <- newManager defaultManagerSettings
165191 pure $ mkClientEnv manager baseUrl
166192
@@ -170,22 +196,21 @@ runSearchClient esHost esPort = (clientEnv esHost esPort >>=) . runClientM
170196
171197### Servant Server Example Using this 3rd-Party Resource
172198
173−So we've got an Elasticsearch server and a client to talk to it with one
174−function that retrieves a document by its id. Let's now build a simple app
175−server that uses this client to retrieve documents. This is somewhat
176−contrived, but hopefully it illustrates the typical three-tier application
177−architecture.
199+So we've got an Elasticsearch server and a client to talk to it. Let's now
200+build a simple app server that uses this client to retrieve documents. This
201+is somewhat contrived, but hopefully it illustrates the typical three-tier
202+application architecture.
178203
179204One note: we're also going to take advantage of `aeson-lens` here, which may
180−look a bit foreign. The gist of it is that we're going to traverse a JSON
205+look a bit foreign. The gist of it is that we're going to traverse a JSON
181206`Value` from Elasticsearch and try to extract some kind of document to
182207return.
183208
184209```haskell
185−type DocApi =
210+type DocApi =
186211 "docs" :> Capture "docId" Integer :> Get '[JSON] Value
187212
188−docServer :: Text -> Text -> Server DocApi Handler
213+docServer :: Text -> Text -> Server DocApi
189214docServer esHost esPort = getDocById esHost esPort
190215
191216-- Our Handler tries to get a doc from Elasticsearch and then tries to parse
@@ -206,9 +231,11 @@ getDocById esHost esPort docId = do
206231
207232### Testing Our Backend
208233
209−So the above represents our application. How shall we test this application?
210−Ideally, we'd like it to make requests of a real Elasticsearch server, but we
211−certainly don't want our tests to trigger requests to a live, production
234+So the above represents our application and is close to a server we may
235+actually deploy. How shall we test this application?
236+
237+Ideally, we'd like it to make requests of a *real* Elasticsearch server, but
238+we certainly don't want our tests to trigger requests to a live, production
212239database. In addition, we don't want to depend on our real Elasticsearch
213240server having specific, consistent results for us to test against, because
214241that would make our tests flaky (and flaky tests are sometimes described as
@@ -234,30 +261,34 @@ withElasticsearch action =
234261esTestApp :: Application
235262esTestApp = serve (Proxy :: Proxy SearchAPI) esTestServer
236263
237−esTestServer :: Server SearchAPI Handler
264+esTestServer :: Server SearchAPI
238265esTestServer = getESDocument
239266
240267getESDocument :: Integer -> Handler Value
241268getESDocument docId
242269 -- arbitrary things we can trigger in our tests to check for failure
243270 -- We want to try to trigger different code paths
244− | docId > 1000 = pure . Left $ ConnectionError "Bad connection!"
271+ | docId > 1000 = throwError err500
245272 | docId > 500 = pure . Object $ HM.fromList [("bad", String "data")]
246− | otherwise = pure $ Object $ HM.fromList [("_source", defaultDocument)]
273+ | otherwise = pure $ Object $ HM.fromList [("_source", Object $ HM.fromList [("a", String "b")])]
247274```
248275
249−Now, we should be ready to write some tests. As mentioned above we're going
250−to use `hspec-wai` in this example to make test requests and to run our own
251−application. This should simplify our testing code somewhat:
276+Now, we should be ready to write some tests.
277+
278+In this case, we're going to use `hspec-wai`, which will give us a simple way
279+to run our application, make requests, and make assertions against the
280+responses we receive.
281+
282+Hopefully, this will simplify our testing code:
252283
253284```haskell
254285thirdPartyResourcesSpec :: Spec
255286thirdPartyResourcesSpec = around_ withElasticsearch $ do
256287 -- we call `with` and pass our servant-server `Application`
257− with (pure $ serve $ docServer "localhost" "9999") $ do
288+ with (pure $ serve (Proxy :: Proxy DocApi) $ docServer "localhost" "9999") $ do
258289 describe "GET /docs" $ do
259290 it "should be able to get a document" $
260− -- `get` is a function from hspec-wai`.
291+ -- `get` is a function from hspec-wai`.
261292 get "/docs/1" `shouldRespondWith` 200
262293 it "should be able to handle connection failures" $
263294 -- We can also make custom HTTP requests with the `request` function
@@ -267,11 +298,36 @@ thirdPartyResourcesSpec = around_ withElasticsearch $ do
267298 it "should be able to handle odd HTTP requests" $
268299 -- we can also make all kinds of arbitrary custom requests to see how
269300 -- our server responds using the `request` function:
270− -- request :: Method -> ByteString -> [Header]
301+ -- request :: Method -> ByteString -> [Header]
271302 -- -> LB.ByteString -> WaiSession SResponse
272− request methodPost "/docs/501" [] "{" `shouldRespondWith` 415
303+ request methodPost "/docs/501" [] "{" `shouldRespondWith` 405
304+ it "we can also do more with the Response using hspec-wai's matchers" $
305+ -- see also `MatchHeader` and JSON-matching tools as well...
306+ get "/docs/1" `shouldRespondWith` 200 { matchBody = MatchBody bodyMatcher }
307+
308+bodyMatcher :: [Network.HTTP.Types.Header] -> Body -> Maybe String
309+bodyMatcher _ body = case (decode body :: Maybe Value) of
310+ -- success in this case means we return `Nothing`
311+ Just val | val == (Object $ HM.fromList [("a", String "b")]) -> Nothing
312+ _ -> Just "This is how we represent failure: this message will be printed"
313+
273314```
274315
316+What happens when we run these tests?
317+
318+```
319+$ cabal new-test all
320+...
321+
322+GET /docs
323+ should be able to get a document
324+ should be able to handle connection failures
325+ should be able to handle parsing failures
326+ should be able to handle odd HTTP requests
327+ we can also do more with the Response using hspec-wai's matchers
328+```
329+
330+Fortunately, they all passed!
275331
276332## Servant Quickcheck
277333
@@ -284,6 +340,8 @@ endpoint throws a 500" or "all 301 status codes also come with a Location
284340header". The project even comes with a number of predicates that reference
285341the [RFCs they originate from](https://github.com/haskell-servant/servant-quickcheck/blob/master/src/Servant/QuickCheck/Internal/Predicates.hs).
286342
343+Thus, it's one way to assert that your APIs conform to specs and best
344+practices.
287345
288346### Quickcheckable API
289347
@@ -301,31 +359,94 @@ server :: IO (Server API)
301359server = do
302360 mvar <- newMVar ""
303361 return $ (\x -> liftIO $ swapMVar mvar x)
304− :<|> (liftIO $ readMVar mvar >>= return . length)
362+ :<|> (liftIO $ readMVar mvar >>= return . Prelude.length)
305363 :<|> (const $ return ())
306364```
307365
308366### Using `servant-quickcheck`
309367
310−`servant-quickcheck` also has a cool mechanism where you can compare two API
311−servers to demonstrate that they respond identically to requests. This may be
312−useful if you are planning to rewrite one API in another language or with
313−another web framework. You have to specify whether you're looking for
314−`jsonEquality` vs regular `ByteString` equality, though:
368+Let's build some tests for our API using `servant-quickcheck`.
369+
370+Similar to the above examples, we're going to create `Spec`s, but in this
371+case, we'll rely on a number of predicates available from `servant-quickcheck`
372+to see if our API server conforms to best practices:
315373
316374```haskell
375+-- Let's set some QuickCheck values
376+args :: Args
377+args = defaultArgs { maxSuccess = 500 }
378+
379+-- Here's a Servant Context object we'll use
380+ctx :: Context '[BasicAuthCheck ()]
381+ctx = BasicAuthCheck (const . return $ NoSuchUser) :. EmptyContext
382+
383+
317384servantQuickcheckSpec :: Spec
318385servantQuickcheckSpec = describe "" $ do
319386 it "API demonstrates best practices" $
320− withServantServer api server $ \burl ->
387+ -- `withServerServer` and `withServantServerAndContext` come from `servant-quickcheck`
388+ withServantServerAndContext api ctx server $ \burl ->
389+ -- `serverSatisfies` and the predicates also come from `servant-quickcheck`
321390 serverSatisfies api burl args (unauthorizedContainsWWWAuthenticate
322391 <%> not500
323392 <%> onlyJsonObjects
324393 <%> mempty)
325394
326395 it "API doesn't have these things implemented yet" $
327− withServantServer api server $ \burl -> do
396+ withServantServerAndContext api ctx server $ \burl -> do
328397 serverDoesntSatisfy api burl args (getsHaveCacheControlHeader
329398 <%> notAllowedContainsAllowHeader
330399 <%> mempty)
331−```
400+```
401+
402+Let's see what happens when we run these tests:
403+
404+
405+```
406+ API demonstrates best practices FAILED [2]
407++++ OK, passed 500 tests.
408+ API doesn't have these things implemented yet
409+
410+ src/Servant/QuickCheck/Internal/QuickCheck.hs:143:11:
411+ 2) Main[339:25] API demonstrates best practices
412+ Failed:
413+ Just Predicate failed
414+ Predicate: onlyJsonObjects
415+
416+ Response:
417+ Status code: 200
418+ Headers: "Transfer-Encoding": "chunked"
419+ "Date": "Fri, 12 Oct 2018 04:36:22 GMT"
420+ "Server": "Warp/3.2.25"
421+ "Content-Type": "application/json;charset=utf-8"
422+ Body: ""
423+
424+ To rerun use: --match "/Main[339:25]/API demonstrates best practices/"
425+
426+Randomized with seed 1046277487
427+
428+Finished in 0.4306 seconds
429+```
430+
431+Hmm. It looks like we *thought* our API only return JSON objects, which is a
432+best practice, but in fact, we *did* have an endpoint that returned an empty
433+body, which you can see in the printed response above: `Body: ""`. We should
434+consider revising our API to only return top-level JSON Objects in the future!
435+
436+### Other Cool Things
437+
438+`servant-quickcheck` also has a cool mechanism where you can compare two API
439+servers to demonstrate that they respond identically to requests. This may be
440+useful if you are planning to rewrite one API in another language or with
441+another web framework. You have to specify whether you're looking for
442+`jsonEquality` vs regular `ByteString` equality, though.
443+
444+## Conclusion
445+
446+There are lots of techniques for testing and we only covered a few here.
447+
448+Useful libraries such as `hspec-wai` have ways of testing Wai `Application`s
449+and sending requests to them, while Servant's type-level DSL for defining APIs
450+allows us to more easily mock out servers. Lastly, if you want a broad
451+overview of where your application fits in with regard to best practices,
452+consider using `servant-quickcheck`.