@@ -1,7 +1,7 @@
1# How To Test Servant Applications
1# How To Test Servant Applications
3Even with a nicely structured API that passes Haskell's strict type checker,
3Even with a nicely structured API that passes Haskell's strict type checker,
4−it's a good idea to write some tests for your application.
4+it's a good idea to write some tests for your application.
6In this recipe we'll work through some common testing strategies and provide
6In this recipe we'll work through some common testing strategies and provide
7examples of utlizing these testing strategies in order to test Servant
7examples of utlizing these testing strategies in order to test Servant
@@ -13,17 +13,27 @@ This recipe uses the following ingredients:
13{-# LANGUAGE OverloadedStrings, TypeFamilies, DataKinds,
13{-# LANGUAGE OverloadedStrings, TypeFamilies, DataKinds,
14 DeriveGeneric, TypeOperators #-}
14 DeriveGeneric, TypeOperators #-}
15import qualified Control.Concurrent as C
15import qualified Control.Concurrent as C
16+import Control.Concurrent.MVar
17+import Control.Exception (bracket)
18+import Control.Lens hiding (Context)
16import Data.Aeson
19import Data.Aeson
17import Data.Aeson.Lens
20import Data.Aeson.Lens
21+import qualified Data.HashMap.Strict as HM
18import Data.Text
22import Data.Text
24+import Network.HTTP.Client hiding (Proxy)
25+import Network.HTTP.Types
19import Network.Wai
26import Network.Wai
20import qualified Network.Wai.Handler.Warp as Warp
27import qualified Network.Wai.Handler.Warp as Warp
21import Servant
29import Servant
22import Servant.Client
30import Servant.Client
23import Servant.Server
31import Servant.Server
32+import Servant.QuickCheck
33+import Servant.QuickCheck.Internal (serverDoesntSatisfy)
25import Test.Hspec
35import Test.Hspec
26−import Test.Hspec.Wai
36+import Test.Hspec.Wai
27import Test.Hspec.Wai.Matcher
37import Test.Hspec.Wai.Matcher
@@ -33,13 +43,13 @@ aspects of our application, and we'll ask `hspec` to run all of our different
35spec :: Spec
45spec :: Spec
37 businessLogicSpec
47 businessLogicSpec
38 thirdPartyResourcesSpec
48 thirdPartyResourcesSpec
39 servantQuickcheckSpec
49 servantQuickcheckSpec
42−Often, codebases will use `hspec`'s
52+Often, codebases will use `hspec`'s
43[autodiscover pragma](http://hspec.github.io/hspec-discover.html)
53[autodiscover pragma](http://hspec.github.io/hspec-discover.html)
44to find all testing modules and `Spec`s inside, but we're going to
54to find all testing modules and `Spec`s inside, but we're going to
45explicitly make a `main` function to run our tests because we have only one
55explicitly make a `main` function to run our tests because we have only one
@@ -55,15 +65,15 @@ main = hspec spec
55Let's say we have an API that looks something like this:
65Let's say we have an API that looks something like this:
58−data User = User {
68+data User = User {
59− name :: Text
69+ name :: Text
60− , user_id :: Integer
70+ , user_id :: Integer
61 } deriving (Eq, Show, Generic)
71 } deriving (Eq, Show, Generic)
63instance FromJSON User
73instance FromJSON User
64instance ToJSON User
74instance ToJSON User
66−type UserApi =
76+type UserApi =
67 -- One endpoint: create a user
77 -- One endpoint: create a user
68 "user" :> Capture "userId" Integer :> Post '[JSON] User
78 "user" :> Capture "userId" Integer :> Post '[JSON] User
@@ -74,11 +84,14 @@ to test. With that said, here's a sample handler for the endpoint described
77−userServer :: Server UserApi Handler
87+userServer :: Server UserApi
78userServer = createUser
88userServer = createUser
79− where creatUser userId
89+
90+createUser :: Integer -> Handler User
91+createUser userId = do
80− | userId > 5000 = pure $ User { name = "some user", user_id = userId }
93+ then pure $ User { name = "some user", user_id = userId }
81− | otherwise = throwError $ err400 { errBody = "userId is too small" }
94+ else throwError $ err400 { errBody = "userId is too small" }
84### Strategy 1: Spin Up a Server, Create a Client, Make Some Requests
97### Strategy 1: Spin Up a Server, Create a Client, Make Some Requests
@@ -97,23 +110,23 @@ Let's write some tests:
97businessLogicSpec :: Spec
110businessLogicSpec :: Spec
98businessLogicSpec = do
111businessLogicSpec = do
99 -- create a test client function
112 -- create a test client function
100− createUser = client (Proxy :: Proxy UserApi)
113+ let createUser = client (Proxy :: Proxy UserApi)
101 -- create a servant-client ClientEnv
114 -- create a servant-client ClientEnv
102− baseUrl <- parseBaseUrl "http://localhost:8888"
115+ baseUrl <- runIO $ parseBaseUrl "http://localhost:8888"
103− manager <- newManager defaultManagerSettings
116+ manager <- runIO $ newManager defaultManagerSettings
104 let clientEnv = mkClientEnv manager baseUrl
117 let clientEnv = mkClientEnv manager baseUrl
106− -- Run the server in another thread.
119+ -- Run the server in another thread (`runIO` is from `hspec`)
107− liftIO $ C.forkIO $ Warp.run 8888 (server userServer)
120+ runIO $ C.forkIO $ Warp.run 8888 (serve (Proxy :: Proxy UserApi) userServer)
109 -- testing scenarios start here
122 -- testing scenarios start here
110 describe "POST /user" $ do
123 describe "POST /user" $ do
111− it "should create a user with a high enough ID" $
124+ it "should create a user with a high enough ID" $ do
112− result <- runClientM clientEnv (createUser 50001)
125+ result <- runClientM (createUser 50001) clientEnv
113− result `shouldEqual` Right $ User { name = "some_user", user_id = "5001 "}
126+ result `shouldBe` (Right $ User { name = "some user", user_id = 50001})
114− it "should fail with a too-small ID" $
127+ it "will it fail with a too-small ID?" $ do
115− result <- runClientM clientEnv (createUser 4999)
128+ result <- runClientM (createUser 4999) clientEnv
116− result `shouldEqual` Right $ User { name = "some_user", user_id = "5001 "}
129+ result `shouldBe` (Right $ User { name = "some user", user_id = 50001})
119### Running These Tests
132### Running These Tests
@@ -122,13 +135,26 @@ Let's run our tests and see what happens:
124$ cabal new-test all
137$ cabal new-test all
139+ should create a user with a high enough ID
140+ should fail with a too-small ID FAILED [1]
145+ 1) POST /user should fail with a too-small ID
146+ expected: Right (User {name = "some user", user_id = 50001})
147+ but got: Left (FailureResponse (Response {responseStatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, responseHeaders = fromList [("Transfer-Encoding","chunked"),("Date","Fri, 12 Oct 2018 04:36:22 GMT"),("Server","Warp/3.2.25")], responseHttpVersion = HTTP/1.1, responseBody = "userId is too small"}))
149+ To rerun use: --match "/POST /user/should fail with a too-small ID/"
127−Great: we passed! Servers obviously get more complex, though, and we may not
152+Hmm. One passed and one failed! It looks like I *was* expecting a success
128−wish to create a whole suite of clients for our server every time. In our next
153+response in the second test, but I actually got a failure. We should fix that,
129−scenario we're going to mock out a 3rd-party resource that our server talks to
154+but first I'd like to introduce `hspec-wai`, which will give us different
130−and we're going to be using `hspec-wai` to run our `Application` instance and
155+mechanisms for making requests of our application and validating the responses
131−to make requests.
156+we get.
133## *Mocking* 3rd Party Resources
159## *Mocking* 3rd Party Resources
@@ -147,7 +173,7 @@ to get data out of Elasticsearch. Let's first define the Elasticsearch server
147and client using Servant API descriptions:
173and client using Servant API descriptions:
149```haskell
175```haskell
150−type SearchAPI =
176+type SearchAPI =
151 -- We're using Aeson's Generic JSON `Value` to make things easier on
177 -- We're using Aeson's Generic JSON `Value` to make things easier on
152 -- ourselves. We're also representing only one Elasticsearch endpoint:
178 -- ourselves. We're also representing only one Elasticsearch endpoint:
153 -- get item by id
179 -- get item by id
@@ -156,11 +182,11 @@ type SearchAPI =
156-- Here's our Servant Client function
182-- Here's our Servant Client function
157getDocument = client (Proxy :: Proxy SearchAPI)
183getDocument = client (Proxy :: Proxy SearchAPI)
159−-- We can use these helpers when we want to make requests
185+-- We can use these helpers when we want to make requests
160-- using our client function
186-- using our client function
161clientEnv :: Text -> Text -> IO ClientEnv
187clientEnv :: Text -> Text -> IO ClientEnv
162clientEnv esHost esPort = do
188clientEnv esHost esPort = do
163− baseUrl <- parseBaseUrl $ T.unpack $ esHost <> ":" <> esPort
189+ baseUrl <- parseBaseUrl $ unpack $ esHost <> ":" <> esPort
164 manager <- newManager defaultManagerSettings
190 manager <- newManager defaultManagerSettings
165 pure $ mkClientEnv manager baseUrl
191 pure $ mkClientEnv manager baseUrl
@@ -170,22 +196,21 @@ runSearchClient esHost esPort = (clientEnv esHost esPort >>=) . runClientM
171### Servant Server Example Using this 3rd-Party Resource
197### Servant Server Example Using this 3rd-Party Resource
173−So we've got an Elasticsearch server and a client to talk to it with one
199+So we've got an Elasticsearch server and a client to talk to it. Let's now
174−function that retrieves a document by its id. Let's now build a simple app
175−server that uses this client to retrieve documents. This is somewhat
200+build a simple app server that uses this client to retrieve documents. This
176−contrived, but hopefully it illustrates the typical three-tier application
201+is somewhat contrived, but hopefully it illustrates the typical three-tier
177−architecture.
202+application architecture.
179One note: we're also going to take advantage of `aeson-lens` here, which may
204One note: we're also going to take advantage of `aeson-lens` here, which may
180−look a bit foreign. The gist of it is that we're going to traverse a JSON
205+look a bit foreign. The gist of it is that we're going to traverse a JSON
181`Value` from Elasticsearch and try to extract some kind of document to
206`Value` from Elasticsearch and try to extract some kind of document to
184```haskell
209```haskell
185−type DocApi =
210+type DocApi =
186 "docs" :> Capture "docId" Integer :> Get '[JSON] Value
211 "docs" :> Capture "docId" Integer :> Get '[JSON] Value
188−docServer :: Text -> Text -> Server DocApi Handler
213+docServer :: Text -> Text -> Server DocApi
189docServer esHost esPort = getDocById esHost esPort
214docServer esHost esPort = getDocById esHost esPort
191-- Our Handler tries to get a doc from Elasticsearch and then tries to parse
216-- Our Handler tries to get a doc from Elasticsearch and then tries to parse
@@ -206,9 +231,11 @@ getDocById esHost esPort docId = do
207### Testing Our Backend
232### Testing Our Backend
209−So the above represents our application. How shall we test this application?
234+So the above represents our application and is close to a server we may
235+actually deploy. How shall we test this application?
210−Ideally, we'd like it to make requests of a real Elasticsearch server, but we
237+Ideally, we'd like it to make requests of a *real* Elasticsearch server, but
211−certainly don't want our tests to trigger requests to a live, production
238+we certainly don't want our tests to trigger requests to a live, production
212database. In addition, we don't want to depend on our real Elasticsearch
239database. In addition, we don't want to depend on our real Elasticsearch
213server having specific, consistent results for us to test against, because
240server having specific, consistent results for us to test against, because
214that would make our tests flaky (and flaky tests are sometimes described as
241that would make our tests flaky (and flaky tests are sometimes described as
@@ -234,30 +261,34 @@ withElasticsearch action =
234esTestApp :: Application
261esTestApp :: Application
235esTestApp = serve (Proxy :: Proxy SearchAPI) esTestServer
262esTestApp = serve (Proxy :: Proxy SearchAPI) esTestServer
237−esTestServer :: Server SearchAPI Handler
264+esTestServer :: Server SearchAPI
238esTestServer = getESDocument
265esTestServer = getESDocument
240getESDocument :: Integer -> Handler Value
267getESDocument :: Integer -> Handler Value
241getESDocument docId
268getESDocument docId
242 -- arbitrary things we can trigger in our tests to check for failure
269 -- arbitrary things we can trigger in our tests to check for failure
243 -- We want to try to trigger different code paths
270 -- We want to try to trigger different code paths
244− | docId > 1000 = pure . Left $ ConnectionError "Bad connection!"
271+ | docId > 1000 = throwError err500
245 | docId > 500 = pure . Object $ HM.fromList [("bad", String "data")]
272 | docId > 500 = pure . Object $ HM.fromList [("bad", String "data")]
246− | otherwise = pure $ Object $ HM.fromList [("_source", defaultDocument)]
273+ | otherwise = pure $ Object $ HM.fromList [("_source", Object $ HM.fromList [("a", String "b")])]
249−Now, we should be ready to write some tests. As mentioned above we're going
276+Now, we should be ready to write some tests.
250−to use `hspec-wai` in this example to make test requests and to run our own
277+
278+In this case, we're going to use `hspec-wai`, which will give us a simple way
279+to run our application, make requests, and make assertions against the
280+responses we receive.
251−application. This should simplify our testing code somewhat:
282+Hopefully, this will simplify our testing code:
253```haskell
284```haskell
254thirdPartyResourcesSpec :: Spec
285thirdPartyResourcesSpec :: Spec
255thirdPartyResourcesSpec = around_ withElasticsearch $ do
286thirdPartyResourcesSpec = around_ withElasticsearch $ do
256 -- we call `with` and pass our servant-server `Application`
287 -- we call `with` and pass our servant-server `Application`
257− with (pure $ serve $ docServer "localhost" "9999") $ do
288+ with (pure $ serve (Proxy :: Proxy DocApi) $ docServer "localhost" "9999") $ do
258 describe "GET /docs" $ do
289 describe "GET /docs" $ do
259 it "should be able to get a document" $
290 it "should be able to get a document" $
260− -- `get` is a function from hspec-wai`.
291+ -- `get` is a function from hspec-wai`.
261 get "/docs/1" `shouldRespondWith` 200
292 get "/docs/1" `shouldRespondWith` 200
262 it "should be able to handle connection failures" $
293 it "should be able to handle connection failures" $
263 -- We can also make custom HTTP requests with the `request` function
294 -- We can also make custom HTTP requests with the `request` function
@@ -267,11 +298,36 @@ thirdPartyResourcesSpec = around_ withElasticsearch $ do
267 it "should be able to handle odd HTTP requests" $
298 it "should be able to handle odd HTTP requests" $
268 -- we can also make all kinds of arbitrary custom requests to see how
299 -- we can also make all kinds of arbitrary custom requests to see how
269 -- our server responds using the `request` function:
300 -- our server responds using the `request` function:
270− -- request :: Method -> ByteString -> [Header]
301+ -- request :: Method -> ByteString -> [Header]
271 -- -> LB.ByteString -> WaiSession SResponse
302 -- -> LB.ByteString -> WaiSession SResponse
272− request methodPost "/docs/501" [] "{" `shouldRespondWith` 415
303+ request methodPost "/docs/501" [] "{" `shouldRespondWith` 405
304+ it "we can also do more with the Response using hspec-wai's matchers" $
305+ -- see also `MatchHeader` and JSON-matching tools as well...
306+ get "/docs/1" `shouldRespondWith` 200 { matchBody = MatchBody bodyMatcher }
308+bodyMatcher :: [Network.HTTP.Types.Header] -> Body -> Maybe String
309+bodyMatcher _ body = case (decode body :: Maybe Value) of
310+ -- success in this case means we return `Nothing`
311+ Just val | val == (Object $ HM.fromList [("a", String "b")]) -> Nothing
312+ _ -> Just "This is how we represent failure: this message will be printed"
316+What happens when we run these tests?
323+ should be able to get a document
324+ should be able to handle connection failures
325+ should be able to handle parsing failures
326+ should be able to handle odd HTTP requests
327+ we can also do more with the Response using hspec-wai's matchers
330+Fortunately, they all passed!
276## Servant Quickcheck
332## Servant Quickcheck
@@ -284,6 +340,8 @@ endpoint throws a 500" or "all 301 status codes also come with a Location
284header". The project even comes with a number of predicates that reference
340header". The project even comes with a number of predicates that reference
285the [RFCs they originate from](https://github.com/haskell-servant/servant-quickcheck/blob/master/src/Servant/QuickCheck/Internal/Predicates.hs).
341the [RFCs they originate from](https://github.com/haskell-servant/servant-quickcheck/blob/master/src/Servant/QuickCheck/Internal/Predicates.hs).
343+Thus, it's one way to assert that your APIs conform to specs and best
288### Quickcheckable API
346### Quickcheckable API
@@ -301,31 +359,94 @@ server :: IO (Server API)
301server = do
359server = do
302 mvar <- newMVar ""
360 mvar <- newMVar ""
303 return $ (\x -> liftIO $ swapMVar mvar x)
361 return $ (\x -> liftIO $ swapMVar mvar x)
304− :<|> (liftIO $ readMVar mvar >>= return . length)
362+ :<|> (liftIO $ readMVar mvar >>= return . Prelude.length)
305 :<|> (const $ return ())
363 :<|> (const $ return ())
308### Using `servant-quickcheck`
366### Using `servant-quickcheck`
310−`servant-quickcheck` also has a cool mechanism where you can compare two API
368+Let's build some tests for our API using `servant-quickcheck`.
311−servers to demonstrate that they respond identically to requests. This may be
369+
312−useful if you are planning to rewrite one API in another language or with
370+Similar to the above examples, we're going to create `Spec`s, but in this
313−another web framework. You have to specify whether you're looking for
371+case, we'll rely on a number of predicates available from `servant-quickcheck`
314−`jsonEquality` vs regular `ByteString` equality, though:
372+to see if our API server conforms to best practices:
316```haskell
374```haskell
375+-- Let's set some QuickCheck values
377+args = defaultArgs { maxSuccess = 500 }
379+-- Here's a Servant Context object we'll use
380+ctx :: Context '[BasicAuthCheck ()]
381+ctx = BasicAuthCheck (const . return $ NoSuchUser) :. EmptyContext
317servantQuickcheckSpec :: Spec
384servantQuickcheckSpec :: Spec
318servantQuickcheckSpec = describe "" $ do
385servantQuickcheckSpec = describe "" $ do
319 it "API demonstrates best practices" $
386 it "API demonstrates best practices" $
320− withServantServer api server $ \burl ->
387+ -- `withServerServer` and `withServantServerAndContext` come from `servant-quickcheck`
388+ withServantServerAndContext api ctx server $ \burl ->
389+ -- `serverSatisfies` and the predicates also come from `servant-quickcheck`
321 serverSatisfies api burl args (unauthorizedContainsWWWAuthenticate
390 serverSatisfies api burl args (unauthorizedContainsWWWAuthenticate
322 <%> not500
391 <%> not500
323 <%> onlyJsonObjects
392 <%> onlyJsonObjects
324 <%> mempty)
393 <%> mempty)
326 it "API doesn't have these things implemented yet" $
395 it "API doesn't have these things implemented yet" $
327− withServantServer api server $ \burl -> do
396+ withServantServerAndContext api ctx server $ \burl -> do
328 serverDoesntSatisfy api burl args (getsHaveCacheControlHeader
397 serverDoesntSatisfy api burl args (getsHaveCacheControlHeader
329 <%> notAllowedContainsAllowHeader
398 <%> notAllowedContainsAllowHeader
330 <%> mempty)
399 <%> mempty)
402+Let's see what happens when we run these tests:
406+ API demonstrates best practices FAILED [2]
407++++ OK, passed 500 tests.
408+ API doesn't have these things implemented yet
410+ src/Servant/QuickCheck/Internal/QuickCheck.hs:143:11:
411+ 2) Main[339:25] API demonstrates best practices
413+ Just Predicate failed
414+ Predicate: onlyJsonObjects
418+ Headers: "Transfer-Encoding": "chunked"
419+ "Date": "Fri, 12 Oct 2018 04:36:22 GMT"
420+ "Server": "Warp/3.2.25"
421+ "Content-Type": "application/json;charset=utf-8"
424+ To rerun use: --match "/Main[339:25]/API demonstrates best practices/"
426+Randomized with seed 1046277487
428+Finished in 0.4306 seconds
431+Hmm. It looks like we *thought* our API only return JSON objects, which is a
432+best practice, but in fact, we *did* have an endpoint that returned an empty
433+body, which you can see in the printed response above: `Body: ""`. We should
434+consider revising our API to only return top-level JSON Objects in the future!
436+### Other Cool Things
438+`servant-quickcheck` also has a cool mechanism where you can compare two API
439+servers to demonstrate that they respond identically to requests. This may be
440+useful if you are planning to rewrite one API in another language or with
441+another web framework. You have to specify whether you're looking for
442+`jsonEquality` vs regular `ByteString` equality, though.
446+There are lots of techniques for testing and we only covered a few here.
448+Useful libraries such as `hspec-wai` have ways of testing Wai `Application`s
449+and sending requests to them, while Servant's type-level DSL for defining APIs
450+allows us to more easily mock out servers. Lastly, if you want a broad
451+overview of where your application fits in with regard to best practices,
452+consider using `servant-quickcheck`.