hurl/docs/asserting-response.md

Asserting Response

Asserts

Asserts are used to test various properties of an HTTP response. Asserts can be implicits (such as version, status, headers) or explicit within an [Asserts] section.

GET https://api/example.org/cats

HTTP 200
Content-Type: application/json; charset=utf-8      # Implicit assert on Content-Type Hedaer
[Asserts]                                          # Explicit asserts section 
bytes count == 120
header "Content-Type" contains "utf-8"
jsonpath "$.cats" count == 49
jsonpath "$.cats[0].name" == "Felix"
jsonpath "$.cats[0].lives" == 9

Implicit asserts

Version - Status

Expected protocol version and status code of the HTTP response.

Protocol version is one of HTTP/1.0, HTTP/1.1, HTTP/2 or HTTP; HTTP describes any version. Note that there are no status text following the status code.

GET https://example.org/404.html
HTTP 404

Wildcard keywords HTTP and * can be used to disable tests on protocol version and status:

GET https://example.org/api/pets

HTTP *
# Check that response status code is > 400 and <= 500
[Asserts]
status > 400
status <= 500

While HTTP/1.0, HTTP/1.1 and HTTP/2 explicitly check HTTP version:

# Check that our server responds with HTTP/2
GET https://example.org/api/pets
HTTP/2 200 

Headers

Optional list of the expected HTTP response headers that must be in the received response.

A header consists of a name, followed by a : and a value.

For each expected header, the received response headers are checked. If the received header is not equal to the expected, or not present, an error is raised. Note that the expected headers list is not fully descriptive: headers present in the response and not in the expected list doesn't raise error.

# Check that user toto is redirected to home after login.
POST https://example.org/login
[FormParams]
user: toto
password: 12345678

HTTP 302
Location: https://example.org/home

Quotes in the header value are part of the value itself.

This is used by the ETag Header

ETag: W/"<etag_value>"
ETag: "<etag_value>"

Testing duplicated headers is also possible.

For example with the Set-Cookie header:

Set-Cookie: theme=light
Set-Cookie: sessionToken=abc123; Expires=Wed, 09 Jun 2021 10:18:14 GMT

You can either test the two header values:

GET https://example.org/index.html
Host: example.net

HTTP 200
Set-Cookie: theme=light
Set-Cookie: sessionToken=abc123; Expires=Wed, 09 Jun 2021 10:18:14 GMT

Or only one:

GET https://example.org/index.html 
Host: example.net

HTTP 200
Set-Cookie: theme=light

If you want to test specifically the number of headers returned for a given header name, or if you want to test header value with predicates (like startsWith, contains, exists) you can use the explicit header assert.

Explicit asserts

Optional list of assertions on the HTTP response within an [Asserts] section. Assertions can describe checks on status code, on the received body (or part of it) and on response headers.

Structure of an assert:

jsonpath "$.book"query containspredicate type "Dune"predicate value

bodyquery matchespredicate type /\d{4}-\d{2}-\d{2}/predicate value

An assert consists of a query followed by a predicate. The format of the query is shared with captures, and can be one of :

• status
• header
• url
• cookie
• body
• bytes
• xpath
• jsonpath
• regex
• sha256
• md5
• variable
• duration
• certificate

Queries are used to extract data from the HTTP response. Queries, in asserts and in captures, can be refined with filters, like count to add tests on collections sizes.

Predicates

Predicates consist of a predicate function and a predicate value. Predicate functions are:

Predicate	Description	Example
==	Query and predicate value are equals	jsonpath "$.book" == "Dune"
!=	Query and predicate value are different	jsonpath "$.color" != "red"
>	Query number is greater than predicate value	jsonpath "$.year" > 1978
>=	Query number is greater than or equal to the predicate value	jsonpath "$.year" >= 1978
<	Query number is less than that predicate value	jsonpath "$.year" < 1978
<=	Query number is less than or equal to the predicate value	jsonpath "$.year" <= 1978
startsWith	Query starts with the predicate value Value is string or a binary content	jsonpath "$.movie" startsWith "The" bytes startsWith hex,efbbbf;
endsWith	Query ends with the predicate value Value is string or a binary content	jsonpath "$.movie" endsWith "Back" bytes endsWith hex,ab23456;
contains	Query contains the predicate value Value is string or a binary content	jsonpath "$.movie" contains "Empire" bytes contains hex,beef;
includes	Query collections includes the predicate value	jsonpath "$.nooks" includes "Dune"
matches	Part of the query string matches the regex pattern described by the predicate value	jsonpath "$.release" matches "\\d{4}" jsonpath "$.release" matches /\d{4}/
exists	Query returns a value	jsonpath "$.book" exists
isEmpty	Query returns an empty collection	jsonpath "$.movies" isEmpty
isInteger	Query returns an integer	jsonpath "$.count" isInteger
isFloat	Query returns a float	jsonpath "$.height" isFloat
isBoolean	Query returns a boolean	jsonpath "$.succeeded" isBoolean
isString	Query returns a string	jsonpath "$.name" isString
isCollection	Query returns a collection	jsonpath "$.books" isCollection

Each predicate can be negated by prefixing it with not (for instance, not contains or not exists)

jsonpath "$.book"query not containspredicate type "Dune"predicate value

A predicate value is typed, and can be a string, a boolean, a number, a bytestream, null or a collection. Note that "true" is a string, whereas true is a boolean.

For instance, to test the presence of a h1 node in an HTML response, the following assert can be used:

GET https://example.org/home

HTTP 200
[Asserts]
xpath "boolean(count(//h1))" == true
xpath "//h1" exists # Equivalent but simpler

As the XPath query boolean(count(//h1)) returns a boolean, the predicate value in the assert must be either true or false without double quotes. On the other side, say you have an article node and you want to check the value of some data attributes:

<article
  id="electric-cars"
  data-visible="true"
...
</article>

The following assert will check the value of the data-visible attribute:

GET https://example.org/home

HTTP 200
[Asserts]
xpath "string(//article/@data-visible)" == "true"

In this case, the XPath query string(//article/@data-visible) returns a string, so the predicate value must be a string.

The predicate function equals can be used with string, numbers or booleans; startWith and contains can only be used with strings and bytes, while matches only works on string. If a query returns a number, using a matches predicate will cause a runner error.

# A really well tested web page...
GET https://example.org/home

HTTP 200
[Asserts]
header "Content-Type" contains "text/html"
header "Last-Modified" == "Wed, 21 Oct 2015 07:28:00 GMT"
xpath "//h1" exists  # Check we've at least one h1
xpath "normalize-space(//h1)" contains "Welcome"
xpath "//h2" count == 13
xpath "string(//article/@data-id)" startsWith "electric"

Status assert

Check the received HTTP response status code. Status assert consists of the keyword status followed by a predicate function and value.

GET https://example.org

HTTP *
[Asserts]
status < 300

Header assert

Check the value of a received HTTP response header. Header assert consists of the keyword header followed by the value of the header, a predicate function and a predicate value.

GET https://example.org

HTTP 302
[Asserts]
header "Location" contains "www.example.net"
header "Last-Modified" matches /\d{2} [a-z-A-Z]{3} \d{4}/

If there are multiple headers with the same name, the header assert returns a collection, so count, includes can be used in this case to test the header list.

Let's say we have this request and response:

> GET /hello HTTP/1.1
> Host: example.org
> Accept: */*
> User-Agent: hurl/2.0.0-SNAPSHOT
>
* Response: (received 12 bytes in 11 ms)
*
< HTTP/1.0 200 OK
< Vary: Content-Type
< Vary: User-Agent
< Content-Type: text/html; charset=utf-8
< Content-Length: 12
< Server: Flask Server
< Date: Fri, 07 Oct 2022 20:53:35 GMT

One can use explicit header asserts:

GET https://example.org/hello

HTTP 200
[Asserts]
header "Vary" count == 2
header "Vary" includes "User-Agent"
header "Vary" includes "Content-Type"

Or implicit header asserts:

GET https://example.org/hello

HTTP 200
Vary: User-Agent
Vary: Content-Type

URL assert

Check the last fetched URL. This is most meaningful if you have told Hurl to follow redirection (see [Options]section or --location option). URL assert consists of the keyword url followed by a predicate function and value.

GET https://example.org/redirecting
[Options]
location: true

HTTP 200
[Asserts]
url == "https://example.org/redirected"

Cookie assert

Check value or attributes of a Set-Cookie response header. Cookie assert consists of the keyword cookie, followed by the cookie name (and optionally a cookie attribute), a predicate function and value.

Cookie attributes value can be checked by using the following format: <cookie-name>[cookie-attribute]. The following attributes are supported: Value, Expires, Max-Age, Domain, Path, Secure, HttpOnly and SameSite.

GET http://localhost:8000/cookies/set

HTTP 200

# Explicit check of Set-Cookie header value. If the attributes are
# not in this exact order, this assert will fail. 
Set-Cookie: LSID=DQAAAKEaem_vYg; Expires=Wed, 13 Jan 2021 22:23:01 GMT; Secure; HttpOnly; Path=/accounts; SameSite=Lax;
Set-Cookie: HSID=AYQEVnDKrdst; Domain=localhost; Expires=Wed, 13 Jan 2021 22:23:01 GMT; HttpOnly; Path=/
Set-Cookie: SSID=Ap4PGTEq; Domain=localhost; Expires=Wed, 13 Jan 2021 22:23:01 GMT; Secure; HttpOnly; Path=/

# Using cookie assert, one can check cookie value and various attributes.
[Asserts]
cookie "LSID" == "DQAAAKEaem_vYg"
cookie "LSID[Value]" == "DQAAAKEaem_vYg"
cookie "LSID[Expires]" exists
cookie "LSID[Expires]" contains "Wed, 13 Jan 2021"
cookie "LSID[Max-Age]" not exists
cookie "LSID[Domain]" not exists
cookie "LSID[Path]" == "/accounts"
cookie "LSID[Secure]" exists
cookie "LSID[HttpOnly]" exists
cookie "LSID[SameSite]" equals "Lax"

Secure and HttpOnly attributes can only be tested with exists or not exists predicates to reflect the Set-Cookie header semantics (in other words, queries <cookie-name>[HttpOnly] and <cookie-name>[Secure] don't return boolean).

Body assert

Check the value of the received HTTP response body when decoded as a string. Body assert consists of the keyword body followed by a predicate function and value. The encoding used to decode the body is based on the charset value in the Content-Type header response.

GET https://example.org

HTTP 200
[Asserts]
body contains "<h1>Welcome!</h1>"

# Our HTML response is encoded with GB 2312 (see https://en.wikipedia.org/wiki/GB_2312)
GET https://example.org/cn

HTTP 200
header "Content-Type" == "text/html; charset=gb2312"
bytes contains hex,c4e3bac3cac0bde7; # 你好世界 encoded in GB 2312
body contains == "你好世界"

Bytes assert

Check the value of the received HTTP response body as a bytestream. Body assert consists of the keyword bytes followed by a predicate function and value.

GET https://example.org/data.bin

HTTP 200
[Asserts]
bytes startsWith hex,efbbbf;
bytes count == 12424
header "Content-Length" == "12424"

XPath assert

Check the value of a XPath query on the received HTTP body decoded as a string (using the charset value in the Content-Type header response). Currently, only XPath 1.0 expression can be used. Body assert consists of the keyword xpath followed by a predicate function and value. Values can be string, boolean or number depending on the XPath query.

Let's say we want to check this HTML response:

$ curl -v https://example.org

< HTTP/1.1 200 OK
< Content-Type: text/html; charset=UTF-8
...
<!doctype html>
<html>
  <head>
    <title>Example Domain</title>
    ...
  </head>

  <body>
    <div>
      <h1>Example</h1>
      <p>This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.</p>
      <p><a href="https://www.iana.org/domains/example">More information...</a></p>
    </div>
  </body>
</html>

With Hurl, we can write multiple XPath asserts describing the DOM content:

GET https://example.org

HTTP 200
Content-Type: text/html; charset=UTF-8
[Asserts]
xpath "string(/html/head/title)" contains "Example" # Check title
xpath "count(//p)" == 2                             # Check the number of <p>
xpath "//p" count == 2                              # Similar assert for <p>
xpath "boolean(count(//h2))" == false               # Check there is no <h2>  
xpath "//h2" not exists                             # Similar assert for <h2> 

XML Namespaces are also supported. Let's say you want to check this XML response:

<?xml version="1.0"?>
<!-- both namespace prefixes are available throughout -->
<bk:book xmlns:bk='urn:loc.gov:books'
         xmlns:isbn='urn:ISBN:0-395-36341-6'>
    <bk:title>Cheaper by the Dozen</bk:title>
    <isbn:number>1568491379</isbn:number>
</bk:book>

This XML response can be tested with the following Hurl file:

GET http://localhost:8000/assert-xpath

HTTP 200
[Asserts]

xpath "string(//bk:book/bk:title)" == "Cheaper by the Dozen"
xpath "string(//*[name()='bk:book']/*[name()='bk:title'])" == "Cheaper by the Dozen"
xpath "string(//*[local-name()='book']/*[local-name()='title'])" == "Cheaper by the Dozen"

xpath "string(//bk:book/isbn:number)" == "1568491379"
xpath "string(//*[name()='bk:book']/*[name()='isbn:number'])" == "1568491379"
xpath "string(//*[local-name()='book']/*[local-name()='number'])" == "1568491379"

The XPath expressions string(//bk:book/bk:title) and string(//bk:book/isbn:number) are written with bk and isbn namespaces.

For convenience, the first default namespace can be used with _

JSONPath assert

Check the value of a JSONPath query on the received HTTP body decoded as a JSON document. JSONPath assert consists of the keyword jsonpath followed by a predicate function and value.

Let's say we want to check this JSON response:

curl -v http://httpbin.org/json

< HTTP/1.1 200 OK
< Content-Type: application/json
...

{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!",
        "type": "all"
      },
       ...
    ],
    "title": "Sample Slide Show"
  }
}

With Hurl, we can write multiple JSONPath asserts describing the DOM content:

GET http://httpbin.org/json

HTTP 200
[Asserts]
jsonpath "$.slideshow.author" == "Yours Truly"
jsonpath "$.slideshow.slides[0].title" contains "Wonder"
jsonpath "$.slideshow.slides" count == 2
jsonpath "$.slideshow.date" != null
jsonpath "$.slideshow.slides[*].title" includes "Mind Blowing!"

Explain that the value selected by the JSONPath is coerced to a string when only one node is selected.

In matches predicates, metacharacters beginning with a backslash (like \d, \s) must be escaped. Alternatively, matches predicate support Javascript-like Regular expression syntax to enhance the readability:

GET https://sample.org/hello

HTTP 200
[Asserts]
# Predicate value with matches predicate:
jsonpath "$.date" matches "^\\d{4}-\\d{2}-\\d{2}$"
jsonpath "$.name" matches "Hello [a-zA-Z]+!"
# Equivalent syntax:
jsonpath "$.date" matches /^\d{4}-\d{2}-\d{2}$/
jsonpath "$.name" matches /Hello [a-zA-Z]+!/

Regex assert

Check that the HTTP received body, decoded as text, matches a regex pattern.

GET https://sample.org/hello

HTTP 200
[Asserts]
regex "^(\\d{4}-\\d{2}-\\d{2})$" == "2018-12-31"
# Same assert as previous using regex literals
regex /^(\d{4}-\d{2}-\d{2})$/ == "2018-12-31"

The regex pattern must have at least one capture group, otherwise the assert will fail. The assertion is done on the captured group value. When the regex pattern is a double-quoted string, metacharacters beginning with a backslash in the pattern (like \d, \s) must be escaped; literal pattern enclosed by / can also be used to avoid metacharacters escaping.

SHA-256 assert

Check response body SHA-256 hash.

GET https://example.org/data.tar.gz

HTTP 200
[Asserts]
sha256 == hex,039058c6f2c0cb492c533b0a4d14ef77cc0f78abccced5287d84a1a2011cfb81;

MD5 assert

Check response body MD5 hash.

GET https://example.org/data.tar.gz

HTTP 200
[Asserts]
md5 == hex,ed076287532e86365e841e92bfc50d8c;

Variable assert

# Test that the XML endpoint return 200 pets 
GET https://example.org/api/pets

HTTP 200
[Captures]
pets: xpath "//pets"
[Asserts]
variable "pets" count == 200

Duration assert

Check the total duration (sending plus receiving time) of the HTTP transaction.

GET https://sample.org/helloworld

HTTP 200
[Asserts]
duration < 1000   # Check that response time is less than one second

SSL certificate assert

Check the SSL certificate properties. Certificate assert consists of the keyword certificate, followed by the certificate attribute value.

The following attributes are supported: Subject, Issuer, Start-Date, Expire-Date and Serial-Number.

GET https://example.org

HTTP 200
[Asserts]
certificate "Subject" == "CN=example.org"
certificate "Issuer" == "C=US, O=Let's Encrypt, CN=R3"
certificate "Expire-Date" daysAfterNow > 15
certificate "Serial-Number" matches "[0-9af]+"

Body

Optional assertion on the received HTTP response body. Body section can be seen as syntactic sugar over body asserts (with equals predicate function). If the body of the response is a JSON string or a XML string, the body assertion can be directly inserted without any modification. For a text based body that is not JSON nor XML, one can use multiline string that starts with ``` and ends with ```. For a precise byte control of the response body, a Base64 encoded string or an input file can be used to describe exactly the body byte content to check.

JSON body

# Get a doggy thing:
GET https://example.org/api/dogs/{{dog-id}}

HTTP 200
{
    "id": 0,
    "name": "Frieda",
    "picture": "images/scottish-terrier.jpeg",
    "age": 3,
    "breed": "Scottish Terrier",
    "location": "Lisco, Alabama"
}

JSON response body can be seen as syntactic sugar of multiline string body with json identifier:

# Get a doggy thing:
GET https://example.org/api/dogs/{{dog-id}}

HTTP 200
```json
{
    "id": 0,
    "name": "Frieda",
    "picture": "images/scottish-terrier.jpeg",
    "age": 3,
    "breed": "Scottish Terrier",
    "location": "Lisco, Alabama"
}
```

XML body

GET https://example.org/api/catalog

HTTP 200
<?xml version="1.0" encoding="UTF-8"?>
<catalog>
   <book id="bk101">
      <author>Gambardella, Matthew</author>
      <title>XML Developer's Guide</title>
      <genre>Computer</genre>
      <price>44.95</price>
      <publish_date>2000-10-01</publish_date>
      <description>An in-depth look at creating applications with XML.</description>
   </book>
</catalog>

XML response body can be seen as syntactic sugar of multiline string body with xml identifier:

GET https://example.org/api/catalog

HTTP 200
```xml
<?xml version="1.0" encoding="UTF-8"?>
<catalog>
   <book id="bk101">
      <author>Gambardella, Matthew</author>
      <title>XML Developer's Guide</title>
      <genre>Computer</genre>
      <price>44.95</price>
      <publish_date>2000-10-01</publish_date>
      <description>An in-depth look at creating applications with XML.</description>
   </book>
</catalog>
```

Multiline string body

GET https://example.org/models

HTTP 200
```
Year,Make,Model,Description,Price
1997,Ford,E350,"ac, abs, moon",3000.00
1999,Chevy,"Venture ""Extended Edition""","",4900.00
1999,Chevy,"Venture ""Extended Edition, Very Large""",,5000.00
1996,Jeep,Grand Cherokee,"MUST SELL! air, moon roof, loaded",4799.00
```

The standard usage of a multiline string is :

```
line1
line2
line3
```

Oneline string body

For text based response body that do not contain newlines, one can use oneline string, started and ending with `.

POST https://example.org/helloworld

HTTP 200
`Hello world!`

Base64 body

Base64 response body assert starts with base64, and end with ;. MIME's Base64 encoding is supported (newlines and white spaces may be present anywhere but are to be ignored on decoding), and = padding characters might be added.

GET https://example.org

HTTP 200
base64,TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIG
FkaXBpc2NpbmcgZWxpdC4gSW4gbWFsZXN1YWRhLCBuaXNsIHZlbCBkaWN0dW0g
aGVuZHJlcml0LCBlc3QganVzdG8gYmliZW5kdW0gbWV0dXMsIG5lYyBydXRydW
0gdG9ydG9yIG1hc3NhIGlkIG1ldHVzLiA=;

File body

To use the binary content of a local file as the body response assert, file body can be used. File body starts with file, and ends with `;``

GET https://example.org

HTTP 200
file,data.bin;

File are relative to the input Hurl file, and cannot contain implicit parent directory (..). You can use --file-root option to specify the root directory of all file nodes.