From 4fffe7440830cd4942f22eea6bbc1aaee19f3f11 Mon Sep 17 00:00:00 2001
From: Simon Michael <simon@joyful.com>
Date: Fri, 19 Apr 2019 09:56:33 -0700
Subject: [PATCH] doc: add a README for the functional tests, linked from
 contrib guide

---
 CONTRIBUTING.md |  1 +
 tests/README.md | 78 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 79 insertions(+)
 create mode 100644 tests/README.md

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index c8e3bd13b..fe3713a64 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -774,6 +774,7 @@ About testing in the hledger project, as of 201809.
     Tests are defined in files under
     [tests/](https://github.com/simonmichael/hledger/tree/master/tests),
     grouped by *component* (command or topic name).
+    For more about these, see the README there.
 
 4.  Code tests
 
diff --git a/tests/README.md b/tests/README.md
new file mode 100644
index 000000000..87ad0c45c
--- /dev/null
+++ b/tests/README.md
@@ -0,0 +1,78 @@
+hledger's functional tests.
+See also [Contributor Guide: Tests].
+
+These mainly test the hledger CLI and (indirectly) hledger-lib.
+They are organised roughly by [component].
+
+Running these requires [shelltestrunner].
+Older test files are in [format 1]; newer ones use [format 3] (preferred).
+Some tests invoke unix commands so will not run in a Windows CMD shell.
+
+[Contributor Guide: Tests]: https://hledger.org/CONTRIBUTING.html#tests
+[component]: https://hledger.org/CONTRIBUTING.html#components
+[shelltestrunner]: https://github.com/simonmichael/shelltestrunner#readme
+[format 1]: https://github.com/simonmichael/shelltestrunner#format-1
+[format 3]: https://github.com/simonmichael/shelltestrunner#format-3
+
+
+Run them all (also builds hledger):
+
+    make functest
+
+See how the Makefile is invoking shelltestrunner:
+
+    $ make functest -n
+    stack build --fast hledger
+    (COLUMNS=80 stack exec -- shelltest --execdir -j16 --hide-successes --exclude=/_ -w `stack exec -- which hledger` tests \
+            && echo functest PASSED) || (echo functest FAILED; false)
+
+These are the most important:
+
+- `COLUMNS=80` makes output independent of your terminal width.
+- `--execdir` runs each test within its own directory.
+- ``-w `stack exec -- which hledger` `` ensures you are testing the hledger executable that was just built.
+- `-j16` runs tests in parallel which is much faster.
+
+Run only the tests matching a regular expression:
+
+    $ COLUMNS=80 shelltest --execdir -w `stack exec -- which hledger` tests -i balance-assertions.*19
+    :tests/journal/balance-assertions.test:19: [OK]
+
+             Test Cases  Total      
+     Passed  1           1          
+     Failed  0           0          
+     Total   1           1          
+
+Run only the tests in one file:
+
+    $ COLUMNS=80 shelltest --execdir -w `stack exec -- which hledger` tests/cli/query-args.test
+    :tests/cli/query-args.test:1: [OK]
+    :tests/cli/query-args.test:2: [OK]
+    :tests/cli/query-args.test:3: [OK]
+
+             Test Cases  Total      
+     Passed  3           3          
+     Failed  0           0          
+     Total   3           3          
+
+Run a test [repeatedly](http://eradman.com/entrproject/) as its file is changed:
+
+    $ ls tests/cli/query-args.test | entr bash -c "COLUMNS=80 shelltest --execdir -w `stack exec -- which hledger` tests/cli/query-args.test -i1"
+    :tests/cli/query-args.test:1: [OK]
+
+             Test Cases  Total      
+     Passed  1           1          
+     Failed  0           0          
+     Total   1           1          
+    :tests/cli/query-args.test:1: [OK]
+
+             Test Cases  Total      
+     Passed  1           1          
+     Failed  0           0          
+     Total   1           1          
+      C-c C-c
+
+More shelltestrunner options:
+
+    $ shelltest --help
+