git.iko.soy/haskell.nix
on df2147d help sign in
haskell.nix/docs/dev/coverage.md
180 lines 8.0 kB history raw
1# Developer Coverage Overview
3## Building
5The implementation of coverage starts with the "doCoverage" flag on
6the builder in `comp-builder.nix`. The doCoverage flag enables and
7disables the Cabal coverage flag and copies any generated coverage
8data to "$out/share/hpc".
10### Mix and tix files
12The coverage information for any derivation consists of "mix" and
13"tix" files.
15Mix files record static information about a source file and are
16generated at build time. They primarily contain a path to the source
17file and information about expressions and regions of the source file,
18which are later referenced by tix files.
20Tix files contain dynamic information about a test run, recording when
21a portion of a source file is touched by a test. These are generated
22when the test is run.
24### Multiple local packages
26In the context of multiple local packages, there are a few types of
27coverage we might be interested in:
28 - How well does the tests for this package cover the package library?
29 - How well does the tests for this package cover the libraries of
30 other packages in this project?
31 - Both of the above.
33To facilitate expressing any of these classifications of coverage, the
34`lib/cover.nix` function provides the `mixLibraries` argument. If
35you're just interested in how the tests cover the package library, you
36provide that library as an argument to `mixLibraries`. If you're
37interested in how the tests also cover other local packages in the
38project, you can also provide those libraries as arguments to
39mixLibraries.
41The `projectCoverageReport` and `coverageReport` attributes that are
42provided by default on projects and packages respectively provide
43coverage information for *all* local packages in the project. This is
44to mimic the behaviour of Stack, which seems to be the expectation of
45most people. Of course, you can use the `projectCoverageReport` and
46`coverageReport` functions to construct your own custom coverage
47reports (as detailed in the [coverage tutorial](../tutorials/coverage.md#custom)).
49## Coverage reports
51### Package reports
53The coverage information generated will look something like this:
55```bash
56/nix/store/...-my-project-0.1.0.0-coverage-report/
57└── share
58 └── hpc
59 └── vanilla
60 ├── html
61 │   └── my-library-0.1.0.0
62 │   ├── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
63 │ │ ├── My.Lib.Config.hs.html
64 │ │ ├── My.Lib.Types.hs.html
65 │ │ └── My.Lib.Util.hs.html
66 │   ├── hpc_index_alt.html
67 │   ├── hpc_index_exp.html
68 │   ├── hpc_index_fun.html
69 │   └── hpc_index.html
70 ├── mix
71 │   └── my-library-0.1.0.0
72 │   └── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
73 │      ├── My.Lib.Config.mix
74 │      ├── My.Lib.Types.mix
75 │      └── My.Lib.Util.mix
76 └── tix
77 └── my-library-0.1.0.0
78 ├── my-library-0.1.0.0.tix
79 ├── my-test-1
80 │   └── my-test-1.tix
81 └── unit-test
82 └── unit-test.tix
83```
85- The mix files are copied verbatim from the library built with
86 coverage.
87- The tix files for each test are copied from the check run verbatim
88 and are output to ".../tix/<libraryname>/<testname>/<testname>.tix".
89- The tix files for each library are generated by summing the tix
90 files for each test, but excluding any test modules. This tix file
91 is output to ".../tix/<libraryname>/<libraryname>.tix".
92 - Test modules are determined by inspecting the plan for the project
93 (i.e. for the project "my-project" and test-suite "my-test-1", the
94 test modules are read from:
95 `my-project.checks.my-test-1.config.modules`)
96- The hpc HTML reports for each library are generated from their
97 respective tix files (i.e. the
98 `share/hpc/vanilla/html/my-library-0.1.0.0` report is generated from
99 the
100 `share/hpc/vanilla/tix/my-library-0.1.0.0/my-library-0.1.0.0.tix`
101 file)
103### Project-wide reports
105The coverage information for an entire project will look something
106like this:
108```bash
109/nix/store/...-coverage-report
110└── share
111 └── hpc
112 └── vanilla
113 ├── html
114 │   ├── index.html
115 │   ├── all
116 │   │   ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
117 │ │   │ ├── My.Lib.Config.hs.html
118 │ │   │ ├── My.Lib.Types.hs.html
119 │ │   │ └── My.Lib.Util.hs.html
120 │ │ ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
121 │ │ │   ├── Other.Lib.A.hs.html
122 │ │ │   └── Other.Lib.B.hs.html
123 │   │   ├── hpc_index_alt.html
124 │   │   ├── hpc_index_exp.html
125 │   │   ├── hpc_index_fun.html
126 │   │   └── hpc_index.html
127 │   ├── my-library-0.1.0.0
128 │   │   ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
129 │ │   │ ├── My.Lib.Config.hs.html
130 │ │   │ ├── My.Lib.Types.hs.html
131 │ │   │ └── My.Lib.Util.hs.html
132 │   │   ├── hpc_index_alt.html
133 │   │   ├── hpc_index_exp.html
134 │   │   ├── hpc_index_fun.html
135 │   │   └── hpc_index.html
136 │ └── other-libray-0.1.0.0
137 │ ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
138 │ │   ├── Other.Lib.A.hs.html
139 │ │   └── Other.Lib.B.hs.html
140 │ ├── hpc_index_alt.html
141 │ ├── hpc_index_exp.html
142 │ ├── hpc_index_fun.html
143 │ └── hpc_index.html
144 ├── mix
145 │   ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
146 │   │   ├── My.Lib.Config.mix
147 │   │   ├── My.Lib.Types.mix
148 │   │   └── My.Lib.Util.mix
149 │   └── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
150 │   ├── Other.Lib.A.mix
151 │   └── Other.Lib.B.mix
152 └── tix
153 ├── all
154 │   └── all.tix
155 ├── my-library-0.1.0.0
156 │ ├── my-library-0.1.0.0.tix
157 │ ├── my-test-1
158 │ │   └── my-test-1.tix
159 │ └── unit-test
160 │ └── unit-test.tix
161 └── another-library-0.1.0.0
162 ├── another-library-0.1.0.0.tix
163 ├── my-test-2
164 │   └── my-test-2.tix
165 └── unit-test
166 └── unit-test.tix
167```
169All of the coverage information is copied verbatim from the coverage
170reports for each of the constituent packages. A few additions are
171made:
172 - `tix/all/all.tix` is generated from the union of all the library
173 tix files.
174 - We use this file when generating coverage reports for
175 "coveralls.io".
176 - An index page (`html/index.html`) is generated which links to the
177 HTML coverage reports of the constituent packages.
178 - A synthetic HTML report is generated from the `tix/all/all.tix`
179 file. This shows the union of all the coverage information
180 generated by each constituent coverage report.