Problem: the current usage of filepaths is error-prone and can be
simplified.
Solution: canonicalize filepaths at the boundaries, so their management
will be safer and will simplify the codebase.
Problem: the danger checks were failing because it was configured to
fetch only and partially the current PR branch.
Solution: force the danger checks CI to get all the repository branches.
Problem: the danger checks that we copied had a license header, and
REUSE now complains that those headers refer to unknown licences.
Solution: copy the license file.
Problem: it would be helpful to have checks for common rules on styling
commits and stuff, to avoid checking those manually every time.
Solution: add Danger checks, mostly the same we had in Morley.
Problem: There is currently some problem in stack or cabal
that produces a warning when building this project on
case-insensitive systems.
Solution: The current workaroud for it is to add the GHC
option '-optP-Wno-nonportable-include-path'.
Problem: We have a Golden test that expects an output in English
and fails if a different language is configured.
Solution: Configure explicitly the language before running the
corresponding test.
Problem: Some Markdown flavours such as the GitHub one are case
insensitive regarding anchors, but our analysis is currently
case sensitive and it produces false positives.
Solution: Support case-insensitivity depending on the configured
Markdown flavour. Apply this also to ambiguous and similar anchors
detection.
Problem: Xrefcheck currently always follows redirect links.
Solution: We are changing its default behaviour regarding redirect
links to fail and report permanent redirects, and to pass for temporary
redirects. Further PRs will allow the user to configure other policies.
Problem: Currently, there are some invalid links in the repository,
mostly for tests, so one has to add some CLI options
for `xrefcheck` to succeed.
Solution: Added a new minimal `.xrefcheck.yaml` config file
such that `xrefcheck` succeeded without any options.
Problem: We have found that the current tag for local file references, current file, may lead to ambiguities.
Solution: Rename the tag that we use for local file references to be file-local instead.
Problem: We are using unicode symbols as visual clues in the program output that are not commonly supported and are therefore not always displayed as intended.
Solution: Remove the usage of these symbols, as the program output is already using other visual clues and the result will remain understandable for the user.
Problem: we are not testing behavior of xrefcheck on Windows
Solution: and add workflow to run
golden and tasty tests on CI
via github-actions windows runner
Some subproblems appear:
1.
Problem: CI build fails beacuse it needs `pcre` package
Solution: add it (somehow), see `install pacman dependencies`
in ci.yml
2.
Problem: Network errors displayed different on different platforms
Solution: collect output from both and use
`assert_diff expected_linux.gold || assert_diff expected_windows.gold`
3:
Problem: "Config matches" test is failing because checkout action
clone files with CRLF, and test assert equality of two ByteStrings
Solution: manually remove CR
Problem: in markdown links, '/' should always be used as path separator ,
e.g. GitHub renderer is not threating link `[x](a\b.md)` as a link
to file in dir `a`
We use OS-dependent file separators everywhere, so
e.g. `canonizeLocalRef "./a.md"="./a.md`"` on Windows
Solution: replace '\' to '/' while constructing repo tree if needed
and then use only '/', preferring functions from `System.FilePath.Posix`.
We can do this, since 'a/b' and `a\b` are equivalent paths on Windows
Problem: xrefcheck uses utf8 symbols in reports, which are not supported
on most of Windows shells by default.
Sometimes they are printed as question marks (and it cause golden tests to fail)
and sometimes printing of them raise an error.
Solution: use function `withCP65001` from `code-page` package which
sets correct codepage on Windows and do nothing on other OSs
Problem:
We have a function `defConfigText :: Flavor -> ByteString` that
uses `fillHoles` to modify `defConfigUnfilled`.
This is a bit error-prone and very complicated way to have a
`ByteString` with parametric blocks. Also using `ByteString`
instead of `Text` to store text leads to CRLF-related issues when
launched on Windows.
Solution:
Remove `fillHoles` and `defConfigUnfilled`,
`defConfigText` creates a `Text` using `nyan-interpolation`.
Improved the readme and fixed several problems:
* Mention support for GitLab - this is important and wasn't mentioned
anywhere.
* Add a FAQ clarifying how xrefcheck behaves in some important
situations.
* We don't need to get into a lot of detail about the syntax of the
`xrefcheck: ignore` annotations, where they're allowed and where
they're not. A general idea and a couple of examples are more than
enough.
* Added the backlink `[↑](#xrefcheck)` where it was missing.
* Fixed inconsistent level headers: we we're using `###` where we should
be using `##`
* `nix run` should now be `nix shell`
* Add a link to `tests/configs/github-config.yaml` which contains a list
of all supported config options.
* Instead of mentioning GitHub Actions in the "usage" section and nix in
a separate section, mention everything in the "usage" section.
* Fixed link to `stack2cabal`
* Fixed typos and rephrased some bits.
Problem: We have a pipeline step to tag docker images on dockerhub
whenever a new version is released:
7dd5c4c3c9/.buildkite/pipeline.yml (L51-L56)
However, this doesn't seem to be working, dockerhub only contains the
`latest` tag: https://hub.docker.com/r/serokell/xrefcheck/tags
The problem *seems* to be that the CI step is only triggered when it
builds a branch with a name matching the regex `/^v[0-9]+.*/`. But we
never use that format for branch names, so it's never triggered.
Solution:
1. Change the CI step to trigger when it detects a tag with a version
number
2. Enable the "Build tags" option in buildkite:
https://buildkite.com/serokell/xrefcheck/settings/repository
Problem: xrefcheck checks only files that are tracked by Git,
but sometimes we want to run xrefcheck on
files without adding them to Git, e.g. when we want to
test some generator of markdown files or when we actively
create markdown files during development.
Solution: add option to treat files that were neither
added to git nor ignored as existing.
Problem: after 0.2.2 release, xrefcheck cares only about files
that were added to Git. That can be confusing for users (see #200)
Solution:
If a scannable (currently it means markdown) file is not ignored
(by git or via config) and not tracked by git, print a warning to
stderr while scanning repo.
If a link target such file, change error message from "file not exists"
to `Link target is not tracked by Git`
Suggest user to run "git add" before running xrefcheck in both cases.
To do this, I've changed the `RepoInfo` type, so it also contains
information about untracked files now.
Currently, if xrefcheck is interrupted, it's not displaying any
`VerifyError`s, even if it has already found some.
When we found error, we are notifying user via progress bar, but he must
wait checking of all references to see it
(and it can be long enough).
Solution: when xrefcheck is interrupted, display all errors
we have already found and a number of
checked references.
To do this, we need to modify `forConcurrentlyCaching`
behaviour on exceptions.
Problem: bats tests are not space sensetive
Solution: remove trailing spaces from xrefcheck output
(see next problems), remove `--ignore-trailing-space`
from `assert_diff`
Problem: there are lines containing only spaces in
xrefcheck's output, because `Fmt.indentF` "indents"
empty lines too.
Solution: add `Xrefcheck.Util.Interpolate.interpolateIndentF`
function that is not indenting empty lines.
Same for `Fmt.blockListF` and `Fmt.blockListF'`.
Those functions are not adding trailing newlines, so it's
easier to use it in interpolation blocks.
Problem: when there is a current file link `[a](#b)`, it is
printed like
```
- text: "a"
- link: (trailing space here)
- anchor: b
```
Solution: like with anchors, print `link: -` instead
Problem: xrefcheck's output contains many redundant newlines, so
it takes more display space than it could.
For list of places where such newlines appear, see #208
Solution: don't print redundant newlines, so output is more compact.
Since tests are newline-sensetive, I've checked that now there
are no extra (e.g. 2 ajacent) blank lines in `expected` part of tests.
Problem:
We often need to create large strings, and we use different
fmt tools for this (by-hand concatenation, unlinesF, etc).
Sometimes it is unclear or too heavy, and it always can
be called error-prone
Solution: use `int` quasiquoter to build large strings and
have nice-looking and easy-to-read code
Problem: Right now, our bats tests ignore empty lines and
leading/trailing whitespace differences between the expected output and
the actual output.
However, this could lead to accidental bugs in xrefcheck's output.
Trailing whitespace isn't very concerning (except when it's excessive
and it causes the terminal to line-wrap), but additional/missing empty
lines and leading whitespace can lead to significant changes.
Solution: Let's make these tests sensitive to empty lines and leading
whitespace.
Problem: as in #169, `ignore file` annotation is ignoring
not file itself but all links at file, which is not obvious
Solution: rename it to `ignore all`
Also renamed `IMFile :: IgnoreMode` and `IMSFile :: IgnoreModeState`
to `IMAll` and `IMSAll`
