Problem: we'd like to get some statistics about our users, so that
we know how many people use `xrefcheck` or at least view its README,
from what OS they do it, etc.
Solution:
There is a tool called [Scarf](https://about.scarf.sh/) that
can be used for this purpose.
1. Create a Docker package in Scarf which acts as a proxy,
so that instead of `docker pull serokell/xrefcheck` one will use
`docker pull serokell.docker.scarf.sh/serokell/xrefcheck` and
Scarf will automatically record some (anonymyzed) data.
2. Create a File package in Scarf which acts as another proxy.
It downloads artifacts from GitHub releases, e. g.:
https://serokell.gateway.scarf.sh/xrefcheck/v0.2.2/xrefcheck-x86_64-linux
3. Mention these links in usage instructions.
4. Add a tracking pixel to README to also get statistics about
README views.
Problem: After recent work on Xrefcheck redirect behavior, it remained
to discuss how to handle 304 redirects.
Solution: With the current default configuration, 304 redirects are
considered as valid, which seems appropriate taking into account that
304 responses usually mean that you previously received a successful
response for that same request and there is no need to retransmit the
resource again. We add a test case for this default config and update
the FAQ section.
Problem: We previously changed the default behaviour of Xrefcheck when
following link redirects, but did not provide a way to configure it.
Solution: We are adding a new field in the configuration file to allow
writing a list of redirect rules that will be applied to links that
match them.
Problem: Some Markdown flavours such as the GitHub one are case
insensitive regarding anchors, but our analysis is currently
case sensitive and it produces false positives.
Solution: Support case-insensitivity depending on the configured
Markdown flavour. Apply this also to ambiguous and similar anchors
detection.
Problem: Xrefcheck currently always follows redirect links.
Solution: We are changing its default behaviour regarding redirect
links to fail and report permanent redirects, and to pass for temporary
redirects. Further PRs will allow the user to configure other policies.
Problem: We are using unicode symbols as visual clues in the program output that are not commonly supported and are therefore not always displayed as intended.
Solution: Remove the usage of these symbols, as the program output is already using other visual clues and the result will remain understandable for the user.
Problem: we are not testing behavior of xrefcheck on Windows
Solution: and add workflow to run
golden and tasty tests on CI
via github-actions windows runner
Some subproblems appear:
1.
Problem: CI build fails beacuse it needs `pcre` package
Solution: add it (somehow), see `install pacman dependencies`
in ci.yml
2.
Problem: Network errors displayed different on different platforms
Solution: collect output from both and use
`assert_diff expected_linux.gold || assert_diff expected_windows.gold`
3:
Problem: "Config matches" test is failing because checkout action
clone files with CRLF, and test assert equality of two ByteStrings
Solution: manually remove CR
Improved the readme and fixed several problems:
* Mention support for GitLab - this is important and wasn't mentioned
anywhere.
* Add a FAQ clarifying how xrefcheck behaves in some important
situations.
* We don't need to get into a lot of detail about the syntax of the
`xrefcheck: ignore` annotations, where they're allowed and where
they're not. A general idea and a couple of examples are more than
enough.
* Added the backlink `[↑](#xrefcheck)` where it was missing.
* Fixed inconsistent level headers: we we're using `###` where we should
be using `##`
* `nix run` should now be `nix shell`
* Add a link to `tests/configs/github-config.yaml` which contains a list
of all supported config options.
* Instead of mentioning GitHub Actions in the "usage" section and nix in
a separate section, mention everything in the "usage" section.
* Fixed link to `stack2cabal`
* Fixed typos and rephrased some bits.
Problem: as in #169, `ignore file` annotation is ignoring
not file itself but all links at file, which is not obvious
Solution: rename it to `ignore all`
Also renamed `IMFile :: IgnoreMode` and `IMSFile :: IgnoreModeState`
to `IMAll` and `IMSAll`
Problem: At the moment, we're using the ignored option for mainly 2
purposes: 1) to ignore all files in the `.git` folder (`.git/**/*`) to
ignore all build-related temporary files (the default config ignores
`.stack-work/**/*`). A more robust alternative might be to ignore all
files implicitly ignored by git.
Solution: Use `git ls-files` to ignore all files implicitly ignored by git.
Problem: The `virtualFiles` config option supports glob patterns. On the
other hand, `ignored` only supports exact matches and `notScanned`
mathches on prefixes. There is also a bug where `ignored` does not
ignore files if they contain broken xrefcheck annotations.
Solution: Add support for glob patterns to `ignored` and
`notScanned`. Filter ignored files before parsing their contents.
Problem: We made `ignoreRefs` a required option. But the `README.md`
still contains a note that `ignoreRefs` is optional.
Solution: Remove the note that says `ignoreRefs` is optional.
Problem:
Currently we support only http and https links. If there is an `ftp://`
link, you will get exception.
Solution:
Use `ftp-client` to check connection to ftp, see response statuses and
check file existence. This produces adding new error types and small
refactoring.
Provide a test which is separate executable, where we have to pass CLA -
ftp host.
Co-authored-by: Alexander Bantyev <alexander.bantyev@serokell.io>
Problem:
I've met problem with one package version. I use cabal to build project and cabal for some reason decided to use old version of this package. With stack everything fine.
So, I think not only me, but other people prefer cabal over stack.
Solution:
Add note to our README.md that in case of cabal usage it's required to run `stack2cabal` first.
Also ignore all cabal staff so it can't be accidentally added to repo.
Problem: now when we include repository type into the config, it seems
to make sense to generate config differently depending on the repository
type. Especially taking into account that currently in some fields we
mix GitHub and GitLab -specific contents.
Solution:
Leave placeholders in the default config and later fill them from the
code depending on the required repository type.
Add a mandatory repository type parameter to `dump-config` CLI command.
Along with a test checking for config validity, add a golden test on the
produced config so that we could assess how sane it looks like.
Problem: Added ignoring features need to
be described somewhere to be visible
for users.
Solution: Add such a description to the
`Usage` part of `README.md`
Problem: there seems to be another popular tool for HTMLs verification
called `linkcheck`, it is stated to be way better than
`broken-link-checker`.
Solution: replace `broken-link-checker` mention with `linkcheck`, say
that there are other solutions to this problem.
Also make the paragraph a bit more clear in general.
Problem: @gromakovsky has recently implemented a github action which
adds xrefcheck pass to CI and it resides in a separate repository, we
want it to be visible in our documentation here.
Solution: mention it in `Usage` section.
Problem: nowadays we want all files to store licensing information in
machine-readable format and to use reuse tool to check that. But the
repo is not REUSE compliant.
Solution: add `LICENSES` folder and licensing information for each
file.
Initially the primary use case for this tool in my point of view was
repeated runs for a repository until documentation is valid, thus I've
set checking only local refs as default behaviour (checking external
refs takes way longer).
Another motivation was that I expected CI to have one mandatory job for
local scan and one optional for remote refs scan.
Now I think that there are more reasons to perform full scan by default.
* It occured that in CI we usually want to have all checks optional, so
that someone could temporarily break links on purpose (e.g. in case
of mutually-dependent pull requests).
* First impression is important, and we want to demostrate all the
features of the tool in default run.
* If use case mentioned in the very top actually takes place, one can
add the necessary option, that's fine.
Problem: currently `crossref-verify` executable does not work without
configuration, and user was supposed to use one from the repository if
he does not have its own. This is very inconvenient.
Solution: make executable remember some config and use it by default.
