Commit Graph

43 Commits

Author SHA1 Message Date
Silvan Mosberger
c701a4dd29 lib.sources.pathType and co.: Move to lib.filesystem
These functions only work with the filesystem, they don't import
anything as sources
2023-04-05 17:18:10 +02:00
Artturin
962929884a lib/sources: make commitIdFromGitRepoOrError internal and use # for comments
so that it doesn't make the manual build fail

> Apparently this is related to the combination of this new function not getting exported from the file, while still getting documented.
2022-11-18 17:17:51 +02:00
Robert Hensing
ec8f8f69bd lib/sources: Make pathIsGitRepo not evaluate toString path
This requires us to avoid the `tryEval` + `throw` combination,
because throw is strict in its error message, and we don't want
to drop our single clue when `commitIdFromGitRepo` is used
incorrectly.
2022-11-07 12:54:44 +01:00
Artturin
8c1b0192c5 lib/sources: remove 2 usages of toString on a path which will be read using fileContents
It gives a warning on the lazy-trees branch of Nix
(NixOS/nix#6530)

"warning: applying 'toString' to path '...' and then accessing it is deprecated, at '...'"

'else toString (/. + "${base}/${path}");' at line 183 may still cause a warning but i don't know how
to reach that codepath and test so im leaving it untouched

changing it to 'else /. + "${base}/${path}";'
caused this error
```
error: a string that refers to a store path cannot be appended to a path

       at /home/systems/nixpkgs/lib/sources.nix:183:20:

          182|               then path
          183|               else /. + "${base}/${path}";
             |                    ^
          184|         in if pathIsRegularFile path
```
2022-11-06 19:41:52 +02:00
figsoda
53052c4fb0 lib/sources.nix: remove unused binding 2022-09-26 22:18:55 -04:00
Jan Tojnar
1e1396aafc lib.sources: Improve docs
Change comment type so than nixdoc picks them up into Nixpkgs manual.
Also improve phrasing a bit and move stuff around so that it is formatted better.
2022-01-30 02:04:51 +01:00
Alyssa Ross
f446318916
lib.cleanSource: ignore sockets
I'm working on a project that involves running a virtual machine
monitor program, which creates a control socket in the project
directory (because it doesn't make sense to put it anywhere else).
This obviously isn't part of the source of my program, so I think
cleanSource should filter it out.
2021-09-17 10:03:00 +00:00
Robert Hensing
4cf56e5640 lib.sources.trace: init 2021-05-29 16:03:55 +02:00
Robert Hensing
970273023a lib.sources.sourceFilesBySuffices: Improve doc 2021-05-29 16:03:54 +02:00
Robert Hensing
d14be76615 lib/tests/sources.sh: init 2021-05-29 14:32:56 +02:00
Robert Hensing
dfd2b1bd90 lib/sources: Internal representation for cleanSourceWith 2021-05-29 14:32:56 +02:00
Léo Gaspard
144a997c8e
lib: fix commitIdFromGitRepo (#117752)
When in the presence of worktrees, it happens that /commondir has a
trailing slash.

In these circumstances, it can lead to `lib.pathType` being passed paths
like `/foo/bar/.git/`, which in turn lead to
`error: attribute '.git' missing`.

With this change, we now make sure send properly-formatted paths to all
other functions.

This, in particular, fixes running NixOS tests on worktrees created by
libgit2 on my machine. (Worktrees created by git itself appear to not
hit the issue.)
2021-03-27 21:16:31 +01:00
V
3e1652f8f0 lib/sources.nix: fix incorrect inherit
split comes from builtins, not lib.

	error: attribute 'split' missing, at /nix/path/nixpkgs/lib/sources.nix:4:4
	(use '--show-trace' to show detailed location information)
2020-10-30 23:19:22 +01:00
Robert Hensing
a4019a40b3 lib/sources.nix: Prefer lib for readFile inherit 2020-10-22 13:57:23 +02:00
Robert Hensing
afa6c51f27 lib: Use Nix's static scope checking, fix error message, optimize
Nix can perform static scope checking, but whenever code is inside
a `with` expression, the analysis breaks down, because it can't
know statically what's in the attribute set whose attributes were
brought into scope. In those cases, Nix has to assume that
everything works out.

Except it doesnt. Removing `with` from lib/ revealed an undefined
variable in an error message.

If that doesn't convince you that we're better off without `with`,
I can tell you that this PR results in a 3% evaluation performance
improvement because Nix can look up local variables by index.
This adds up with applications like the module system.

Furthermore, removing `with` makes the binding site of each
variable obvious, which helps with comprehension.
2020-10-22 13:46:47 +02:00
Jörg Thalheim
d7e89fa661
commitIdFromGitRepo: fix stackoverflow if many branches are used.
If many branches are created than builtins.match stack overflows because
of a bug in libstdc++: see https://github.com/NixOS/nix/issues/2147
2020-07-17 10:44:08 +01:00
Michael Peyton Jones
07f363fae3
cleanSourceWith: don't use baseNameOf
Currently, not providing `name` to `cleanSourceWith` will use the name
of the imported directory. However, a common case is for this to be the
top level of some repository. In that case, the name will be the name of
the checkout on the current machine, which is not necessarily
reproducible across different settings, and can lead to e.g. cache
misses in CI.

This is documented in the comment on `cleanSourceWith`, but this does
not stop it being a subtle trap for users.

There are different tradeoffs in each case:

1. If `cleanSourceWith` defaults to `"source"`, then we may end up with a
user not knowing what directory a source store path corresponds to.
However, it being called "unnamed" may give them a clue that there is a
way for them to name it, and lead them to the definition of the
function, which has a clear `name` parameter.

2. If `cleanSoureWith` defaults to the directory name, then a user may face
occasional loss of caching, which is hard to notice, and hard to track
down. Tracking it down likely requires use of more advanced tools like
`nix-diff`, and reading the source of a lot of nix code.

I think the downside of the status quo is worse.

This is really another iteration of
https://github.com/NixOS/nix/issues/1305: that led to adding the `name`
argument in the first place, this just makes us use a better default
`name`.
2020-03-23 09:53:07 +00:00
Franz Pletz
46773a15b3
nixos/version: fix case where .git is a symlink
Before c9214c394b and
9d396d2e42 if .git is symlink the version
would gracefully default to no git revision. With those changes an
exception is thrown instead.

This introduces a new function `pathIsGitRepo` that checks if
`commitIdFromGitRepo` fails without error so we don't have to
reimplement this logic again and can fail gracefully.
2020-01-20 00:53:44 +01:00
elseym
9d396d2e42
lib.commitIdFromGitRepo: fix support for git-submodule
Adds handling for relative references from .git-files, fixing a bug introduced by c9214c394b.
2020-01-14 21:08:38 +01:00
elseym
c9214c394b
lib.commitIdFromGitRepo: support git-worktree
lib.commitIdFromGitRepo now resolves the refs from the
parent repository in case the supplied path is a file
containing the path to said repository. this adds support
for git-worktree and things alike. see gitrepository-layout(5).

this also:
- adds a new boolean function lib.pathIsRegularFile to
  check whether a path is a regular file
- patches lib.revisionWithDefault and
  the revision and versionSuffix attributes in
  config.system.nixos in order to support git-worktrees
2020-01-10 22:29:48 +01:00
Robert Hensing
9a2180fa0b lib.cleanSourceWith: Allow name to be set, optional filter, doc
This change is API-compatible and hash-compatible with the previous
version.

At first I considered to write a rename function too, but adding
it name to cleanSourceWith was a no-brainer for ease of use. It
turns out that a rename function isn't any more useful than
cleanSourceWith.
To avoid having to write the identity predicate when renaming,
the filter is now optional.

builtins.path is supported since Nix 2.0 which is required by nixpkgs
2019-09-03 10:36:57 +02:00
Bas van Dijk
58ea28eb2c lib: allow sourceByRegex to be composed after cleanSourceWith
`sourceByRegex src regexes` should include a source file if one of the
regular expressions `regexes` matches the path of that file relative
to `src`.

However to compute this relative path `sourceByRegex` uses:

```
relPath = lib.removePrefix (toString src + "/") (toString path);
```

Note that `toString path` evaluates to an absolute file somewhere
under `src` and not under `/nix/store`.

The problem is that this doesn't work if `src` is a `cleanSourceWith`
invocation as well because `toString src` will then evaluate to
`src.outPath` which will evaluate to `builtins.filterSource ...` which
evaluates to a path in `/nix/store` which is not a prefix of `path`.

The solution is to replace `src` with `origSrc` where

```
origSrc = if isFiltered then src.origSrc else src;
isFiltered = src ? _isLibCleanSourceWith;
```

Test this by executing the following from the nixpkgs repo:

```
(cat << 'EOI'
let
  pkgs = import ./. {};
in pkgs.runCommand "test-sourceByRegex" {
  test_sourceByRegex =
    let
      src1 = pkgs.lib.sourceByRegex ./.  [ "^test-sourceByRegex.nix$" ];
      src2 = pkgs.lib.sourceByRegex src1 [ "^test-sourceByRegex.nix$" ];
    in src2 + "/test-sourceByRegex.nix";
} ''
 cp $test_sourceByRegex $out
''
EOI
) > test-sourceByRegex.nix
nix-build test-sourceByRegex.nix
```
2019-07-19 16:23:11 +02:00
Daniel Schaefer
786f02f7a4 treewide: Remove usage of isNull
isNull "is deprecated; just write e == null instead" says the Nix manual
2019-04-29 14:05:50 +02:00
Silvan Mosberger
eb09fd5a88
lib.cleanSourceFilter: Filter all .git, not just directories
In the case of a worktree created with `git worktree add`, .git is
actually a file with contents pointing to the original repository.
2019-04-08 16:20:09 +02:00
Jan Malakhovski
2f5e4c733b lib: commitIdFromGitRepo: simplify a tiny bit 2018-11-08 05:20:18 +00:00
Tobias Pflug
1d68b5ee84 docs: documentation for cleanSource 2018-10-02 22:05:06 +02:00
Shea Levy
4e78aeb441
callCabal2nix: Fix calling with a path in the store. 2018-01-11 10:17:56 -05:00
Will Fancher
7e5f37d73b Added lib.cleanSourceWith as composable version of filterSource 2018-01-09 18:59:03 -05:00
Graham Christensen
152c63c9ff
Convert libs to a fixed-point
This does break the API of being able to import any lib file and get
its libs, however I'm not sure people did this.

I made this while exploring being able to swap out docFn with a stub
in #2305, to avoid functor performance problems. I don't know if that
is going to move forward (or if it is a problem or not,) but after
doing all this work figured I'd put it up anyway :)

Two notable advantages to this approach:

1. when a lib inherits another lib's functions, it doesn't
   automatically get put in to the scope of lib
2. when a lib implements a new obscure functions, it doesn't
   automatically get put in to the scope of lib

Using the test script (later in this commit) I got the following diff
on the API:

  + diff master fixed-lib
  11764a11765,11766
  > .types.defaultFunctor
  > .types.defaultTypeMerge
  11774a11777,11778
  > .types.isOptionType
  > .types.isType
  11781a11786
  > .types.mkOptionType
  11788a11794
  > .types.setType
  11795a11802
  > .types.types

This means that this commit _adds_ to the API, however I can't find a
way to fix these last remaining discrepancies. At least none are
_removed_.

Test script (run with nix-repl in the PATH):

  #!/bin/sh

  set -eux

  repl() {
      suff=${1:-}
      echo "(import ./lib)$suff" \
          | nix-repl 2>&1
  }

  attrs_to_check() {
      repl "${1:-}" \
          | tr ';'  $'\n' \
          | grep "\.\.\." \
          | cut -d' ' -f2 \
          | sed -e "s/^/${1:-}./" \
          | sort
  }

  summ() {
      repl "${1:-}" \
          | tr ' ' $'\n' \
          | sort \
          | uniq
  }

  deep_summ() {
      suff="${1:-}"
      depth="${2:-4}"
      depth=$((depth - 1))
      summ "$suff"

      for attr in $(attrs_to_check "$suff" | grep -v "types.types"); do
          if [ $depth -eq 0 ]; then
              summ "$attr" | sed -e "s/^/$attr./"
          else
              deep_summ "$attr" "$depth" | sed -e "s/^/$attr./"
          fi
      done
  }

  (
      cd nixpkgs

      #git add .
      #git commit -m "Auto-commit, sorry" || true
      git checkout fixed-lib
      deep_summ > ../fixed-lib
      git checkout master
      deep_summ > ../master
  )

  if diff master fixed-lib; then
      echo "SHALLOW MATCH!"
  fi

  (
      cd nixpkgs
      git checkout fixed-lib
      repl .types
  )
2017-09-16 21:36:43 -04:00
Tuomas Tynkkynen
9275c3387e lib.cleanSourceFilter: Fix VIM swap file filtering
The backslash wasn't properly escaped, and "\." is apparently equal to
".". So it's accidentally filtering out these valid file names (in
Nixpkgs):

trace: excluding clfswm
trace: excluding larswm
trace: excluding mkpasswd

While at it, turn the file filter stricter to what it was before
e2589b3ca2. That is, the file name must
start with a dot: '.swp', '.foo.swo' are filtered but 'bar.swf' is not.
2017-09-12 14:58:46 +03:00
Dan Peebles
e2589b3ca2 Deduplicate some filterSource invocations
This version should have more conventional regexes that work across many
platforms and regex engines. This is an issue because up until Nix 1.11,
Nix called out to the libc regex matcher, which behaved differently on
Darwin and Linux. And in Nix 1.12, we're moving to std::regex which will
also behave differently here.

And yes, I do actually evaluate make-disk-image.nix on Darwin ;)
2017-08-29 20:27:04 -04:00
Tom Hunger
cb96ed615e lib: Add a function to filter sources by regular expressions. 2017-02-17 18:56:39 +00:00
Elliot Cameron
402c07c563 Expose guts of cleanSource
Ref #20511
2016-11-17 22:21:18 -05:00
obadz
f242b752fe lib/sources.nix@commitIdFromGitRepo: parenthesize what should always have been
Fixes error: cannot coerce a partially applied built-in function to a string, at lib/sources.nix:59:32

I don't understand how this used to work, but doesn't work anymore?
2016-08-08 11:46:41 +01:00
Domen Kožar
39e689e316 cleanSource: filter out also nix result symlinks (#16120) 2016-08-06 19:53:18 +01:00
Eric Sagnes
85d8b01660 lib: refactor commitIdFromGitRepo with fileContents 2016-08-01 18:35:26 +09:00
obadz
fee334f672 lib/sources.nix@commitIdFromGitRepo: remove use of lib.splitString
lib.splitString was blowing up the stack in instances where the
.git/packed-refs file was too large. We now use a regexp over the
whole file instead.

Closes #16570
2016-07-27 15:47:08 +01:00
obadz
4c5fdf42ed nixos/modules/misc/version.nix: check that .git is a directory
That's not the case for git submodules
Fixes #15928
2016-06-03 13:38:41 +01:00
obadz
47950b5353 modules/misc/version.nix: populate nixosRevision based on <nixpkgs/.git> when possible (#15624)
Example:

$ nixos-option system.nixosLabel
Value:
"16.09.git.4643ca1"
2016-05-24 23:34:28 +01:00
Eelco Dolstra
b1ce3cc172 Manual: Handle XML files in subdirectories 2014-08-25 14:33:33 +02:00
Eelco Dolstra
de9ed78ab2 cleanSources: Filter *.o and *.so 2014-02-06 12:30:26 +01:00
Eelco Dolstra
34638ebc1e cleanSource: Support Mercurial 2014-02-03 23:44:11 +01:00
Eelco Dolstra
5fef92c4a0 Move pkgs/lib/ to lib/ 2013-10-10 13:28:21 +02:00