nix-dot-dev/nix.dev
1
1
Fork 0
mirror of https://github.com/nix-dot-dev/nix.dev.git synced 2024-07-15 00:50:28 +03:00
Code Issues Projects Releases Wiki Activity

Review in meeting

Browse Source
This commit is contained in:
Silvan Mosberger 2023-11-23 17:05:37 +01:00
parent fb3fb0c060
commit b6700a9c45
1 changed files with 71 additions and 108 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

179
source/tutorials/file-sets.md
View File

@ -20,13 +20,12 @@ Using these features directly can be tricky however:
can address these problems.
However, it's often hard to express the desired path selection using the `filter` function interface.
In this tutorial you'll learn how to use Nixpkgs' [`lib.fileset` library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations.
It abstracts over built-in functionality with a safer and more convenient interface.
In this tutorial you'll learn how to use the Nixpkgs [`lib.fileset` library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations.
It abstracts over built-in functionality and offers a safer and more convenient interface.
## File sets
The basic concept is that of a _file set_,
a data type representing a collection of local files.
A _file set_ is a data type representing a collection of local files.
File sets can be created, composed, and manipulated with the various functions of the library.
You can explore and learn about the library with [`nix repl`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-repl):
@ -49,7 +48,7 @@ trace: /home/user (all files in directory)
null
```
All functions that expect a file set for an argument also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path) instead.
All functions that expect a file set for an argument can also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path).
Such path arguments are then [implicitly turned into sets](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion) that contain _all_ files under the given path.
In the previous trace this is indicated by `(all files in directory)`.
@ -65,14 +64,15 @@ trace: /home/user (all files in directory)
:::
Even though file sets conceptually contain local files, these files are *never* added to the Nix store unless explicitly requested.
So You don't have to worry as much about accidentally copying secrets into the world-readable store.
Therefore you don't have to worry as much about accidentally copying secrets into the world-readable store.
In this example, although we pretty-printed the home directory, no files were copied.
This is in contrast to coercion of paths to strings such as in `"${./.}"`,
which copies the whole directory to the Nix store on evaluation!
:::{warning}
When using [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), local directories with a `flake.nix` are always copied into the Nix store *completely* unless it is a Git repository!
When using the [`flakes` and `nix-command` experimental features](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake),
a local directory within a Flake is always copied into the Nix store *completely* unless it is a Git repository!
:::
This implicit coercion also works for files:
@ -129,10 +129,9 @@ $ echo world > world.txt
## Adding files to the Nix store
To add files in a given file set to the Nix store,
we use [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource).
This function needs a `root` attribute to know which path to use as the root of the resulting store path,
but only exactly the files in the `fileset` argument are included in the result.
Files in a given file set can be added to the Nix store with [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource).
The argument to this function requires a `root` attribute to determine which source directory to copy to the store.
Only the files in the `fileset` attribute are included in the result.
Define `build.nix` as follows:
@ -161,7 +160,11 @@ stdenv.mkDerivation {
The call to `fs.trace` prints the file set that will be used as a derivation input.
The build will succeed (note that it will take some time to fetch the first time around):
Try building it:
:::{note}
It will take a while to fetch Nixpkgs the first time around.
:::
```
$ nix-build
@ -244,7 +247,7 @@ result -> /nix/store/xknflcvjaa8dj6a6vkg629zmcrgz10rh-fileset
Since `src` refers to the whole directory, and its contents change when `nix-build` succeeds, Nix will have to start over every time.
:::{note}
This will also happen with `src = ./.`, without using the file set library.
This will also happen without the file set library, e.g. when setting `src = ./.;` directly.
:::
The [`difference`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.difference) function subtracts one file set from another.
@ -374,22 +377,17 @@ One way to fix this is to use [`unions`](https://nixos.org/manual/nixpkgs/unstab
Create a file set containing a union of the files to exclude (`fs.unions [ ... ]`), and subtract it (`difference`) from the complete directory (`./.`):
```{code-block} diff
```{code-block} nix
:caption: build.nix
{ stdenv, lib }:
let
fs = lib.fileset;
- sourceFiles = fs.difference ./. (fs.maybeMissing ./result);
+ sourceFiles =
+ fs.difference
+ ./.
+ (fs.unions [
+ (fs.maybeMissing ./result)
+ ./default.nix
+ ./build.nix
+ ./nix
+ ]);
in
sourceFiles =
fs.difference
./.
(fs.unions [
(fs.maybeMissing ./result)
./default.nix
./build.nix
./nix
]);
```
Changing any of the excluded files now doesn't necessarily cause a new build anymore:
@ -408,16 +406,12 @@ trace: - world.txt (regular)
## Filter
With the [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function,
you can find sets of files satisfying a criteria.
The [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function allows filtering file sets such that each included file satisfies the given criteria.
Use it to select all files with a name ending in `.nix`:
```{code-block} diff
:caption: build.nix
{ stdenv, lib }:
let
fs = lib.fileset;
sourceFiles =
fs.difference
./.
@ -428,7 +422,6 @@ Use it to select all files with a name ending in `.nix`:
+ (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.)
./nix
]);
in
```
This does not change the result, even if we add a new `.nix` file.
@ -458,47 +451,37 @@ $ touch build.sh src/select.{c,h}
Then create a file set from only the files to be included explicitly:
```{code-block} diff
```{code-block} nix
:caption: build.nix
{ stdenv, lib }:
let
fs = lib.fileset;
- sourceFiles =
- fs.difference
- ./.
- (fs.unions [
- (fs.maybeMissing ./result)
- (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.)
- ./nix
- ]);
+ sourceFiles = fs.unions [
+ ./hello.txt
+ ./world.txt
+ ./build.sh
+ (fs.fileFilter
+ (file:
+ lib.hasSuffix ".c" file.name
+ || lib.hasSuffix ".h" file.name
+ )
+ ./src
+ )
+ ];
in
{ stdenv, lib }:
let
fs = lib.fileset;
sourceFiles = fs.unions [
./hello.txt
./world.txt
./build.sh
(fs.fileFilter
(file:
lib.hasSuffix ".c" file.name
|| lib.hasSuffix ".h" file.name
)
./src
)
];
in
fs.trace sourceFiles
fs.trace sourceFiles
stdenv.mkDerivation {
name = "fileset";
src = fs.toSource {
root = ./.;
fileset = sourceFiles;
};
postInstall = ''
- mkdir $out
- cp -v {hello,world}.txt $out
+ cp -vr . $out
'';
}
stdenv.mkDerivation {
name = "fileset";
src = fs.toSource {
root = ./.;
fileset = sourceFiles;
};
postInstall = ''
cp -vr . $out
'';
}
```
The `postInstall` script is simplified to rely on the sources to be pre-filtered appropriately:
@ -552,30 +535,14 @@ $ git add -A
$ git reset src/select.o result
```
Now we can re-use this selection of files using `gitTracked`:
Re-use this selection of files with `gitTracked`:
```{code-block} diff
```{code-block} nix
:caption: build.nix
{ stdenv, lib }:
let
fs = lib.fileset;
- sourceFiles = fs.unions [
- ./hello.txt
- ./world.txt
- ./build.sh
- (fs.fileFilter
- (file:
- lib.hasSuffix ".c" file.name
- || lib.hasSuffix ".h" file.name
- )
- ./src
- )
- ];
+ sourceFiles = fs.gitTracked ./.;
in
sourceFiles = fs.gitTracked ./.;
```
Building we get
Build it again:
```shell-session
$ nix-build
@ -600,7 +567,8 @@ this derivation will be built:
This includes too much though, as not all of these files are needed to build the derivation as originally intended.
:::{note}
When using [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with
When using the [`flakes` and `nix-command` experimental features](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake),
it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with
```shell-session
@ -617,22 +585,17 @@ It allows creating a file set that consists only of files that are in _both_ of
Select all files that are both tracked by Git *and* relevant for the build:
```{code-block} diff
```{code-block} nix
:caption: build.nix
{ stdenv, lib }:
let
fs = lib.fileset;
- sourceFiles = fs.gitTracked ./.;
+ sourceFiles =
+ fs.intersection
+ (fs.gitTracked ./.)
+ (fs.unions [
+ ./hello.txt
+ ./world.txt
+ ./build.sh
+ ./src
+ ]);
in
sourceFiles =
fs.intersection
(fs.gitTracked ./.)
(fs.unions [
./hello.txt
./world.txt
./build.sh
./src
]);
```
This will produce the same output as in the other approach and therefore re-use a previous build result: