nix-dot-dev/nix.dev
1
1
Fork 0
mirror of https://github.com/nix-dot-dev/nix.dev.git synced 2024-08-17 17:20:44 +03:00
Code Issues Projects Releases Wiki Activity

Review in meeting

Browse Source
This commit is contained in:
Silvan Mosberger 2023-11-23 17:05:37 +01:00
parent fb3fb0c060
commit b6700a9c45
1 changed files with 71 additions and 108 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

179
source/tutorials/file-sets.md
View File

@ -20,13 +20,12 @@ Using these features directly can be tricky however:
can address these problems. can address these problems.
However, it's often hard to express the desired path selection using the `filter` function interface. However, it's often hard to express the desired path selection using the `filter` function interface.
In this tutorial you'll learn how to use Nixpkgs' [`lib.fileset` library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations. In this tutorial you'll learn how to use the Nixpkgs [`lib.fileset` library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations.
It abstracts over built-in functionality with a safer and more convenient interface. It abstracts over built-in functionality and offers a safer and more convenient interface.
## File sets ## File sets
The basic concept is that of a _file set_, A _file set_ is a data type representing a collection of local files.
a data type representing a collection of local files.
File sets can be created, composed, and manipulated with the various functions of the library. File sets can be created, composed, and manipulated with the various functions of the library.
You can explore and learn about the library with [`nix repl`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-repl): You can explore and learn about the library with [`nix repl`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-repl):
@ -49,7 +48,7 @@ trace: /home/user (all files in directory)
null null
``` ```
All functions that expect a file set for an argument also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path) instead. All functions that expect a file set for an argument can also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path).
Such path arguments are then [implicitly turned into sets](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion) that contain _all_ files under the given path. Such path arguments are then [implicitly turned into sets](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion) that contain _all_ files under the given path.
In the previous trace this is indicated by `(all files in directory)`. In the previous trace this is indicated by `(all files in directory)`.
@ -65,14 +64,15 @@ trace: /home/user (all files in directory)
::: :::
Even though file sets conceptually contain local files, these files are *never* added to the Nix store unless explicitly requested. Even though file sets conceptually contain local files, these files are *never* added to the Nix store unless explicitly requested.
So You don't have to worry as much about accidentally copying secrets into the world-readable store. Therefore you don't have to worry as much about accidentally copying secrets into the world-readable store.
In this example, although we pretty-printed the home directory, no files were copied. In this example, although we pretty-printed the home directory, no files were copied.
This is in contrast to coercion of paths to strings such as in `"${./.}"`, This is in contrast to coercion of paths to strings such as in `"${./.}"`,
which copies the whole directory to the Nix store on evaluation! which copies the whole directory to the Nix store on evaluation!
:::{warning} :::{warning}
When using [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), local directories with a `flake.nix` are always copied into the Nix store *completely* unless it is a Git repository! When using the [`flakes` and `nix-command` experimental features](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake),
a local directory within a Flake is always copied into the Nix store *completely* unless it is a Git repository!
::: :::
This implicit coercion also works for files: This implicit coercion also works for files:
@ -129,10 +129,9 @@ $ echo world > world.txt
## Adding files to the Nix store ## Adding files to the Nix store
To add files in a given file set to the Nix store, Files in a given file set can be added to the Nix store with [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource).
we use [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource). The argument to this function requires a `root` attribute to determine which source directory to copy to the store.
This function needs a `root` attribute to know which path to use as the root of the resulting store path, Only the files in the `fileset` attribute are included in the result.
but only exactly the files in the `fileset` argument are included in the result.
Define `build.nix` as follows: Define `build.nix` as follows:
@ -161,7 +160,11 @@ stdenv.mkDerivation {
The call to `fs.trace` prints the file set that will be used as a derivation input. The call to `fs.trace` prints the file set that will be used as a derivation input.
The build will succeed (note that it will take some time to fetch the first time around): Try building it:
:::{note}
It will take a while to fetch Nixpkgs the first time around.
:::
``` ```
$ nix-build $ nix-build
@ -244,7 +247,7 @@ result -> /nix/store/xknflcvjaa8dj6a6vkg629zmcrgz10rh-fileset
Since `src` refers to the whole directory, and its contents change when `nix-build` succeeds, Nix will have to start over every time. Since `src` refers to the whole directory, and its contents change when `nix-build` succeeds, Nix will have to start over every time.
:::{note} :::{note}
This will also happen with `src = ./.`, without using the file set library. This will also happen without the file set library, e.g. when setting `src = ./.;` directly.
::: :::
The [`difference`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.difference) function subtracts one file set from another. The [`difference`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.difference) function subtracts one file set from another.
@ -374,22 +377,17 @@ One way to fix this is to use [`unions`](https://nixos.org/manual/nixpkgs/unstab
Create a file set containing a union of the files to exclude (`fs.unions [ ... ]`), and subtract it (`difference`) from the complete directory (`./.`): Create a file set containing a union of the files to exclude (`fs.unions [ ... ]`), and subtract it (`difference`) from the complete directory (`./.`):
```{code-block} diff ```{code-block} nix
:caption: build.nix :caption: build.nix
{ stdenv, lib }: sourceFiles =
let fs.difference
fs = lib.fileset; ./.
- sourceFiles = fs.difference ./. (fs.maybeMissing ./result); (fs.unions [
+ sourceFiles = (fs.maybeMissing ./result)
+ fs.difference ./default.nix
+ ./. ./build.nix
+ (fs.unions [ ./nix
+ (fs.maybeMissing ./result) ]);
+ ./default.nix
+ ./build.nix
+ ./nix
+ ]);
in
``` ```
Changing any of the excluded files now doesn't necessarily cause a new build anymore: Changing any of the excluded files now doesn't necessarily cause a new build anymore:
@ -408,16 +406,12 @@ trace: - world.txt (regular)
## Filter ## Filter
With the [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function, The [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function allows filtering file sets such that each included file satisfies the given criteria.
you can find sets of files satisfying a criteria.
Use it to select all files with a name ending in `.nix`: Use it to select all files with a name ending in `.nix`:
```{code-block} diff ```{code-block} diff
:caption: build.nix :caption: build.nix
{ stdenv, lib }:
let
fs = lib.fileset;
sourceFiles = sourceFiles =
fs.difference fs.difference
./. ./.
@ -428,7 +422,6 @@ Use it to select all files with a name ending in `.nix`:
+ (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.) + (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.)
./nix ./nix
]); ]);
in
``` ```
This does not change the result, even if we add a new `.nix` file. This does not change the result, even if we add a new `.nix` file.
@ -458,47 +451,37 @@ $ touch build.sh src/select.{c,h}
Then create a file set from only the files to be included explicitly: Then create a file set from only the files to be included explicitly:
```{code-block} diff ```{code-block} nix
:caption: build.nix :caption: build.nix
{ stdenv, lib }: { stdenv, lib }:
let let
fs = lib.fileset; fs = lib.fileset;
- sourceFiles = sourceFiles = fs.unions [
- fs.difference ./hello.txt
- ./. ./world.txt
- (fs.unions [ ./build.sh
- (fs.maybeMissing ./result) (fs.fileFilter
- (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.) (file:
- ./nix lib.hasSuffix ".c" file.name
- ]); || lib.hasSuffix ".h" file.name
+ sourceFiles = fs.unions [ )
+ ./hello.txt ./src
+ ./world.txt )
+ ./build.sh ];
+ (fs.fileFilter in
+ (file:
+ lib.hasSuffix ".c" file.name
+ || lib.hasSuffix ".h" file.name
+ )
+ ./src
+ )
+ ];
in
fs.trace sourceFiles fs.trace sourceFiles
stdenv.mkDerivation { stdenv.mkDerivation {
name = "fileset"; name = "fileset";
src = fs.toSource { src = fs.toSource {
root = ./.; root = ./.;
fileset = sourceFiles; fileset = sourceFiles;
}; };
postInstall = '' postInstall = ''
- mkdir $out cp -vr . $out
- cp -v {hello,world}.txt $out '';
+ cp -vr . $out }
'';
}
``` ```
The `postInstall` script is simplified to rely on the sources to be pre-filtered appropriately: The `postInstall` script is simplified to rely on the sources to be pre-filtered appropriately:
@ -552,30 +535,14 @@ $ git add -A
$ git reset src/select.o result $ git reset src/select.o result
``` ```
Now we can re-use this selection of files using `gitTracked`: Re-use this selection of files with `gitTracked`:
```{code-block} diff ```{code-block} nix
:caption: build.nix :caption: build.nix
{ stdenv, lib }: sourceFiles = fs.gitTracked ./.;
let
fs = lib.fileset;
- sourceFiles = fs.unions [
- ./hello.txt
- ./world.txt
- ./build.sh
- (fs.fileFilter
- (file:
- lib.hasSuffix ".c" file.name
- || lib.hasSuffix ".h" file.name
- )
- ./src
- )
- ];
+ sourceFiles = fs.gitTracked ./.;
in
``` ```
Building we get Build it again:
```shell-session ```shell-session
$ nix-build $ nix-build
@ -600,7 +567,8 @@ this derivation will be built:
This includes too much though, as not all of these files are needed to build the derivation as originally intended. This includes too much though, as not all of these files are needed to build the derivation as originally intended.
:::{note} :::{note}
When using [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with When using the [`flakes` and `nix-command` experimental features](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake),
it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with
```shell-session ```shell-session
@ -617,22 +585,17 @@ It allows creating a file set that consists only of files that are in _both_ of
Select all files that are both tracked by Git *and* relevant for the build: Select all files that are both tracked by Git *and* relevant for the build:
```{code-block} diff ```{code-block} nix
:caption: build.nix :caption: build.nix
{ stdenv, lib }: sourceFiles =
let fs.intersection
fs = lib.fileset; (fs.gitTracked ./.)
- sourceFiles = fs.gitTracked ./.; (fs.unions [
+ sourceFiles = ./hello.txt
+ fs.intersection ./world.txt
+ (fs.gitTracked ./.) ./build.sh
+ (fs.unions [ ./src
+ ./hello.txt ]);
+ ./world.txt
+ ./build.sh
+ ./src
+ ]);
in
``` ```
This will produce the same output as in the other approach and therefore re-use a previous build result: This will produce the same output as in the other approach and therefore re-use a previous build result: