nix-community/dream2nix
1
1
Fork 0
mirror of https://github.com/nix-community/dream2nix.git synced 2024-12-31 18:31:51 +03:00
Code Issues Projects Releases Wiki Activity

Merge pull request #1013 from phaer/guides

Browse Source 
Add pip & devshells guide
This commit is contained in:
Paul Haerle 2024-07-29 13:25:48 +02:00 committed by GitHub
commit ece62d699f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
17 changed files with 393 additions and 40 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/README.md
View File

@ -32,7 +32,7 @@ i.e. from inside the shell in `./docs`:
``` shellsession
# update
ln -sfT $(nix build --print-out-paths --no-link ..#optionsReference) ./src/reference
ln -sfT $(nix build --print-out-paths --no-link .#optionsReference) ./src/reference
# remove
rm ./src/reference
```

1
docs/src/guides/.pages
View File

@ -1,2 +1,3 @@
nav:
- getting-started.md
- "Python & pip": pip.md

14
docs/src/guides/getting-started.md
View File

@ -18,7 +18,7 @@ We start by creating a new git repository with the following two files:
- `hello.nix` declares a dream2nix module that imports [mkDerivation](../reference/mkDerivation/index.md) and
uses that to build GNU hello.
Check out the code below and its annotations to learn more!
Check out the code below and don't miss the annotations, hidden behind those little plusses, to learn more!
!!! note
@ -93,14 +93,6 @@ Check out the code below and its annotations to learn more!
version = "2.12.1";
# (4)
# deps = {nixpkgs, ...}: {
# inherit
# (nixpkgs)
# stdenv
# ;
# };
# (5)
mkDerivation = {
src = builtins.fetchTarball {
url = "https://ftp.gnu.org/gnu/hello/hello-${config.version}.tar.gz";
@ -114,9 +106,7 @@ Check out the code below and its annotations to learn more!
Inputs include `dream2nix`, a reference to package itself in `config`, and the nixpkgs library in `lib`.
2. Import the [`mkDerivation`](../reference/mkDerivation/index.md) module.
3. Define `name` and `version` of the package. Unlike most other options, those are not namespaced and defined in dream2nix `core` module.
4. Define a function that returns **dep**endencie**s** as *module options* from the given *package sets*.
`hello` here needs a compiler already in the defaults of `mkDerivation`, we don't need `deps` here but include it for demonstration purposes.
5. Define *module options* to further customize your build. In this case we just set `mkDerivation.src` to fetch a source tarball as well.
4. Define *module options* to further customize your build. In this case we just set `mkDerivation.src` to fetch a source tarball as well.
But you could use other arguments from `pkgs.mkDerivation`, such as `buildInputs` or `buildPhase` here as well.
## Build it

336
docs/src/guides/pip.md Normal file
View File

@ -0,0 +1,336 @@
---
title: Build a python project with pip
---
!!! info
We recommend reading our [Getting Started](./getting-started.md) guide first if you have not done so yet!
this guide we are going to take a look at two annotated examples using the [pip module](../reference/pip/index.md):
- The first one builds [Pillow](https://python-pillow.org/) from upstream sources fetched from PyPi.
- The second one builds a fictional python project living in the same repository as the nix sources
and a development environment around it.
## Start with a flake
We start both examples by creating a new git repository and adding almost the same `flake.nix` template we already used in [Getting Started](./getting-started.md#start-a-project). The only difference
are the packages name, `default` instead of `hello`:
```nix title="flake.nix"
{
inputs = {
dream2nix.url = "github:nix-community/dream2nix";
nixpkgs.follows = "dream2nix/nixpkgs";
};
outputs = {
self,
dream2nix,
nixpkgs,
}:
let
eachSystem = nixpkgs.lib.genAttrs [
"aarch64-darwin"
"aarch64-linux"
"x86_64-darwin"
"x86_64-linux"
];
in {
packages = eachSystem (system: {
default = dream2nix.lib.evalModules { # (1)
packageSets.nixpkgs = nixpkgs.legacyPackages.${system};
modules = [
./default.nix # (2)
{
paths.projectRoot = ./.;
paths.projectRootFile = "flake.nix";
paths.package = ./.;
}
];
};
});
}
```
1. We call our package attribute `default` here...
2. ...and the nix file `default.nix` here.
## Example: Pillow
Things get a bit more interesting in `default.nix` where we define a package module which fetches Pillow from pypi and builds it with minimal features - just JPEG support. Click the plus to expand any code annotation below for details.
The code we are going to end up with is also available in [./examples/packages/languages/python-packaging-pillow](https://github.com/nix-community/dream2nix/tree/main/examples/packages/languages/python-packaging-pillow).
### Code
```nix title="default.nix"
{
config,
lib,
dream2nix,
...
}: {
imports = [
dream2nix.modules.dream2nix.pip # (1)
];
deps = {nixpkgs, ...}: {
python = nixpkgs.python3; # (2)
inherit # (3)
(nixpkgs)
pkg-config
zlib
libjpeg
;
};
name = "pillow"; # (4)
version = "10.4.0";
mkDerivation = { # (5)
nativeBuildInputs = [
config.deps.pkg-config
];
propagatedBuildInputs = [
config.deps.zlib
config.deps.libjpeg
];
};
buildPythonPackage = { # (6)
pythonImportsCheck = [
"PIL"
];
};
pip = {
requirementsList = ["${config.name}==${config.version}"]; # (7)
pipFlags = [ # (8)
"--no-binary"
":all:"
];
};
}
```
1. Import the dream2nix [pip module](../reference/pip/index.md) into our module.
2. Declare external dependencies, like the python interpreter to use and libraries from nixpkgs. We use whatever the latest `python3` in nixpkgs is as our python.
3. Declare which build tools we need to pull from nixpkgs for use in `mkDerivation` below.
4. Declare name and version of our package. Those will also be used for `pip.requirementsList` below.
5. Set dependencies, `pkg-config` is only required
during build-time, while the libraries should be propagated. We use `config.deps` instead of a conventional `pkg` here to be able to "override" inputs via the [module system](../modules.md).
6. Tell the [buildPythonPackage module](../reference/buildPythonPackage/index.md) to verify that it can import the given python module from our package after a build.
7. Tell the [pip module](../reference/pip/index.md) which dependencies to lock using the same syntax as
a `requirements.txt` file. Here: `pillow==10.4.0`.
8. `pip` uses binary wheel files if available by default. We will not do so in order to ensure a build from source.
### Initialise the repostory
If you use `git`, you need to add `flake.nix` and `default.nix` to your git index so that they get copied to the `/nix/store` and the commands below see them:
```shell-session
$ git init
$ git add flake.nix default.nix
```
### Create a lock file
The next step is to create a lock file by running the packages `lock` attribute. This does a `pip install --dry-run` under the hood and pins the exact packages pip would install.
```shell-session
$ nix run .#default.lock
$ git add lock.json
```
!!! note
Due to limitations in `pip`s cross-platform support, the resulting
lock-files are platform-specific!
We therefore recommend setting `paths.lockFile` to `lock.${system}.json`
for all projects where you use the pip module.
Check out the [pdm module](../reference/WIP-python-pdm/index.md) if you need a solution that
allows locking for multiple platforms at once!
### Build it
After that's done, we can finally build it:
```shell-session
$ nix build .#default
```
Congratulations, you just built your first python package with dream2nix! The resulting package can be used with any other nix python package as long as it uses the same version of python.
## Example: my-tool
In our second example, we package are going to package a simple, fictional python package called `my_tool`. Its code and nix expressions
are stored in the same repository. For reference, they are available in full in [./examples/packages/languages/python-local-development](https://github.com/nix-community/dream2nix/tree/main/examples/packages/languages/python-local-development).
### Code
```nix title="default.nix"
{
config,
lib,
dream2nix,
...
}: let
pyproject = lib.importTOML (config.mkDerivation.src + /pyproject.toml); # (1)
in {
imports = [
dream2nix.modules.dream2nix.pip # (2)
];
deps = {nixpkgs, ...}: {
python = nixpkgs.python3; # (3)
};
inherit (pyproject.project) name version; # (4)
mkDerivation = {
src = lib.cleanSourceWith { # (5)
src = lib.cleanSource ./.;
filter = name: type:
!(builtins.any (x: x) [
(lib.hasSuffix ".nix" name)
(lib.hasPrefix "." (builtins.baseNameOf name))
(lib.hasSuffix "flake.lock" name)
]);
};
};
buildPythonPackage = {
pyproject = true; # (6)
pythonImportsCheck = [ # (7)
"my_tool"
];
};
pip = {
# (8)
requirementsList =
pyproject.build-system.requires or []
++ pyproject.project.dependencies or [];
flattenDependencies = true; # (9)
overrides.click = { # (10)
buildPythonPackage.pyproject = true;
mkDerivation.nativeBuildInputs = [config.deps.python.pkgs.flit-core];
};
};
}
```
1. Load `pyproject.toml` from our source directory, which is the filtered
source defined in `mkDerivation.src` below.
2. Import the dream2nix [pip module](../reference/pip/index.md) into our module
3. Define external, non-python dependencies. We use whatever the latest `python3` in nixpkgs is as our python.
4. Get our projects `name` and `version` straight from `pyproject.toml`. You could of course also hard-code them here if e.g. your project still uses `setup.py`.
5. Define the source for our project. Here we take the current directory, but filter out `*.nix` files, hidden files and `flake.lock` before copying to `/nix/store` in order to avoid unecessary rebuilds.
6. Tell the dream2nix [buildPythonPackage module](../reference/buildPythonPackage/index.md), imported by the pip module to use pyproject-specific hooks here.
Don't set it if your project doesn't include a `pyproject.toml` or your are using a wheel.
7. Tell the [buildPythonPackage module](../reference/buildPythonPackage/index.md) to verify that it can import the given python module from our package after a build.
8. Declare a list of requirements for `pip` to lock by concatenating
both the build-systems and normal dependencies in `pyproject.toml`.
9. By default, the [pip module](../reference/pip/index.md) assumes that it finds the top-level package inside the lock file. This isn't the case
here as the top-level package comes from the local repository. So we
instruct the module to just install all requirements into a flat environment.
10. Declare overrides for package attributes that can't be detected heuristically by dream2nix yet. Here: use pyproject-hooks for click and
add `poetry-core` to its build-time dependencies.
### Build it
Just as in the same example, we need to lock our python dependencies and add the lock file before we build our package:
```shell-session
$ git init
$ git add flake.nix default.nix
$ nix run .#default.lock
[...]
lock file written to [...]/lock.x86_64-linux.json
Add this file to git if flakes is used.
$ git add lock.json
$ nix build .#
$ ./result/bin/my_tool
Hello world!
```
# Development Shells
Now that we got `my_tool` built, let's try out a *devShell* for it. A shell environment, containing an [editable install](#editable-installs) of our
package , all its dependencies and scripts as well as other tools that are useful during development, but shouldn't end up in shipped packages.
!!! notice
If you use a *flat layout*, i.e. your python module is in the top-level of your repo you
might discover that you can just import it if you start a python process there.
This works because python searches the current directory for modules, but it will miss its
dependencies and scripts declared in `pyproject.toml` won't be on the path.
To get started, add the following definition to your `flake.nix` (it's the same as in the [example](https://github.com/nix-community/dream2nix/blob/main/examples/packages/languages/python-local-development/flake.nix), so we omit some sections marked with `[...]`). Click on the pluses to expand annotations in the code below:
```nix title="flake.nix"
{
# [...]
outputs = {
self,
dream2nix,
nixpkgs,
}: {
# [...]
devShells = eachSystem (system:
let
pkgs = nixpkgs.legacyPackages.${system}; # (1)
my_tool = self.packages.${system}.default; # (2)
python = my_tool.config.deps.python; # (3)
in {
default = pkgs.mkShell { # (4)
inputsFrom = [my_tool.devShell]; # (5)
packages = [
python.pkgs.python-lsp-server # (6)
python.pkgs.python-lsp-ruff
python.pkgs.pylsp-mypy
python.pkgs.ipython
pkgs.ruff # (7)
pkgs.black
];
};
});
};
}
```
1. Get an instance of `nixpkgs` to use for `mkShell`, `ruff`, etc below.
2. Get our default package, `my_tool` from this flakes outputs...
3. ...and a reference to the python interpreter it uses.
4. Call `pkgs.mkShell` in order to be able to add custom `packages` or a `shellHook` to it. You could just use `my_tool.devShell` here if you are happy with the defaults.
5. Get inputs from `mytool.devShell`, this includes your package & its dependencies and a `shellHook` to set up editable installs.
6. Use `python.pkgs` to get additional python tools into the shell while ensuring that the correct python interpreter is used.
7. Use `pkgs` to get tools which aren't implemented in python, such as `ruff` and `black`.
With that done, let's start a development shell, an ipython interpreter in it, import `my_tool` and `requests`, its dependency and
see where they are imported from:
```shell-session
$ nix develop
Some python dependencies of my-tool are installed in editable mode
To disable editable mode for a package, remove the corresponding entry from the 'editables' field in the dream2nix configuration file.
$ ipython
[...]
In [1]: import my_tool
In [2]: my_tool.__file__
Out[2]: '[path_to_your_repo]/src/my_tool/__init__.py'
In [3]: import requests
In [4]: requests.__file__
Out[4]: '/nix/store/[nix hash]-python3-3.11.9-env/lib/python3.11/site-packages/requests/__init__.py'
```
* `my_tool` is imported from your repository, as it's an editable install, akin to `pip install -e .`
* `requests` is loaded from a python environment, generated by our expression above. Changes to it will only be visible after the shell is restarted.
All the other tools declared in our `devShell` above, i.e. `ruff` and `black` should be in `PATH` and/or `PYTHONPATH` as well.

2
examples/packages/languages/python-local-development-pdm/default.nix
View File

@ -25,7 +25,7 @@
buildPythonPackage = {
pythonImportsCheck = [
"mytool"
"my_tool"
];
};
}

0
examples/packages/languages/python-local-development-pdm/mytool/__init__.py → examples/packages/languages/python-local-development-pdm/my_tool/__init__.py
View File

2
examples/packages/languages/python-local-development-pdm/pyproject.toml
View File

@ -3,7 +3,7 @@ requires = [ "setuptools" ]
build-backend = "setuptools.build_meta"
[project]
name = "mytool"
name = "my-tool"
description = "my tool"
version = "1.0.0"
dependencies = [

12
examples/packages/languages/python-local-development/default.nix
View File

@ -32,18 +32,16 @@ in {
buildPythonPackage = {
pyproject = true;
pythonImportsCheck = [
"mytool"
"my_tool"
];
};
paths.lockFile = "lock.${config.deps.stdenv.system}.json";
pip = {
# Setting editables.$pkg.null will link the current project root as an editable
# for the root package (my-tool here), or otherwise copy the contents of mkDerivation.src
# to .dream2nix/editables to make them writeable.
# Alternatively you can point it to an existing checkout via an absolute path, i.e.:
# editables.charset-normalizer = "/home/my-user/src/charset-normalizer";
editables.charset-normalizer = ".editables/charset_normalizer";
# Setting editables.$pkg to an absolute path will link this path as an editable
# install to .dream2nix/editables in devShells. The root package is always installed
# as editable.
# editables.charset-normalizer = "/home/my-user/src/charset-normalizer";
requirementsList =
pyproject.build-system.requires

19
examples/packages/languages/python-local-development/flake.nix
View File

@ -46,13 +46,22 @@
];
};
});
devShells = eachSystem (system: {
default = nixpkgs.legacyPackages.${system}.mkShell {
devShells = eachSystem (system: let
pkgs = nixpkgs.legacyPackages.${system};
my_tool = self.packages.${system}.default;
python = my_tool.config.deps.python;
in {
default = pkgs.mkShell {
# inherit from the dream2nix generated dev shell
inputsFrom = [self.packages.${system}.default.devShell];
# add extra packages
inputsFrom = [my_tool.devShell];
packages = [
nixpkgs.legacyPackages.${system}.hello
python.pkgs.python-lsp-server
python.pkgs.python-lsp-ruff
python.pkgs.pylsp-mypy
python.pkgs.ipython
pkgs.ruff
pkgs.black
];
};
});

2
examples/packages/languages/python-local-development/pyproject.toml
View File

@ -3,7 +3,7 @@ requires = [ "setuptools" ]
build-backend = "setuptools.build_meta"
[project]
name = "mytool"
name = "my-tool"
description = "my tool"
version = "1.0.0"
dependencies = [

0
examples/packages/languages/python-local-development/src/mytool/__init__.py → examples/packages/languages/python-local-development/src/my_tool/__init__.py
View File

4
examples/packages/languages/python-packaging-pillow/default.nix
View File

@ -11,7 +11,7 @@
];
deps = {nixpkgs, ...}: {
python = nixpkgs.python39;
python = nixpkgs.python311;
inherit
(nixpkgs)
pkg-config
@ -21,7 +21,7 @@
};
name = "pillow";
version = "9.5.0";
version = "10.4.0";
mkDerivation = {
nativeBuildInputs = [

8
examples/packages/languages/python-packaging-pillow/lock.aarch64-darwin.json
View File

@ -3,10 +3,10 @@
"sources": {
"pillow": {
"is_direct": false,
"sha256": "bf548479d336726d7a0eceb6e767e179fbde37833ae42794602631a070d630f1",
"sha256": "166c1cd4d24309b30d61f79f4a9114b7b2313d7450912277855ff5dfd7cd4a06",
"type": "url",
"url": "https://files.pythonhosted.org/packages/00/d5/4903f310765e0ff2b8e91ffe55031ac6af77d982f0156061e20a4d1a8b2d/Pillow-9.5.0.tar.gz",
"version": "9.5.0"
"url": "https://files.pythonhosted.org/packages/cd/74/ad3d526f3bf7b6d3f408b73fde271ec69dfac8b81341a318ce825f2b3812/pillow-10.4.0.tar.gz",
"version": "10.4.0"
}
},
"targets": {
@ -15,5 +15,5 @@
}
}
},
"invalidationHash": "e475932e083c19152414893b4fe55b06436717b5bfe7ee367a828b48aad19a61"
"invalidationHash": "602362bfb149c7d9ffaa19a96f5bd1cf296e12c83b378bbd913e3235b355521b"
}

8
examples/packages/languages/python-packaging-pillow/lock.x86_64-linux.json
View File

@ -3,10 +3,10 @@
"sources": {
"pillow": {
"is_direct": false,
"sha256": "bf548479d336726d7a0eceb6e767e179fbde37833ae42794602631a070d630f1",
"sha256": "166c1cd4d24309b30d61f79f4a9114b7b2313d7450912277855ff5dfd7cd4a06",
"type": "url",
"url": "https://files.pythonhosted.org/packages/00/d5/4903f310765e0ff2b8e91ffe55031ac6af77d982f0156061e20a4d1a8b2d/Pillow-9.5.0.tar.gz",
"version": "9.5.0"
"url": "https://files.pythonhosted.org/packages/cd/74/ad3d526f3bf7b6d3f408b73fde271ec69dfac8b81341a318ce825f2b3812/pillow-10.4.0.tar.gz",
"version": "10.4.0"
}
},
"targets": {
@ -15,5 +15,5 @@
}
}
},
"invalidationHash": "e475932e083c19152414893b4fe55b06436717b5bfe7ee367a828b48aad19a61"
"invalidationHash": "602362bfb149c7d9ffaa19a96f5bd1cf296e12c83b378bbd913e3235b355521b"
}

7
modules/dream2nix/pip/pip-hotfixes/interface.nix
View File

@ -10,7 +10,12 @@ in {
flattenDependencies = l.mkOption {
type = t.bool;
description = ''
Use all dependencies as top-level dependencies
Use all dependencies as top-level dependencies, even transitive ones.
Without this, we would walk the dependency tree from the root package upwards,
adding only the necessary packages to each dependency. With this, it's flat.
Useful if we are mostly interested in a working environment.
'';
default = false;
};

14
modules/dream2nix/python-editables/interface.nix
View File

@ -7,10 +7,24 @@
in {
options = {
editables = lib.mkOption {
description = ''
An attribute set mapping package names to absolute paths of source directories
which should be installed in editable mode in [editablesShellHook](#pipeditablesshellhook).
i.e.
```
pip.editables.charset-normalizer = "/home/user/src/charset-normalizer".
```
The top-level package is added automatically.
'';
type = t.attrsOf t.str;
};
editablesShellHook = lib.mkOption {
description = ''
A shellHook to be included into your devShells to install [editables](#pipeditables)
'';
type = t.str;
readOnly = true;
};

2
modules/flake-parts/reference-website/default.nix
View File

@ -147,7 +147,7 @@
shellHook = ''
cd $PRJ_ROOT/docs
if [ ! -d src/reference ]; then
echo "linking .#reference to src/reference, you need to update this manually\
echo "linking .#optionsReference to src/reference, you need to update this manually\
and remove it before a production build"
ln -sfT $(nix build .#optionsReference --no-link --print-out-paths) src/reference
fi