diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index 7dbaccec..7ba2605d 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -31,3 +31,16 @@ # Development Roundups - [April - June 2022](./development-roundups/2022-april-june.md) - [July - September 2022](./development-roundups/2022-july-september.md) + +# WIP +- [v1 API](./v1-api/summary.md) + - [problems of the current dream2nix](./v1-api/problems.md) + - [users of dream2nix](./v1-api/users.md) + - [v1 packaging: project initialization](./v1-api/packaging/nodejs-init-project.md) + - [v1 packaging: workspaces](./v1-api/packaging/nodejs-workspaces.md) + - [v1 packaging: multiple repos](./v1-api/packaging/nodejs-multiple-repos.md) + - [v1 packaging: monorepo](./v1-api/packaging/monorepo.md) + - [v1 consuming: inspect package options](./v1-api/consuming/inspect-options.md) + - [v1 consuming: override packages](./v1-api/consuming/override.md) + - [v1 integrating: lang2nix tool (pure)](./v1-api/integrating/integrate-lang2nix-pure.md) + - [v1 integrating: lang2nix tool (code-gen/impure)](./v1-api/integrating/integrate-lang2nix-impure.md) diff --git a/docs/src/v1-api/consuming/inspect-options.md b/docs/src/v1-api/consuming/inspect-options.md new file mode 100644 index 00000000..c08a6b34 --- /dev/null +++ b/docs/src/v1-api/consuming/inspect-options.md @@ -0,0 +1,28 @@ +# Inspect the API of a package +Downstream users can inspect the api of any consumed package as well as raw package modules + +## Load the dream2nix shell +```shell +nix-shell https://dream2nix.dev -A devShells.default +``` + +## Get manual of package module +Assuming a package module in `./upstream/my-package.nix` + +```shell +$ dream2nix man ./upstream/my-package.nix +``` + +## Get manual of derivation +Assuming derivations defined via `./upstream/default.nix` + +```shell +dream2nix man ./upstream/default.nix -A packages.my-package +``` + +## Get manual of flake attribute +Assuming derivations defined via a flake on github + +```shell +dream2nix man github:user/repo#some-package +``` diff --git a/docs/src/v1-api/consuming/override.md b/docs/src/v1-api/consuming/override.md new file mode 100644 index 00000000..05189643 --- /dev/null +++ b/docs/src/v1-api/consuming/override.md @@ -0,0 +1,198 @@ +# Consume and modify dream2nix packages + +## Given the following package +`upstream/my-package.nix` +```nix +{config, lib, dream2nix, ...}: { + + imports = [ + dream2nix.modules.nodejs.mkDerivation + dream2nix.modules.nodejs.package-lock + ]; + + pname = "my-package"; + version = "2.0.0"; + + src = { + type = github; + owner = "my-user"; + repo = "my-repo"; + ref = config.version; + hash = "sha256-mia90VYv/YTdWNhKpvwvFW9RfbXZJSWhJ+yva6EnLE8="; + }; + + # declare dependency on python3 + deps = {nixpkgs, ...}: { + python3 = nixpkgs.python39; + }; + + nativeBuildInputs = [ + config.deps.python3 + ]; + + configurePhase = '' + python3 --version + ''; + + buildPhase = '' + python3 -c 'print("Hello World!")' > $out + ''; +} +``` + +`upstream/default.nix` +```nix +{ + nixpkgs ? import {}, + dream2nix ? + import + (builtins.fetchTarball "https://dream2nix.dev/tarball/1.0") + {inherit nixpkgs;}, +}: { + packages.my-package = dream2nix.eval ./my-package.nix; +} +``` + +## 1. Override using modules + +### 1.1 Define a module for the override +`my-package-override.nix` +```nix +{config, lib, ... }: { + + version = "2.1.0"; + + # No need to re-define other fetcher attributes. + # The module system updates them for us. + src.hash = "sha256-LM5GDNjLcmgZVQEeANWAOO09KppwGaYEzJBjYmuSwys="; + + deps = {nixpkgs, ...}: { + + # change the python version + python3 = lib.mkForce nixpkgs.python310; + + # add a dependency on hello + hello = nixpkgs.hello; + }; + + # add hello to nativeBuildInputs + # (`oldAttrs.nativeBuildInputs + ...` not needed here) + nativeBuildInputs = [ + config.deps.hello + ]; + + # add lines to configurePhase + postConfigure = '' + hello --version + ''; + + # replace the build phase via mkForce + buildPhase = lib.mkForce " + hello > $out + "; +} +``` + +### 1.2 Apply `my-package-override.nix` via extendModules +Using `extendModules` is simple. +It allows to extend an existing package with another module. +This doesn't require knowledge about the original modules that went into the package. + +`./default.nix` +```nix +let + nixpkgs = import {}; + upstream = import ./upstream {inherit nixpkgs;}; + my-package = upstream.packages.my-package; + + # The recommended way of modifying a package is using extendModules, + # which uses the module systems merge logic to apply changes. + my-package-extended = my-package.extendModules { + modules = [./my-package-override.nix]; + }; + +in { + inherit my-package-extended; +} +``` + +### 1.3 Or apply `my-package-override.nix` via dream2nix.eval +This approach is a bit cleaner. +It doesn't introduce a chain of extendModules function calls. +This style also makes it obvious which modules went into the package. +Though, this requires access to the original `my-package.nix` module and knowledge about the `packageSets` that went into it. + +`default.nix` +```nix +{ + nixpkgs ? import {}, + dream2nix ? + import + (builtins.fetchTarball "https://dream2nix.dev/tarball/1.0") + {inherit nixpkgs;}, + +}: let + + my-package-extended = dream2nix.eval + {packagetSets = {inherit nixpkgs;};} + [ + ./upstream/my-package.nix + ./my-package-override.nix + ]; + +in { + my-package-extended +} +``` + + +## 2. Override package via `override[Attrs]` functions + +It is recommended to use modules for overriding, like described above, but for backward compatibility, `overrideAttrs` and `override` are still supported. + +```nix +let + nixpkgs = import {}; + upstream = import ./upstream {inherit nixpkgs;}; + my-package = upstream.packages.my-package; + + # Override the package via `override` and `overrideAttrs` + my-package-overridden' = my-package.override + (oldAttrs: { + + # change the python version + python3 = nixpkgs.python310; + }); + + my-package-overridden = my-package-overridden'.overrideAttrs + (oldAttrs: rec { + + version = "2.1.0"; + + src = nixpkgs.fetchFromGithub { + owner = "my-owner"; + repo = "my-repo"; + ref = version; + hash = "sha256-LM5GDNjLcmgZVQEeANWAOO09KppwGaYEzJBjYmuSwys="; + }; + + # add hello to nativeBuildInputs + nativeBuildInputs = [ + nixpkgs.hello + ]; + + # add lines to configurePhase + postConfigure = '' + hello --version + ''; + + # replace the build phase + buildPhase = '' + hello > $out + ''; + }); + +in { + inherit my-package-overridden; +} +``` diff --git a/docs/src/v1-api/integrating/integrate-lang2nix-impure.md b/docs/src/v1-api/integrating/integrate-lang2nix-impure.md new file mode 100644 index 00000000..fcfd0c55 --- /dev/null +++ b/docs/src/v1-api/integrating/integrate-lang2nix-impure.md @@ -0,0 +1,58 @@ +# Integrate lang2nix tool (impure/code-gen) + +We use [gomod2nix](https://github.com/nix-community/gomod2nix) as an example here to demonstrate creating a dream2nix integration. + +Gomod2nix is a nix code generator that requires network access, a great example for an impure dream2nix integration. + +`dream2nix/modules/go.gomod2nix.nix` +```nix +{config, lib, dream2nix, system, ...}: rec { + + imports = [ + + # import generic mkDerivation interface, which will add options like: + # - buildInputs + # - nativeBuildInputs + # - ... + dream2nix.modules.mkDerivation-interfaces + + # Generic interface for impure lang2nix tools (code generators) + # This provides options like `generateBin` (see below) + dream2nix.modules.integrations.impure + ]; + + options = { + modules = lib.mkOption { + description = "The path to the gomod2nix.toml"; + type = lib.types.str; + default = "${config.dream2nix.artifactsLocation}/gomod2nix.toml" ; + }; + + }; + + config = { + # Generated code will end up in: + # {repo}/dream2nix/artifacts/{engineName}/{package_identifier} + dream2nix.engineName = "gomod2nix"; + + # An executable that generates nix code for the given `src` + dream2nix.generateBin = dream2nix.utils.writePureShellScript "gomod2nix-generate.sh" + [ + # add gomod2nix tool to PATH + dream2nix.inputs.gomod2nix.packages.${system}.gomod2nix + ] + '' + targetDir=$1 + gomod2nix --dir "${config.src}" --outdir "$targetDir" + ''; + + # signal that all options should be passed to the final derivation function + argsForward = l.mapAttrs (_: _: true) options; + + # the final derivation is built by calling gomod2nix.buildGoApplication + config.final.derivation = + dream2nix.inputs.gomod2nix.lib.${system}.buildGoApplication + config.final.derivation-args; + }; +} +``` diff --git a/docs/src/v1-api/integrating/integrate-lang2nix-pure.md b/docs/src/v1-api/integrating/integrate-lang2nix-pure.md new file mode 100644 index 00000000..56e09e46 --- /dev/null +++ b/docs/src/v1-api/integrating/integrate-lang2nix-pure.md @@ -0,0 +1,46 @@ +# Integrate lang2nix tool (pure) +We use [crane](https://crane.dev) as an example here to demonstrate creating a dream2nix integration + +`dream2nix/modules/rust.crane-buildPackage.nix` +```nix +{config, lib, dream2nix, system, ...}: rec { + + imports = [ + # import generic mkDerivation interface, which will add options like: + # - buildInputs + # - nativeBuildInputs + # - ... + dream2nix.modules.mkDerivation-interfaces + ]; + + options = { + buildPhaseCargoCommand = lib.mkOption { + description = "A command to run during the derivation's build phase. Pre and post build hooks will automatically be run."; + type = lib.types.nullOr lib.types.str; + default = null; + }; + cargoArtifacts = lib.mkOption { + description = "A path (or derivation) which contains an existing cargo target directory, which will be reused at the start of the derivation. Useful for caching incremental cargo builds."; + type = lib.types.nullOr lib.types.str; + default = null; + }; + cargoBuildCommand = lib.mkOption { + description = "A cargo invocation to run during the derivation's build phase"; + type = lib.types.nullOr lib.types.str; + default = null; + } + + # ... more options of crane's buildPackage + }; + + config = { + # signal that all options should be passed to the final derivation function + argsForward = l.mapAttrs (_: _: true) options; + + # the final derivation is built by calling crane.buildPackage + config.final.derivation = + dream2nix.inputs.crane.lib.${system}.buildPackage + config.final.derivation-args; + }; +} +``` diff --git a/docs/src/v1-api/packaging/monorepo.md b/docs/src/v1-api/packaging/monorepo.md new file mode 100644 index 00000000..a761ead1 --- /dev/null +++ b/docs/src/v1-api/packaging/monorepo.md @@ -0,0 +1,94 @@ +# build packages in a monorepo +The example mono repo has 3 packages: `nodejs-app`, `python-tool`, `rust-tool`. + +The packages `python-tool` and `rust-tool` might or might not be built with dream2nix. + +The package `nodejs-app` is built with dream2nix and depends on `python-tool` and `rust-tool`. + +## Assuming this repo structure +``` +├── default.nix +├── overrides +│ ├── nodejs +│ ├── python +│ └── rust +├── nodejs-app +│ └── default.nix +├── python-tool +│ └── default.nix +└── rust-tool + └── default.nix +``` + +## Contents of `./nodejs-app/default.nix` +`./nodejs-app/default.nix` +```nix +{config, lib, dream2nix, ...}: { + + imports = [ + # default module to create a nodejs package + dream2nix.modules.nodejs.mkDerivation + # get package dependencies from package-lock + dream2nix.modules.nodejs.package-lock + ]; + + # Overrides allow to manipulate dependency builds + overrides.local.path = ../overrides/nodejs; + + src = ./.; + + # include dependencies from nixpkgs and the local monorepo + # see definition of `packageSets` in ../default.nix + deps = {nixpkgs, monorepo, ...} @ packageSets: { + inherit (nixpkgs) + hello + ; + inherit (monorepo) + python-tool + rust-tool + ; + }; + + nativeBuildInputs = [ + config.deps.hello + config.deps.python-tool + config.deps.rust-tool + ]; + + configurePhase = '' + hello --version + python-tool --version + rust-tool --version + ''; + + # add more mkDerivation attributes here to customize... +} +``` + +## Contents of `./default.nix` +`./default.nix` +```nix +{ + nixpkgs ? import {}, + dream2nix ? + import + (builtins.fetchTarball "https://dream2nix.dev/tarball/1.0") + {inherit nixpkgs;}, + +} @ inputs: let + + makePackage = modules: dream2ix.mkDerivation + # Package sets available to each package's `deps` function + {packageSets = {inherit monorepo nixpkgs;};} + modules; + + monorepo = { + nodejs-app = makePackage ./nodejs-app; + python-tool = makePackage ./python-tool; + rust-tool = makePackage ./rust-tool; + }; + +in { + packages = monorepo; +} +``` diff --git a/docs/src/v1-api/packaging/nodejs-init-project.md b/docs/src/v1-api/packaging/nodejs-init-project.md new file mode 100644 index 00000000..234f9a46 --- /dev/null +++ b/docs/src/v1-api/packaging/nodejs-init-project.md @@ -0,0 +1,123 @@ +# initialize nodejs project + dev shell + +## load shell with nodejs + npm +```console tesh-session="next-app" tesh-setup="setup.sh" +$ nix-shell -p https://dream2nix.dev -A devShells.nodejs +``` + +## create my-app +```console tesh-session="next-app" +npx create-next-app my-app +``` +This creates `./my-app/package.json` and more, using `create-next-app` as a helper. + +## create `my-app.nix` +`my-app.nix` +```nix +{config, lib, dream2nix, ...}: { + + imports = [ + # default module to create a nodejs package + dream2nix.modules.nodejs.mkDerivation + # get package dependencies from package-lock + dream2nix.modules.nodejs.package-lock + ]; + + # Allows to manipulate dependency builds + overrides.local.path = ./overrides; + + src = ./my-app; + + # add more mkDerivation attributes here to customize... +} +``` + +## create `my-app-shell.nix` for your dev shell +`my-app-shell.nix` +```nix +{config, lib, dream2nix, ...}: { + + imports = [ + # the default dev shell for nodejs + dream2nix.modules.nodejs.mkShell + # adds dependencies of my-app to the dev shell + dream2nix.modules.nodejs.package-lock + ]; + + src = ./my-app; + + # include hello from nixpkgs. + # `deps` is the single source of truth for inputs from the `outside world`. + # `deps` will later allow us to safely override any dependency. + deps = {nixpkgs, ...}: { + inherit (nixpkgs) hello; + }; + + # add hello from nixpkgs to the dev shell + buildInputs = [ + config.deps.hello + ] +} +``` + +## create `default.nix` entry point +`default.nix` +```nix +{ + nixpkgs ? import {}, + dream2nix ? + import + (builtins.fetchTarball "https://dream2nix.dev/tarball/1.0") + {inherit nixpkgs;}, +}: { + packages.my-app = dream2nix.eval ./my-app.nix; + devShells.my-app = dream2nix.eval ./my-app-shell.nix; +} +``` + +## build my-app +```command +nix-build -f ./default.nix -A packages.my-app +``` +## create `shell.nix` (used by `nix-shell` command) +`shell.nix` +```nix +(import ./default.nix {}).devShells.my-app +``` +Enter the dev shell: +```command +nix-shell +``` +all dependencies of my-app are available + +## fix build of dependencies via `./overrides/` +Files in `./overrides/` must always be named like the package they apply to. + +Example: `./overrides/keytar.nix` +## +```nix +{config, ...}: { + + # include dependencies from nixpkgs. + deps = {nixpkgs, ...}: { + inherit (nixpkgs) + libsecret + pkg-config + ; + }; + + # add build time dependencies + nativeBuildInputs = [ + config.deps.libsecret + config.deps.pkg-config + ]; +} +``` + +Scoped package example: `./overrides/@babel/core.nix` +## +```nix +{config, ...}: { + # ... +} +``` diff --git a/docs/src/v1-api/packaging/nodejs-multiple-repos.md b/docs/src/v1-api/packaging/nodejs-multiple-repos.md new file mode 100644 index 00000000..be17bc2b --- /dev/null +++ b/docs/src/v1-api/packaging/nodejs-multiple-repos.md @@ -0,0 +1,67 @@ +# handle multiple repos +Assuming that `./repo1` and `./repo2` are separate git repositories. + +Both repos have a single package `repo1/my-app` and `repo2/my-tool`. + +In order to build `repo1/my-app` we need `repo2/my-tool` as a build time dependency. + +The following structure is assumed: +``` +├── repo1 +│ ├── default.nix +│ └── my-app.nix +└── repo2 + ├── default.nix + └── my-tool.nix +``` + +## contents of `repo1/my-app.nix` +`repo1/my-app.nix` +```nix +{config, lib, dream2nix, ...}: { + + imports = [ + dream2nix.modules.nodejs.mkDerivation + dream2nix.modules.nodejs.package-lock + ]; + + src = ./.; + + # include my-tool from repo2 + deps = {repo2, ...}: { + inherit (repo2) my-tool; + }; + + # add my-tool as build time dependency + nativeBuildInputs = [ + config.deps.my-tool + ]; + + # use my-tool to build my-app + buildPhase = '' + my-tool build + echo "done building" + ''; +} +``` + +## contents of `repo1/default.nix` +`repo1/default.nix` +```nix +{ + pkgs ? import {}, + dream2nix ? + import + (builtins.fetchTarball "https://dream2nix.dev/tarball/1.0") + {inherit pkgs;}, +}: { + packages.my-app = dream2nix.eval + { + packageSets.nixpkgs = pkgs; + + # fetchGit could be used here alternatively + packageSets.repo2 = import ../repo2/default.nix {}; + } + ./my-app.nix; +} +``` diff --git a/docs/src/v1-api/packaging/nodejs-workspaces.md b/docs/src/v1-api/packaging/nodejs-workspaces.md new file mode 100644 index 00000000..ab573b0f --- /dev/null +++ b/docs/src/v1-api/packaging/nodejs-workspaces.md @@ -0,0 +1,82 @@ +# build + develop on nodejs workspaces +## assuming a `package.json` with workspaces +`package.json` +``` +{ + "name": "my-workspaces", + "workspaces": [ + "my-tool" + "my-first-app" + "my-second-app" + ] +} +``` + +## define package set via `workspaces.nix` +```nix +{config, lib, dream2nix, ...}: { + + imports = [ + dream2nix.modules.nodejs.workspaces + dream2nix.modules.nodejs.package-lock + ]; + + src = ./.; + + # Allows to manipulate builds of workspace members and their dependencies + overrides.local.path = ./overrides; +} +``` + +## create `default.nix` entry point +`default.nix` +```nix +{ + nixpkgs ? import {}, + dream2nix ? + import + (builtins.fetchTarball "https://dream2nix.dev/tarball/1.0") + {inherit nixpkgs;}, +}: { + packages = { + inherit (dream2nix.lib.mkPackageSet ./workspaces.nix) + my-tool + my-first-app + my-second-app + ; + }; +} +``` + +## configure package builds via `./overrides/` +Files in `./overrides/` must always be named like the the package they apply to. + +Manipulate my-tool via `./overrides/my-tool.nix` +```nix +{config, ...}: { + + # include python from nixpkgs + deps = {nixpkgs, ...}: { + inherit (nixpkgs) python; + }; + + buildInputs = [ + config.deps.python + ]; +} +``` + +Manipulate my-first-app via `./overrides/my-first-app.nix` +```nix +{config, ...}: { + + # include my-tool from the local workspace + deps = {workspace, ...}: { + inherit (workspace) my-tool; + }; + + buildInputs = [ + config.deps.my-tool + ]; +} +``` diff --git a/docs/src/v1-api/problems.md b/docs/src/v1-api/problems.md new file mode 100644 index 00000000..e8ee0c9f --- /dev/null +++ b/docs/src/v1-api/problems.md @@ -0,0 +1,22 @@ +# Problems of the current dream2nix + +## Integration of existing lang2nix tools +Until now, integrating existing 2nix solutions into dream2nix was hard because dream2nix imposed standards which are not met by most existing tools. +With v1 we want to lift most of these restrictions to make integration a no-brainer + +(see sections [integrate lang2nix tool (pure)](../v1-api/integrate-lang2nix-pure.md) and [integrate lang2ix tool (impure)](../v1-api/integrate-lang2nix-impure.md). + +## Tied to flakes +The current api is tied to flakes. The v1 API should not depend on flakes anymore. + +## Composability +Composability with the current `makeFlakeOutputs` is bad. Flakes itself aren't nicely composable. Filtering and merging of nested attrsets isn't user friendly. + +The v1 api will focus on delivering individual derivations, not flakes. +While we might provide templates, recommendations, and tools for composition, we should not enforce a specific solution onto the user. + +## Overridability +The experience of overriding package- and dependency builds was a bit bumpy so far, as the overriding mechanism was built ontop of override functions provided by nixpkgs' `mkDerivation`. The v1 API will make use of the nixos module system instead to handle derivation attributes. + +## Discoverability of package options +We want users to be able to inspect the API of an individual package. This will also be made possible by the nixos module system. diff --git a/docs/src/v1-api/summary.md b/docs/src/v1-api/summary.md new file mode 100644 index 00000000..9bfaaabe --- /dev/null +++ b/docs/src/v1-api/summary.md @@ -0,0 +1,16 @@ +# dream2nix v1 API + +- [problems of the current dream2nix](../v1-api/problems.md) +- [users of dream2nix](../v1-api/users.md) +- v1 API examples: + - package maintainers: + - [project initialization](../v1-api/packaging/nodejs-init-project.md) + - [workspaces](../v1-api/packaging/nodejs-workspaces.md) + - [multiple repos](../v1-api/packaging/nodejs-multiple-repos.md) + - [monorepo](../v1-api/packaging/monorepo.md) + - consumers: + - [inspect package options](../v1-api/consuming/inspect-options.md) + - [override packages](../v1-api/consuming/override.md) + - integration maintainers: + - [integrate lang2nix tool (pure)](../v1-api/integrating/integrate-lang2nix-pure.md) + - [integrate lang2nix tool (code-gen/impure)](../v1-api/integrating/integrate-lang2nix-impure.md) diff --git a/docs/src/v1-api/users.md b/docs/src/v1-api/users.md new file mode 100644 index 00000000..f0275679 --- /dev/null +++ b/docs/src/v1-api/users.md @@ -0,0 +1,11 @@ +# The users of dream2nix +The following groups of users are relevant regarding the v1 API design. + +## Integration Maintainers (Level1) +People who use dream2nix to maintain language2nix integrations + +## Package Maintainers (Level2) +People who use dream2nix to maintain nix derivations for packages + +## Consumers (Level3) +People who use and customize packages created via dream2nix