python improvements, docs and guide.

- add warning to readme and docs stating that dream2nix is unstable software
- refactor structure of docs: remove title .md pages, instead add titles via (# title) to SUMMARY
- change order of doc topics. move `guides` and `subsystems` to the front
- add docs for python subsystem
- add getting started guide for python
- extend ./examples/python to make use of translator options
- add new function to dlib `systemsFromFile` that allows to load the `supportedSystems` from a file. This alows changing the supported systems without touching the flake.
- move pip translator args to extra file `args.nix` to allow embedding in docs
- fix pip translator
- add a .gitignore to the basic flake template
- use new systemsFromFile function in basic flake template

README.md

@@ -2,7 +2,9 @@
 <img width="400" src="https://gist.githubusercontent.com/DavHau/755fed3774e89c0b9b8953a0a25309fa/raw/e2a12a60ae49aa5eb11b42775abdd1652dbe63c0/dream2nix-01.png">
 </p>
 
-## [WIP] dream2nix - A framework for automated nix packaging
+## dream2nix - A framework for automated nix packaging
+
+!!! Warning: dream2nix is unstable software. While simple UX is one of our main focus points, the APIs  are still under development. Do expect changes that will break your setup.
 
 dream2nix is a framework for automatically converting packages from other build systems to nix.
 

docs/src/SUMMARY.md

@@ -1,14 +1,23 @@
-# Summary
 
 - [Introduction](./intro.md)
-  - [Architectural Considerations](./intro/architectural-considerations.md)
-  - [Nixpkgs improvements](./intro/nixpkgs-improvements.md)
-  - [Override system](./intro/override-system.md)
-  - [Translators](./intro/translators.md)
-  - [Indexers](./intro/indexers.md)
-- [Subsystems](./subsystems.md)
-  - [Rust](./subsystems/rust.md)
+
+# Guides
+- [Python: getting started in 10 minutes](./guides/getting-started-python.md)
+# Subsystems
+- [Rust](./subsystems/rust.md)
+- [Python](./subsystems/python.md)
+
+# Concept
+- [Architectural Considerations](./intro/architectural-considerations.md)
+- [Nixpkgs improvements](./intro/nixpkgs-improvements.md)
+- [Override system](./intro/override-system.md)
+- [Translators](./intro/translators.md)
+- [Indexers](./intro/indexers.md)
+
+
+# Contributing
 - [Extending dream2nix](./extending-dream2nix.md)
 - [Contributing](./contributing.md)
-- [Development Roundups](./development-roundups.md)
-  - [April - June 2022](./development-roundups/2022-april-june.md)
+
+# Development Roundups
+- [April - June 2022](./development-roundups/2022-april-june.md)

docs/src/development-roundups.md

@@ -1,3 +0,0 @@
-# Regular development roundups for dream2nix
-
-- [2022 April - June Development Roundup](./development-roundups/2022-april-june.md)

docs/src/guides/getting-started-python.md

@@ -0,0 +1,112 @@
+# Build your python project with nix in 10 minutes
+
+{{#include ../warning.md}}
+
+This guide walks you through the process of setting up nix for your project using dream2nix. This will allow your project's build and dev-environment to be reproduced by machines of other developers or CI systems with high accuracy.
+ 
+## Install nix
+If you don't have nix already, check out [nixos.org/download.html](https://nixos.org/download.html) on how to install it.
+
+## Enable the nix flakes feature
+For internal dependency management dream2nix requires the experimental nix feature `flakes` being enabled.
+```
+export NIX_CONFIG="extras-experimental-features = flakes nix-command"
+```
+
+If you find yourself using dream2nix regularly, you can permanently save these settings by adding the following line to your `/etc/nix/nix.conf`:
+```
+experimental-features = flakes nix-command
+```
+
+## Navigate to your python project
+In this example I will clone the python project [`httpie`](https://github.com/httpie/httpie) to `/tmp/my_project` as an example.
+```command
+> git clone https://github.com/httpie/httpie /tmp/my_project
+> cd ./my_project
+```
+
+## Initialize the dream2nix flake.nix
+```command
+> nix flake init -t github:nix-community/dream2nix#simple
+wrote: /tmp/my_project/flake.nix
+```
+Great, this created a new file `flake.nix` which is like a recipe that tells nix how to build our python project or how to assemble a development environment for it.  
+By modifying this file, we can tweak settings and change the way our package gets built by nix. But for now we just go with the defaults.
+
+## Define the target platform
+Before we can start, we need to tell dream2nix which platform we want to build software for. dream2nix will read these platforms from the file `./nix_systems`. To get started we will jsut add our current platform to it with the following command.
+```command
+> nix eval --impure --raw --expr 'builtins.currentSystem' > ./nix_systems
+> git add ./nix_systems
+```
+Don't forget to add the file `./nix_systems` to git, otherwise it will be ignored.  
+If you want to support more platforms later, just add more lines to that file.
+
+## List the available packages
+Let's get an overview of what the `flake.nix` allows us to do with our project.
+```command
+> nix flake show
+warning: Git tree '/tmp/my_project' is dirty
+warning: creating lock file '/tmp/my_project/flake.lock'
+warning: Git tree '/tmp/my_project' is dirty
+git+file:///tmp/my_project
+├───packages
+│   └───x86_64-linux
+│       ├───main: package 'main'
+│       └───resolveImpure: package 'resolve'
+└───projectsJson: unknown
+```
+
+What we can observe here:
+1. ```warning: Git tree '/tmp/my_project' is dirty```  
+Nix warns us that the current git repo has uncommited changes. Thats fine, because we like to experiment for now. This warning will go away as soon as we commit our changes.
+1. `warning: creating lock file '/tmp/my_project/flake.lock'`  
+Our flake.nix imported external libraries. The versions of these libraries have now been locked inside a new file `flake.lock`. We should later commit this file to the repo, in order to allow others to reproduce our build exactly.
+1.
+    ```
+      git+file:///tmp/my_project
+      ├───packages
+      │   └───x86_64-linux
+      │       ├───main: package 'main'
+      │       └───resolveImpure: package 'resolve'
+      └───projectsJson: unknown
+    ```
+    Similar like a .json file defines a structure of data, our flake.nix defines a structure of `nix attributes` which are things that we can build or run with nix.  
+    We can see that it contains packages for my current platform `x86_64-linux`.
+    
+    The packages which we can see here is my python package and a package called`resolveImpure`, which is a special package provided by dream2nix which we will learn more about later.
+
+## Build the Project
+Let's try building our project.
+If you get an error about `unresolved impurities`, see [Resolve Impurities](#resolve-impurities)
+```
+nix build .#default
+```
+Congratulations, your build artifacts will now be accessible via the `./result` directory. If your project contains executables, you can run these via `./result/bin/executable-name`.
+If you want to develop on your python project, see [Create a development shell](#create-a-development-shell)
+
+## Create a development shell
+Nix can provide you with a development shell containing all your project's dependencies.
+First, ensure that your project [is resolved](#resolve-impurities), then execute the following command.
+```command
+nix develop -c $SHELL
+```
+The `-c $SHELL` part is only necessary if you use a different shell than bash and would like to bring that shell with you into the dev environment.
+
+## Resolve Impurities
+If you try to build, you might run into the following error.
+```
+> nix build .#default
+error: The python package main contains unresolved impurities.
+       Resolve by running the .resolve attribute of this derivation
+       or by resolving all impure projects by running the `resolveImpure` package
+```
+Oops. It seems like our project does not contain enough information for dream2nix to construct a reproducible build. But this is not a problem as we can fix this by using the `resolveImpure` package that dream2nix provides.
+```command
+nix run .#resolveImpure
+...
+adding file to git: dream2nix-packages/main/dream-lock.json
+```
+Fine, that created a new file `dream-lock.json` which is a lock file specifically for our python project. If we later add any dependencies, we will have to re-run `resolveImpure` to update this lock file.
+
+Now everything should be ready to [Build the Project](#build-the-project)

docs/src/intro.md

@@ -1,8 +1,10 @@
 # dream2nix
 
+{{#include ./warning.md}}
+
 dream2nix is a framework for automatically converting packages from other build systems to nix.
 
-It focuses on the following aspects:
+dream2nix focuses on the following aspects:
 
 - Modularity
 - Customizability

docs/src/subsystems.md

@@ -1,11 +0,0 @@
-# Subsystems
-
-This section documents subsystem specific behaviour and interfaces.
-
-## Available subsystems
-
-- [Rust (`rust`)](./subsystems/rust.md)
-- Nodejs (`nodejs`)
-- Python (`python`)
-- Go (`go`)
-- Haskell (`haskell`)

docs/src/subsystems/python.md

@@ -0,0 +1,28 @@
+# Python subsystem
+
+This section documents the Python subsystem.
+
+## Examples
+flake.nix
+```nix
+{{#include ../../../examples/python/flake.nix}}
+```
+## Translators
+
+### pip (impure)
+
+Can translate all pip compatible python projects, including projects managed with poetry or other tool-chains which integrate via `pyproject.toml`.
+
+This translator simply executes `pip download` on the given source and observes which sources are downloaded by pip.
+The downside of this approach is, that this translator cannot be used with a granular builder. It does not understand the exact relation between the dependencies, and therefore it only allows to build all dependencies in one large derivation.
+
+#### **pip** optional translator arguments
+```
+{{#include ../../../src/subsystems/python/translators/pip/args.nix}}
+```
+
+## Builders
+
+### simple-builder (pure) (default)
+
+Builds a package including all its dependencies in a single derivation.

docs/src/warning.md

@@ -0,0 +1 @@
+!!! Warning: dream2nix is unstable software. While simple UX is one of our main focus points, the APIs  are still under development. Do expect changes that will break your setup.

examples/python/flake.nix

@@ -14,6 +14,14 @@
       systems = ["x86_64-linux"];
       config.projectRoot = ./.;
       source = src;
+      settings = [
+        {
+          # optionally define python version
+          subsystemInfo.pythonAttr = "python38";
+          # # optionally define extra setup requirements;
+          subsystemInfo.extraSetupDeps = ["cython > 0.29"];
+        }
+      ];
     })
     // {
       checks.x86_64-linux.aiohttp = self.packages.x86_64-linux.main;

flake.nix

@@ -339,8 +339,7 @@
         "dream2nix-docs"
         {nativeBuildInputs = [pkgs.mdbook];}
         ''
-          cp -r ${./docs}/* .
-          mdbook build -d $out
+          mdbook build -d $out ${./.}/docs
         '';
     });
 

src/lib.nix

@@ -177,8 +177,9 @@
       flakifiedOutputsList;
   in
     flakeOutputs;
-in {
+in rec {
   inherit init makeFlakeOutputs makeFlakeOutputsForIndexes;
+  inherit (dlib) systemsFromFile;
   dlib = import ./lib {
     inherit lib;
     config = (import ./utils/config.nix).loadConfig {};

src/lib/default.nix

@@ -30,6 +30,7 @@
       sanitizePath
       sanitizeRelativePath
       subsystems
+      systemsFromFile
       traceJ
       modules
       warnIfIfd
@@ -295,6 +296,17 @@
     then "/${sanitizedRelPath}"
     else sanitizedRelPath;
 
+  systemsFromFile = file:
+    if ! l.pathExists file
+    then
+      throw ''
+        The system for your flake.nix is not initialized yet.
+        Please execute the following command to initialize it:
+
+        nix eval --impure --raw --expr 'builtins.currentSystem' > ./${l.baseNameOf file} && git add ./${l.baseNameOf file}
+      ''
+    else l.filter (l: l != "") (l.splitString "\n" (l.readFile file));
+
   traceJ = toTrace: eval: l.trace (l.toJSON toTrace) eval;
 
   ifdWarnMsg = module: ''

src/subsystems/python/translators/pip/args.nix

@@ -0,0 +1,24 @@
+{
+  pythonAttr = {
+    description = "python version to translate for";
+    default = "python3";
+    examples = [
+      "python27"
+      "python39"
+      "python310"
+    ];
+    type = "argument";
+  };
+
+  extraSetupDeps = {
+    description = ''
+      a list of extra setup reqirements to install before executing 'pip download'
+    '';
+    default = [];
+    examples = [
+      "cython"
+      "numpy"
+    ];
+    type = "argument";
+  };
+}

src/subsystems/python/translators/pip/default.nix

@@ -70,6 +70,7 @@ in {
       $python -m pip download \
         --dest $tmp \
         --progress-bar off \
+        -r __extra_setup_reqs.txt \
         -r __setup_reqs.txt
 
       # download files according to requirements
@@ -91,32 +92,5 @@ in {
     '';
 
   # define special args and provide defaults
-  extraArgs = {
-    # the python attribute
-    pythonAttr = {
-      default = "python3";
-      description = "python version to translate for";
-      examples = [
-        "python27"
-        "python39"
-        "python310"
-      ];
-      type = "argument";
-    };
-
-    extraSetupDeps = {
-      default = [];
-      description = "a list of extra requirements to add";
-      examples = [
-        "cython"
-        "numpy"
-      ];
-      type = "argument";
-    };
-
-    application = {
-      description = "build application instead of package";
-      type = "flag";
-    };
-  };
+  extraArgs = import ./args.nix;
 }

templates/simple/.gitignore

@@ -0,0 +1 @@
+result

templates/simple/flake.nix

@@ -3,7 +3,7 @@
   outputs = inp:
     inp.dream2nix.lib.makeFlakeOutputs {
       # modify according to your supported systems
-      systems = ["x86_64-linux"];
+      systems = inp.dream2nix.lib.systemsFromFile ./nix_systems;
       config.projectRoot = ./.;
       source = ./.;
     };