docs: setup mdbook book for documentation, add CLI app for docs (#173)

* docs: setup mdbook book for documentation

* ci: add github action for docs

* docs: add a cli command for showing docs in terminal

* Update docs/src/subsystems/rust.md

Co-authored-by: Ilan Joselevich <personal@ilanjoselevich.com>

* docs(rust): add example for crane overriding

* docs: add a further reading section to intro

* docs: fix a few links

* docs(readme): move documentation section up and fit some sections into 80 columns

Co-authored-by: Ilan Joselevich <personal@ilanjoselevich.com>

.github/workflows/pages.yml

@@ -0,0 +1,30 @@
+name: github pages
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Install nix
+        uses: cachix/install-nix-action@v17
+        with:
+          extra_nix_config: |
+            experimental-features = nix-command flakes
+          nix_path: nixpkgs=channel:nixos-unstable
+
+      - name: Build documentation book
+        run: nix build -L --show-trace .#docs
+
+      - name: Deploy
+        uses: peaceiris/actions-gh-pages@v3
+        if: ${{ github.ref == 'refs/heads/main' }}
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ./result

README.md

@@ -4,7 +4,9 @@
 
 ## [WIP] dream2nix - A framework for automated nix packaging
 
-dream2nix is a framework for automatically converting packages from other build systems to nix.
+dream2nix is a framework for automatically converting packages from other build
+systems to nix.
+
 It focuses on the following aspects:
 
 - Modularity
@@ -19,11 +21,16 @@ It focuses on the following aspects:
 - Exploration and adoption of new nix features
 - Simplified updating of packages
 
-The goal of this project is to create a standardized, generic, modular framework for automated packaging solutions, aiming for better flexibility, maintainability and usability.
+The goal of this project is to create a standardized, generic, modular framework
+for automated packaging solutions, aiming for better flexibility, maintainability
+and usability.
 
-The intention is to integrate many existing 2nix converters into this framework, thereby improving many of the previously named aspects and providing a unified UX for all 2nix solutions.
+The intention is to integrate many existing 2nix converters into this framework,
+thereby improving many of the previously named aspects and providing a unified
+UX for all 2nix solutions.
 
 ### Test the experimental version of dream2nix
+
 (Currently only nodejs and rust packaging is supported)
 
 1. Make sure you use a nix version >= 2.4 and have `experimental-features = "nix-command flakes"` set.
@@ -121,23 +128,34 @@ Extensive Example `flake.nix`:
 An example for instancing dream2nix per pkgs and using it to create outputs
 can be found at [`examples/d2n-init-pkgs`](./examples/d2n-init-pkgs/flake.nix).
 
+### Documentation
+
+Documentation for `main` branch is deployed to https://nix-community.github.io/dream2nix.
+
+You can also build documentation by running `nix build .#docs`, or by entering
+the development shell (`nix develop`) and running `mdbook build docs`.
+
+A CLI app is available if you want to read documentation in your terminal.
+`nix run .#docs` can be used to list available documentation, and you can access
+any of those by running `nix run .#docs -- name`. Subsystem documentation can be
+accessed with just `nix run .#docs -- subsystem`.
+
 ### Watch the presentation
+
 (The code examples of the presentation are outdated)
 [![dream2nix - A generic framework for 2nix tools](https://gist.githubusercontent.com/DavHau/755fed3774e89c0b9b8953a0a25309fa/raw/3c8b2c56f5fca3bf5c343ffc179136eef39d4d6a/dream2nix-youtube-talk.png)](https://www.youtube.com/watch?v=jqCfHMvCsfQ)
 
-### Further Reading
-
-- [Summary of the core concepts and benefits](/docs/concepts-and-benefits.md)
-- [How would this improve the packaging situation in nixpkgs](/docs/nixpkgs-improvements.md)
-- [Override System](/docs/override-system.md)
-- [Contributors Guide](/docs/contributors-guide.md)
-- [Extending dream2nix](/docs/extending-dream2nix.md)
-
 ### Funding
-This project receives financial support by [NLNet](https://nlnet.nl/) as part of the [NGI Assure Programme](https://nlnet.nl/assure/) funded by the European Commission.
 
-If your organization wants to support the project with extra funding in order to add support for more languages or new featuress, please contact one of the maintainers.
+This project receives financial support by [NLNet](https://nlnet.nl/) as part
+of the [NGI Assure Programme](https://nlnet.nl/assure/) funded by the European
+Commission.
+
+If your organization wants to support the project with extra funding in order
+to add support for more languages or new featuress, please contact one of the
+maintainers.
 
 ### Community
+
 matrix: https://matrix.to/#/#dream2nix:nixos.org
 

docs/.gitignore

@@ -0,0 +1 @@
+book

docs/book.toml

@@ -0,0 +1,6 @@
+[book]
+authors = ["Yusuf Bera Ertan"]
+language = "en"
+multilingual = false
+src = "src"
+title = "dream2nix documentation"

docs/src/SUMMARY.md

@@ -0,0 +1,11 @@
+# Summary
+
+- [Introduction](./intro.md)
+  - [Concepts and benefits](./intro/concepts-and-benefits.md)
+  - [Nixpkgs improvements](./intro/nixpkgs-improvements.md)
+  - [Override system](./intro/override-system.md)
+  - [Translators](./intro/translators.md)
+- [Subsystems](./subsystems.md)
+  - [Rust](./subsystems/rust.md)
+- [Extending dream2nix](./extending-dream2nix.md)
+- [Contributing](./contributing.md)

docs/src/extending-dream2nix.md

@@ -23,7 +23,7 @@ this file should look like this:
 }
 ```
 
-See the [`d2n-extended-new-subsystem` example](../examples/d2n-extended-new-subsystem) for a basic example.
+See the [`d2n-extended-new-subsystem` example](https://github.com/nix-community/dream2nix/tree/main/examples/d2n-extended-new-subsystem) for a basic example.
 
 ## Declare `extra`s as an attribute set
 
@@ -50,7 +50,7 @@ note: you can't declare modules using functions here. This is
 because the `config` may need to be serialized to JSON and passed
 around in impure situations (mostly impure translators).
 
-See the [`d2n-extended` example](../examples/d2n-extended/flake.nix) for a basic example.
+See the [`d2n-extended` example](https://github.com/nix-community/dream2nix/tree/main/examples/d2n-extended/flake.nix) for a basic example.
 
 ## Compose multiple different `extra`s
 

docs/src/intro.md

@@ -0,0 +1,37 @@
+# dream2nix
+
+dream2nix is a framework for automatically converting packages from other
+build systems to nix.
+
+It focuses on the following aspects:
+
+- Modularity
+- Customizability
+- Maintainability
+- Nixpkgs Compatibility, by not enforcing [IFD (import from derivation)][glossary]
+- Code de-duplication across 2nix converters
+- Code de-duplication in nixpkgs
+- Risk-free opt-in aggregated fetching (larger [FODs][glossary], less checksums)
+- Common UI across 2nix converters
+- Reduce effort to develop new 2nix solutions
+- Exploration and adoption of new nix features
+- Simplified updating of packages
+
+The goal of this project is to create a standardized, generic, modular
+framework for automated packaging solutions, aiming for better flexibility,
+maintainability and usability.
+
+The intention is to integrate many existing 2nix converters into this framework,
+thereby improving many of the previously named aspects and providing a unified
+UX for all 2nix solutions.
+
+### Further reading
+
+- [Summary of the core concepts and benefits](./intro/concepts-and-benefits.md)
+- [How would this improve the packaging situation in nixpkgs](./intro/nixpkgs-improvements.md)
+- [Override System](./intro/override-system.md)
+- [Contributors Guide](./contributing.md)
+- [Extending dream2nix](./extending-dream2nix.md)
+- [Subsystems](./subsystems.md)
+
+[glossary]: https://nixos.wiki/wiki/Glossary "glossary"

docs/src/intro/concepts-and-benefits.md

@@ -86,7 +86,7 @@ Translator:
     - `pure-nix`: translate input by using the nix language only
     - `IFD/recursive`: translate using a nix build
     - `external`: translate using an external tool which resolves against an online package index
-  - for more information about translators and how nixpkgs compatibility is guaranteed, check [./translators.md](/docs/translators.md)
+  - for more information about translators and how nixpkgs compatibility is guaranteed, check [Translators](./translators.md)
 
 Generic Lock (standardized format):
   - Produced by `Translator`. Contains URLs + hashes for sources and metadata relevant for building.

docs/src/intro/translators.md

@@ -1,5 +1,8 @@
 # Translators
-This document classifies different methods for translating requirement/lock files to the generic lock format and describes how the resulting packages can be integrated into nix builds inside and outside of nixpkgs.
+
+This document classifies different methods for translating requirement/lock
+files to the generic lock format and describes how the resulting packages
+can be integrated into nix builds inside and outside of nixpkgs.
 
 
 ## pure-nix (preferred)

docs/src/subsystems.md

@@ -0,0 +1,10 @@
+# Subsystems
+
+This section documents subsystem specific behaviour and interfaces.
+
+## Available subsystems
+
+- [Rust (`rust`)](./subsystems/rust.md)
+- Nodejs (`nodejs`)
+- Python (`python`)
+- Go (`go`)

docs/src/subsystems/rust.md

@@ -0,0 +1,45 @@
+# Rust subsystem
+
+This section documents the Rust subsystem.
+
+## Translators
+
+### cargo-lock (pure)
+
+Translates a `Cargo.lock` file to a dream2nix lockfile.
+
+### cargo-toml (impure)
+
+Translates a `Cargo.toml` file to a dream2nix lockfile by generating a
+`Cargo.lock` file first and then using `cargo-lock` translator on the
+generated lockfile.
+
+## Builders
+
+### build-rust-package (pure)
+
+Builds a package using `buildRustPackage` from `nixpkgs`.
+
+### crane (ifd)
+
+Builds a package using [`crane`](https://github.com/ipetkov/crane).
+
+#### Override gotchas
+
+This builder builds two separate derivations, one for your crate's dependencies
+and another for your crate. This means that if you want to override stuff for
+the dependencies, you need to use the `<crate-name>-deps` key for your override
+where `<crate-name>` is the name of the crate you are building.
+
+```nix
+{
+  # ...
+  packageOverrides = {
+    # this will apply to your crate
+    crate.my-overrides = { /* ... */ };
+    # this will apply to your crate's dependencies
+    crate-deps.my-overrides = { /* ... */ };
+  };
+  # ...
+}
+```

flake.nix

@@ -164,20 +164,21 @@
           };
         }
     );
-  in {
+
     # System independent dream2nix api.
     # Similar to drem2nixFor but will require 'system(s)' or 'pkgs' as an argument.
     # Produces flake-like output schema.
-    lib =
+    d2n-lib =
       (import ./src/lib.nix {
         inherit externalPaths externalSources overridesDirs lib;
         nixpkgsSrc = "${nixpkgs}";
       })
       # system specific dream2nix library
       // (forAllSystems (system: pkgs: dream2nixFor."${system}"));
-
-    # with project discovery enabled
-    lib2 = self.lib;
+  in {
+    lib = d2n-lib;
+    # kept for compat
+    lib2 = d2n-lib;
 
     # all apps including cli, install, etc.
     apps = forAllSystems (
@@ -235,6 +236,50 @@
               export PATH="${alejandra.defaultPackage.${system}}/bin"
               ${pkgs.treefmt}/bin/treefmt --clear-cache "$@"
             '');
+
+          docs.type = "app";
+          docs.program =
+            l.toString
+            (pkgs.writeScript "docs" ''
+              # convert to lowercase, all of our doc paths are in lowercase
+              __docToShow="''${1,,}"
+              # remove .md suffix if it exists, we add this later ourselves
+              # doing this we can allow both "path" and "path.md"
+              _docToShow="''${__docToShow%".md"}"
+              # replace dots with slashes so "subsystems.name" can be used to access doc
+              docToShow="''${_docToShow//./\/}"
+              docs="${self}/docs/src"
+
+              function showDoc {
+                ${pkgs.glow}/bin/glow -lp "$docs/''${1}''${docToShow}.md"
+              }
+              function docExists {
+                test -f "$docs/''${1}''${docToShow}.md"
+              }
+
+              # if no doc to show was passed then list available docs
+              if [[ "$docToShow" == "" ]]; then
+                echo "available documentation:''\n"
+                cd $docs
+                ${pkgs.ripgrep}/bin/rg --files --sort=path
+              # first we check for the doc in subsystems
+              elif $(docExists "subsystems/"); then
+                showDoc "subsystems/"
+              # then we check in intro
+              elif $(docExists "intro/"); then
+                showDoc "intro/"
+              # then in the root documentation directory
+              # this also allows for any arbitrary doc path to be accessed
+              elif $(docExists ""); then
+                showDoc ""
+              else
+                echo "no documentation for '$docToShow'"
+                echo "suggestions:''\n"
+                cd $docs
+                ${pkgs.ripgrep}/bin/rg --files-with-matches \
+                  --sort=path "$docToShow"
+              fi
+            '');
         }
     );
 
@@ -258,6 +303,7 @@
         commands =
           [
             {package = pkgs.nix;}
+            {package = pkgs.mdbook;}
             {
               package = pkgs.treefmt;
               category = "formatting";
@@ -316,6 +362,17 @@
       };
     });
 
+    packages = forAllSystems (system: pkgs: {
+      docs =
+        pkgs.runCommand
+        "dream2nix-docs"
+        {nativeBuildInputs = [pkgs.mdbook];}
+        ''
+          cp -r ${./docs}/* .
+          mdbook build -d $out
+        '';
+    });
+
     templates = {
       simple = {
         description = "Simple dream2nix flake";