nix-community/dream2nix
1
1
Fork 0
mirror of https://github.com/nix-community/dream2nix.git synced 2024-12-22 22:11:50 +03:00
Code Issues Projects Releases Wiki Activity

improve readme, add docs/translators.md

Browse Source
This commit is contained in:
DavHau 2021-09-06 16:57:34 +02:00
parent 25c940d79b
commit fe2fdd25ae
3 changed files with 137 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

101
README.md
View File

@ -4,20 +4,21 @@ It focuses on the following aspects:
- Modularity
- Customizability
- Maintainability
- Compatibility (not enforcing features like IFD)
- Code de-duplication across 2nix tools
- Standardised UI across different solutions
- Reduced effort to develop new 2nix solutions
- Common UI across 2nix tools
- Reduce effort to develop new 2nix solutions
### Motivation
2nix tools, or in other words, tools converting instructions of other build systems to nix build instructions, are an important part of the nix/nixos ecosystem. These tools make packaging workflows easier and often allow to manage complexity that would be hard/impossible to manage without the 2nix tools.
Yet the current landscape of 2nix tools has certain weaknesses. Existing 2nix tools are very monolithic. Authors of 2nix tools are often motivated by some specific use case they try to solve and therefore the individual approaches are strongly biased. All existing tools have quite different user interfaces, use different strategies of parsing, resolving, fetching, building. As a user of these tools it often feels like there is some part of the tool that suits the needs well, but at the same time it has undesirable hard coded behaviour. Often one would like to use some aspect of one tool, combined with some aspect of another tool. One tool, for example, might do a good job in reading a specific lock file format, but doesn't come with good customizability for building. Another tool might come with a good customization interface, but lacks the capabilities of parsing the lock file format provided by upstream. Some tools are restricted to use IFD or FOD, while others enforce code generation.
Yet the current landscape of 2nix tools has certain weaknesses. Existing 2nix tools are very monolithic. Authors of these tools are often motivated by some specific use case and therefore the individual approaches are strongly biased and not flexible. All existing tools have quite different user interfaces, use different strategies of parsing, resolving, fetching, building with significantly different options for customizability. As a user of these tools it often feels like there is some part of the tool that suits the needs well, but at the same time it has undesirable hard coded behaviour. Often one would like to use some aspect of one tool combined with some aspect of another tool. One tool, for example, might do a good job in reading a specific lock file format, but lacks customizability for building. Another tool might come with a good customization interface, but is unable to parse the lock file format. Some tools are restricted to use IFD or FOD, while others enforce code generation.
The idea of this project is therefore to create a standardized, generic, modular, framework for 2nix tools, aiming for better flexibility, maintainability and usability.
The idea of this project is therefore to create a standardized, generic, modular framework for 2nix solutions, aiming for better flexibility, maintainability and usability.
### Modularity:
Individual phases like:
- parsing requiremnts
- parsing requirements
- resolving/locking dependencies
- fetching sources
- building/installing packages
@ -26,11 +27,11 @@ Individual phases like:
This will allow a free compsition of different approaches for these phases.
Examples:
- Often more than one requirements / lock-file format exists within an ecosystem. Switching between these formats, or adding support for a new one should be easy.
- Different resolving/fetching strategies: Some users might prefer a more automated approach via IFD or FOD, while others are focusing on upstreaming stuff to nixpkgs, where generating intermediary code or lock-files might be the only option.
- Different resolving/fetching strategies: Some users might prefer a more automated approach via IFD, while others are focusing on upstreaming stuff to nixpkgs, where generating intermediary code or lock-files might be the only option.
- Fetching a list of sources in theory should be a standard process. Yet, in practice, many 2nix tools struggle fetching sources from git or including local source trees. A generic fetching layer can reduce effort for maintainers.
### Customizability
Every Phase mentioned in the previous section should be customizable at a high degree via override functions. Examples:
Every Phase mentioned in the previous section should be customizable at a high degree via override functions. Practical examples:
- Inject extra requirements/dependencies
- fetch sources from alternative locations
- replace or modify sources
@ -41,26 +42,44 @@ Every Phase mentioned in the previous section should be customizable at a high d
Due to the modular architecture with well defined interfaces, contributers can add support for new lock-file formats or new strategies for fetching, building, installing more easily.
### Compatibility
Depending on where the nix code is used, different approaches are desired or discouraged. While IFD might be desired for some out of tree projects to achieve simplified UX, it is strictly prohibited in nixpkgs due to nix/hydra limitations.
All solutions which follow the dream2nix specification will be compatible with both approaches without having to re-invent the tool.
### Code de-duplication
Common problems that apply to many 2nix solutions can be solved once:
- handling cyclic dependencies
- handling sources from various origins (http, git, local, ...)
- generate nixpkgs/hydra friendly output (no IFD)
- good user interface
### Common UI across 2nix tools
2nix solutions which follow the dream2nix framework will have a unified UI for workflows like project initialization or gode generation. This will allow quicker onboarding of new users by providing familiar workflows across different build systems.
### Reduced effort to develop new 2nix solutions
Since the framework already solves common problems and provides interfaces for integrating new build systems, developers will have an easier time creating their next 2nix solution.
### Architecture
The general architecture should be like this:
`Input -> Translation -> URLs, metadata -> Fetching -> Building`
The general architecture should consist of these components:
`Input -> Translation -> Generic Lock -> Fetching -> Building`
```
┌───────┐
│ Input │◄──── Arbitrary
└────┬──┘ Build instructions in standardized
│ ┌───────────┐ minimalistic form (json)
└──►│Translation│ │
└────────┬──┘ ▼
▲ │ ┌──────────────┐
│ └──►│URLs, metadata│
└───────────┬──┘ reads instructions
impure/pure │ ┌────────┐ from metadata
online/offline └──►│Fetcher │ │
IFD/FOD/external └─────┬──┘ ▼
│ ┌────────┐
└──►│Builder │
└────────┘
│ Input │◄── Arbitrary
└────┬──┘ URLs + Metadata containing Build instructions
│ ┌──────────┐ in standardized minimalistic form (json)
└──►│Translator│ │
└───────┬──┘ ▼
▲ │ ┌────────────┐
│ └──►│Generic Lock│
└─────────┬──┘
impure/pure │ ┌────────┐
online/offline ├──►│Fetcher │◄── Same across all
IFD/external │ └────────┘ languages/frameworks
│ ▼
│ ┌────────┐
└──►│Builder │◄── Reads extra metadata
└────────┘ from generic lock
```
Input:
@ -68,20 +87,28 @@ Input:
- requirement files
- lock-files
Translation:
- read input and generate standardized output format containing URLs + hashes for sources and metadata required for the build
- different strategies can be used to achieve the required output:
- impure translation: resolve against online package index
- pure translation (IFD compatible): resolve against offline package index
- pure translation (FOD compatible): resolve against online index, reproducibly
Translator:
- read input and generate generic lock format containing URLs + hashes for sources and metadata required for the build
- different strategies can be used to generate the generic lock:
- pure-nix: translate input by using the nix language only
- IFD/recursive: translate using a nix build
- external: translate using external tool to resolve against online package index
- for more information about translators check [docs/translators.md](/docs/translators.md)
URLs, metadata (standardized format):
- Produced by `Translation`. Contains URLs + hashes for sources and metadata relevant for building. The basic format is standardized and equivalent across all languages/frameworks, so that fetching works always the same. It is not relevant which steps/strategies have been taken to create this data. From this point on, there are no impurities. This format will contain everything necessary for a fully reproducible build. The metadata allows different attributes for different languages/frameworks as those require individual approaches. A specific builder for every framework will later read this metadata and transform it into build instructions.
This format can always be copied into nixpkgs, not requiring any IFD (given the nix code for the builder exists within nixpkgs).
Generic Lock (standardized format):
- Produced by `Translator`. Contains URLs + hashes for sources and metadata relevant for building.
- The basic format is standardized and equivalent across all languages/frameworks, so that fetching works always the same.
- The metadata allows different attributes for different languages/frameworks as those require individual approaches for building. A specific builder for every framework will later read this metadata and transform it into build instructions.
- It is not relevant which steps/strategies have been taken to create this lock. From this point on, there are no impurities. This format will contain everything necessary for a fully reproducible build.
- This format can always be put into nixpkgs, not requiring any IFD (given the nix code for the builder exists within nixpkgs).
- In case of a pure-nix translator, dumping the generic lock to JSON can be omitted and instead the data be passed directly to the builder, preventing unnecessary IFD usage.
Fetching:
- Since a generic format was produced in the previous step, the fetching layer can be the same across all languages and frameworks.
Building:
- Inputs are the fetched sources coming from the fetcher and the metadata produced by the Translation layer. The builder now transforms the metadata into build instructions.
Fetcher:
- Since a generic lock was produced in the previous step, the fetching layer can be the same across all languages and frameworks.
Builder:
- Receives sources from fetcher and metadata produced by the translator.
- The builder transforms the metadata into build instructions.
- Strictly separating the builder from previous phases allows:
- switching between different build strategies or upgrading the builder without having to re-run the translator each time.
- reducing code duplication if a project contains multiple packages built via dream2nix.

40
docs/translators.md Normal file
View File

@ -0,0 +1,40 @@
# Translators
This document classifies different methods for translating requirement/lock files to the generic lock format and describes how the resulting packages can be integrated into nix builds inside and outside of nixpkgs.
## pure-nix (preferred)
Suitable if:
- the input contains information like URLs and hashes
- nix supports the hashing algorithm
- the input can be processed with the nix language directly
Usage outside of nixpkgs:
- The input + translator are enough, not requiring any pre-processing.
Usage inside nixpkgs:
- The input + translator are enough, not requiring any pre-processing.
## IFD/recursive (compatible with import from derivation or recursive nix)
Suitable if:
- the input contains information like URLs and hashes
- nix understands the hashing algorithm
- to process the input, a nix build is required, because for example:
- the format cannot be parsed with the nix language (yaml etc.)
- processing the input is too complex and therefore inefficient in nix language
Usage outside of nixpkgs:
- The input + translator are enough. The generic lock file is generated via IFD
Usage inside nixpkgs:
- generic lock file must be pre-generated using dream2nix cli
## external (running outside of nix build)
Suitable if the method used to process the input contains impurities, like for example:
- queries to an online index
- packages must be downloaded to require important meta data like dependencies.
Usage outside of nixpkgs:
- generic lock file must be pre-generated using dream2nix cli
Usage inside nixpkgs:
- generic lock file must be pre-generated using dream2nix cli

33
specifications/generic-lock-example.json Normal file
View File

@ -0,0 +1,33 @@
{
"sources": {
"requests": {
"url": "https://download.pypi.org/requests/2.28.0",
"hash": "000000000000000000000000000000000000000",
"type": "tarball"
},
"certifi": {
"url": "https://download.pypi.org/certifi/2.0",
"hash": "000000000000000000000000000000000000000",
"type": "tarball"
}
},
"meta-generic": {
"buildsystem": "python",
"buildsystemFormatVersion": 1,
"producedBy": "translator-poetry-1",
"system": "x86_64-linux",
"dependencyGraph": {
"requests": [
"certifi"
]
}
},
"meta-buildsystem": {
"sourceFormats": {
"requests": "sdist",
"certifi": "wheel"
}
}
}