mirror of
https://github.com/nix-community/dream2nix.git
synced 2024-08-17 02:10:31 +03:00
update readme/docs/schemas
This commit is contained in:
parent
fe2fdd25ae
commit
094b4c92fa
138
README.md
138
README.md
@ -1,33 +1,35 @@
|
||||
## [WIP] dream2nix - A generic framework for 2nix tools
|
||||
dream2nix is an approach to create a generic framework for 2nix tools.
|
||||
dream2nix is a generic framework for 2nix tools (tools translating from other build systems to nix).
|
||||
It focuses on the following aspects:
|
||||
- Modularity
|
||||
- Customizability
|
||||
- Maintainability
|
||||
- Compatibility (not enforcing features like IFD)
|
||||
- Nixpkgs Compatibility (not enforcing IFD)
|
||||
- Code de-duplication across 2nix tools
|
||||
- Common UI across 2nix tools
|
||||
- Reduce effort to develop new 2nix solutions
|
||||
|
||||
### Motivation
|
||||
2nix tools, or in other words, tools converting instructions of other build systems to nix build instructions, are an important part of the nix/nixos ecosystem. These tools make packaging workflows easier and often allow to manage complexity that would be hard/impossible to manage without the 2nix tools.
|
||||
2nix tools, or in other words, tools converting instructions of other build systems to nix build instructions, are an important part of the nix/nixos ecosystem. These tools make packaging workflows easier and often allow to manage complexity that would be hard or impossible to manage without.
|
||||
|
||||
Yet the current landscape of 2nix tools has certain weaknesses. Existing 2nix tools are very monolithic. Authors of these tools are often motivated by some specific use case and therefore the individual approaches are strongly biased and not flexible. All existing tools have quite different user interfaces, use different strategies of parsing, resolving, fetching, building with significantly different options for customizability. As a user of these tools it often feels like there is some part of the tool that suits the needs well, but at the same time it has undesirable hard coded behaviour. Often one would like to use some aspect of one tool combined with some aspect of another tool. One tool, for example, might do a good job in reading a specific lock file format, but lacks customizability for building. Another tool might come with a good customization interface, but is unable to parse the lock file format. Some tools are restricted to use IFD or FOD, while others enforce code generation.
|
||||
|
||||
The idea of this project is therefore to create a standardized, generic, modular framework for 2nix solutions, aiming for better flexibility, maintainability and usability.
|
||||
|
||||
Ideally this repository will become a hub for re-usable nix code delivered with a nice UI to allow for simpler, more automated packaging.
|
||||
|
||||
### Modularity:
|
||||
Individual phases like:
|
||||
- parsing requirements
|
||||
- resolving/locking dependencies
|
||||
The following phases which are generic to basically all existing 2nix solutions:
|
||||
- parsing project metadata
|
||||
- resolving/locking dependencies (not always required)
|
||||
- fetching sources
|
||||
- building/installing packages
|
||||
|
||||
... should be separated from each other with well defined interfaces.
|
||||
This will allow a free compsition of different approaches for these phases.
|
||||
This will allow for free compsition of different approaches for these phases.
|
||||
Examples:
|
||||
- Often more than one requirements / lock-file format exists within an ecosystem. Switching between these formats, or adding support for a new one should be easy.
|
||||
- Different resolving/fetching strategies: Some users might prefer a more automated approach via IFD, while others are focusing on upstreaming stuff to nixpkgs, where generating intermediary code or lock-files might be the only option.
|
||||
- Often more than one requirements / lock-file format exists within an ecosystem. Adding support for a new format should be easy and won't require re-inventing the whole procedure.
|
||||
- Different resolving/fetching strategies: Some users might prefer a more automated approach via IFD, while others are focusing on upstreaming to nixpkgs, where generating intermediary code or lock-files might be the only option.
|
||||
- Fetching a list of sources in theory should be a standard process. Yet, in practice, many 2nix tools struggle fetching sources from git or including local source trees. A generic fetching layer can reduce effort for maintainers.
|
||||
|
||||
### Customizability
|
||||
@ -39,25 +41,24 @@ Every Phase mentioned in the previous section should be customizable at a high d
|
||||
|
||||
|
||||
### Maintainability
|
||||
Due to the modular architecture with well defined interfaces, contributers can add support for new lock-file formats or new strategies for fetching, building, installing more easily.
|
||||
|
||||
Due to the modular architecture with strict interfaces, contributers can add support for new lock-file formats or new strategies for fetching, building, installing more easily.
|
||||
|
||||
### Compatibility
|
||||
Depending on where the nix code is used, different approaches are desired or discouraged. While IFD might be desired for some out of tree projects to achieve simplified UX, it is strictly prohibited in nixpkgs due to nix/hydra limitations.
|
||||
All solutions which follow the dream2nix specification will be compatible with both approaches without having to re-invent the tool.
|
||||
|
||||
### Code de-duplication
|
||||
Common problems that apply to many 2nix solutions can be solved once:
|
||||
Common problems that apply to many 2nix solutions can be solved once. Examples:
|
||||
- handling cyclic dependencies
|
||||
- handling sources from various origins (http, git, local, ...)
|
||||
- generate nixpkgs/hydra friendly output (no IFD)
|
||||
- good user interface
|
||||
|
||||
### Common UI across 2nix tools
|
||||
2nix solutions which follow the dream2nix framework will have a unified UI for workflows like project initialization or gode generation. This will allow quicker onboarding of new users by providing familiar workflows across different build systems.
|
||||
### Common UI across many 2nix tools
|
||||
2nix solutions which follow the dream2nix framework will have a unified UI for workflows like project initialization or code generation. This will allow quicker onboarding of new users by providing familiar workflows across different build systems.
|
||||
|
||||
### Reduced effort to develop new 2nix solutions
|
||||
Since the framework already solves common problems and provides interfaces for integrating new build systems, developers will have an easier time creating their next 2nix solution.
|
||||
Since the framework already solves common problems and provides an interface for integrating new build systems, developers will have an easier time creating their next 2nix solution.
|
||||
|
||||
### Architecture
|
||||
The general architecture should consist of these components:
|
||||
@ -75,7 +76,7 @@ The general architecture should consist of these components:
|
||||
└─────────┬──┘
|
||||
impure/pure │ ┌────────┐
|
||||
online/offline ├──►│Fetcher │◄── Same across all
|
||||
IFD/external │ └────────┘ languages/frameworks
|
||||
pure-nix/IFD/external │ └────────┘ languages/frameworks
|
||||
│ ▼
|
||||
│ ┌────────┐
|
||||
└──►│Builder │◄── Reads extra metadata
|
||||
@ -83,32 +84,113 @@ The general architecture should consist of these components:
|
||||
```
|
||||
|
||||
Input:
|
||||
- requirement contsraints
|
||||
- can consist of:
|
||||
- requirement contstraints
|
||||
- requirement files
|
||||
- lock-files
|
||||
|
||||
Translator:
|
||||
- read input and generate generic lock format containing URLs + hashes for sources and metadata required for the build
|
||||
- different strategies can be used to generate the generic lock:
|
||||
- pure-nix: translate input by using the nix language only
|
||||
- IFD/recursive: translate using a nix build
|
||||
- external: translate using external tool to resolve against online package index
|
||||
- for more information about translators check [docs/translators.md](/docs/translators.md)
|
||||
- read input and generate generic lock format containing:
|
||||
- URLs + hashes of sources
|
||||
- metadata for building
|
||||
- different strategies can be used:
|
||||
- `pure-nix`: translate input by using the nix language only
|
||||
- `IFD/recursive`: translate using a nix build
|
||||
- `external`: translate using an external tool which resolves against an online package index
|
||||
- for more information about translators and how nixpkgs compatibility is guaranteed, check [docs/translators.md](/docs/translators.md)
|
||||
|
||||
Generic Lock (standardized format):
|
||||
- Produced by `Translator`. Contains URLs + hashes for sources and metadata relevant for building.
|
||||
- The basic format is standardized and equivalent across all languages/frameworks, so that fetching works always the same.
|
||||
- The metadata allows different attributes for different languages/frameworks as those require individual approaches for building. A specific builder for every framework will later read this metadata and transform it into build instructions.
|
||||
- The contained format for sources and dependency relations is independent of the build system. Fetching works always the same.
|
||||
- The metadata also contains build system specific attributes as individual approaches are required here. A specific builder for the individual build system will later read this metadata and transform it into nix derivations.
|
||||
- It is not relevant which steps/strategies have been taken to create this lock. From this point on, there are no impurities. This format will contain everything necessary for a fully reproducible build.
|
||||
- This format can always be put into nixpkgs, not requiring any IFD (given the nix code for the builder exists within nixpkgs).
|
||||
- In case of a pure-nix translator, dumping the generic lock to JSON can be omitted and instead the data be passed directly to the builder, preventing unnecessary IFD usage.
|
||||
- In case of a pure-nix translator, the generic lock data can be generated on the fly and passed directly to the builder, preventing unnecessary usage of IFD.
|
||||
|
||||
Fetcher:
|
||||
- Since a generic lock was produced in the previous step, the fetching layer can be the same across all languages and frameworks.
|
||||
- Since a generic lock was produced in the previous step, the fetching layer can be the same across all build systems.
|
||||
|
||||
Builder:
|
||||
- Receives sources from fetcher and metadata produced by the translator.
|
||||
- The builder transforms the metadata into build instructions.
|
||||
- The builder transforms the metadata into nix derivation(s).
|
||||
- Strictly separating the builder from previous phases allows:
|
||||
- switching between different build strategies or upgrading the builder without having to re-run the translator each time.
|
||||
- reducing code duplication if a project contains multiple packages built via dream2nix.
|
||||
|
||||
|
||||
### Example (walk through the phases)
|
||||
#### python project with poetry.lock
|
||||
As an example we package a python project that uses poetry for dependency management.
|
||||
Potery uses `pyproject.toml` and `poetry.lock` to lock dependencies
|
||||
- Input: pyproject.toml, poetry.lock (toml)
|
||||
- Translator: written in pure nix, reading the toml input and generating the generic lock format
|
||||
- Generic Lock (for explanatory purposes dumped to json and commented):
|
||||
```json5
|
||||
{
|
||||
// generic lock format version
|
||||
"version": 1,
|
||||
|
||||
// format for sources is always the same (not specific to python)
|
||||
"sources": {
|
||||
"requests": {
|
||||
"type": "tarball",
|
||||
"url": "https://download.pypi.org/requests/2.28.0",
|
||||
"hash": "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef",
|
||||
},
|
||||
"certifi": {
|
||||
"type": "github",
|
||||
"owner": "certifi",
|
||||
"repo": "python-certifi",
|
||||
"hash": "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef"
|
||||
}
|
||||
},
|
||||
|
||||
// generic metadata (not specific to python)
|
||||
"generic": {
|
||||
|
||||
// this indicates which builder must be used
|
||||
"buildSystem": "python",
|
||||
|
||||
// versioning the format to ensure builder compatibility
|
||||
"buildSystemFormatVersion": 1,
|
||||
|
||||
// translator which generated this file
|
||||
// (not relevant for building)
|
||||
"producedBy": "translator-poetry-1",
|
||||
|
||||
// target system
|
||||
"system": "x86_64-linux",
|
||||
|
||||
// dependency graph of the packages
|
||||
"dependencyGraph": {
|
||||
"requests": [
|
||||
"certifi"
|
||||
]
|
||||
}
|
||||
},
|
||||
|
||||
// all fields inside 'buildSystem' are specific to
|
||||
// the selected buildSystem (python)
|
||||
"buildSystem": {
|
||||
|
||||
// tell the python builder how the inputs must be handled
|
||||
"sourceFormats": {
|
||||
"requests": "sdist", // triggers build instructions for sdist
|
||||
"certifi": "wheel" // triggers build instructions for wheel
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
- This lock data can now either:
|
||||
- be dumped to a .json file and committed to a repo
|
||||
- passed directly to the fetching/building layer
|
||||
- the fetcher will only read the sources section and translate it to standard fetcher calls.
|
||||
- the building layer will read the "buildSystem" attribute and select the python builder for building.
|
||||
- the python builder will read all information from "buildSystem" and translate the data to a final derivation.
|
||||
|
||||
Notes on IFD, FOD and code generation:
|
||||
- No matter which type of tanslator is used, it is always possible to export the generic lock to a file, which can later be evaluated without using IFD or FOD, similar to current nix code generators, just with a standardized format.
|
||||
- If the translator supports IFD or is written in pure nix, it is optional to the user to skip exporting the generic lock and instead evaluate everything on the fly.
|
||||
|
||||
|
||||
|
||||
|
@ -29,9 +29,11 @@ Usage inside nixpkgs:
|
||||
- generic lock file must be pre-generated using dream2nix cli
|
||||
|
||||
## external (running outside of nix build)
|
||||
Suitable if the method used to process the input contains impurities, like for example:
|
||||
- queries to an online index
|
||||
- packages must be downloaded to require important meta data like dependencies.
|
||||
Suitable if:
|
||||
- the input is missing URLs or hashes
|
||||
- the method used to process the input contains impurities, like for example:
|
||||
- queries to an online index with varying responses
|
||||
- packages must be downloaded to discover important meta data like dependencies.
|
||||
|
||||
Usage outside of nixpkgs:
|
||||
- generic lock file must be pre-generated using dream2nix cli
|
||||
|
@ -12,9 +12,9 @@
|
||||
}
|
||||
},
|
||||
|
||||
"meta-generic": {
|
||||
"buildsystem": "python",
|
||||
"buildsystemFormatVersion": 1,
|
||||
"generic": {
|
||||
"buildSystem": "python",
|
||||
"buildSystemFormatVersion": 1,
|
||||
"producedBy": "translator-poetry-1",
|
||||
"system": "x86_64-linux",
|
||||
"dependencyGraph": {
|
||||
@ -24,7 +24,7 @@
|
||||
}
|
||||
},
|
||||
|
||||
"meta-buildsystem": {
|
||||
"buildSystem": {
|
||||
"sourceFormats": {
|
||||
"requests": "sdist",
|
||||
"certifi": "wheel"
|
||||
|
101
specifications/generic-lock-schema.json
Normal file
101
specifications/generic-lock-schema.json
Normal file
@ -0,0 +1,101 @@
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema",
|
||||
"title": "manifest",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
||||
|
||||
"sources": {
|
||||
"type": "object",
|
||||
"patternProperties": {
|
||||
"^.*$": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {
|
||||
"enum": [
|
||||
"fetchurl",
|
||||
"git",
|
||||
"github",
|
||||
"gitlab",
|
||||
"tarball"
|
||||
]
|
||||
},
|
||||
"url": {
|
||||
"type": "string"
|
||||
},
|
||||
"hash": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"allOf": [
|
||||
{
|
||||
"if": {
|
||||
"properties": { "type": { "const": "fetchurl" } }
|
||||
},
|
||||
"then": {
|
||||
"properties": { "url": { "type": "string" } }
|
||||
}
|
||||
},
|
||||
{
|
||||
"if": {
|
||||
"properties": { "type": { "const": "git" } }
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"url": { "type": "string" },
|
||||
"rev": { "type": "string" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"if": {
|
||||
"properties": { "type": { "pattern": "(github)|(gitlab)" } }
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"owner": { "type": "string" },
|
||||
"repo": { "type": "string" },
|
||||
"rev": { "type": "string" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"if": {
|
||||
"properties": { "type": { "const": "tarball" } }
|
||||
},
|
||||
"then": {
|
||||
"properties": { "url": { "type": "string" } }
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
|
||||
"generic": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"buildSystem": { "type": "string" },
|
||||
"buildSystemFormatVersion": { "type": "integer" },
|
||||
"producedBy": { "type": "string" },
|
||||
"system": { "type": "string" },
|
||||
"dependencyGraph": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"^.*$": {
|
||||
"type": "array",
|
||||
"items": { "type": "string" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
|
||||
"buildSystem": {
|
||||
"description": "build system specifics",
|
||||
"type": "object"
|
||||
}
|
||||
}
|
||||
}
|
Loading…
Reference in New Issue
Block a user