eda44b7415
This file walks the entire nixpkgs tree and emits a superset of all release attrnames in only 44 seconds on a 3ghz CPU, using 5 gbytes of memory. By comparison, on the same CPU the `nix-env` hack used by ofborg on every PR submission requires 41 *minutes* and peaks at 60 gbytes, even with checkMeta turned off. Full details below. This is: - 46x faster (or 2.1% of the elapsed time) - 12.5x less memory (or 8.0% of the peak memory usage) In order to replace the ofborg check, this list of attrnames must then be post-filtered for platform-relevance. However, crucially, the post-filtering can be done *in parallel* on multiple cores by splitting the attrname list in to chunks. Generating the list of attrnames cannot be parallelized because it is a single-threaded cppnix task. This PR also adds `recurseForDerivations` where necessary within nixpkgs in order to make this possible -- it screens out various non-tryEval-catchable failures and infinite recursions. Before undraftifying, I will add an invocation of this command to the CI tests, to ensure that the work performed here is not immediately undone. My next PR will then add an additional CI check confirming that the emitted attrpaths are in fact a superset of the release attrpaths calculated by the slow-memory-hog ofborg method. I have manually confirmed that this is the case at the tip commit of this PR, but we need CI to make sure this remains true until ofborg switches to this more-efficient method of calculation; at that point the superset-check can be dropped. According to GNU Time, Command being timed: "nix-instantiate --eval --strict --json pkgs/top-level/release-attrpaths-superset.nix -A names" User time (seconds): 44.88 System time (seconds): 8.09 Percent of CPU this job got: 99% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:53.20 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 4823028 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 3611240 Voluntary context switches: 113 Involuntary context switches: 949 Swaps: 0 File system inputs: 1480 File system outputs: 5944 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 Compared to release-outpaths.nix: Command being timed: "nix-env -qaP --no-name --out-path --arg checkMeta false --argstr path /git/work/pr/release-outpaths -f pkgs/top-level/release-outpaths.nix" User time (seconds): 2120.67 System time (seconds): 337.80 Percent of CPU this job got: 98% Elapsed (wall clock) time (h:mm:ss or m:ss): 41:37.91 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 60171768 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 2 Minor (reclaiming a frame) page faults: 230608113 Voluntary context switches: 8876 Involuntary context switches: 22275 Swaps: 0 File system inputs: 62624 File system outputs: 72 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 |
||
---|---|---|
.github | ||
doc | ||
lib | ||
maintainers | ||
nixos | ||
pkgs | ||
.editorconfig | ||
.git-blame-ignore-revs | ||
.gitattributes | ||
.gitignore | ||
.mailmap | ||
.version | ||
CONTRIBUTING.md | ||
COPYING | ||
default.nix | ||
flake.nix | ||
README.md |
Nixpkgs is a collection of over 80,000 software packages that can be installed with the Nix package manager. It also implements NixOS, a purely-functional Linux distribution.
Manuals
- NixOS Manual - how to install, configure, and maintain a purely-functional Linux distribution
- Nixpkgs Manual - contributing to Nixpkgs and using programming-language-specific Nix expressions
- Nix Package Manager Manual - how to write Nix expressions (programs), and how to use Nix command line tools
Community
- Discourse Forum
- Matrix Chat
- NixOS Weekly
- Community-maintained wiki
- Community-maintained list of ways to get in touch (Discord, Telegram, IRC, etc.)
Other Project Repositories
The sources of all official Nix-related projects are in the NixOS organization on GitHub. Here are some of the main ones:
- Nix - the purely functional package manager
- NixOps - the tool to remotely deploy NixOS machines
- nixos-hardware - NixOS profiles to optimize settings for different hardware
- Nix RFCs - the formal process for making substantial changes to the community
- NixOS homepage - the NixOS.org website
- hydra - our continuous integration system
- NixOS Artwork - NixOS artwork
Continuous Integration and Distribution
Nixpkgs and NixOS are built and tested by our continuous integration system, Hydra.
- Continuous package builds for unstable/master
- Continuous package builds for the NixOS 23.11 release
- Tests for unstable/master
- Tests for the NixOS 23.11 release
Artifacts successfully built with Hydra are published to cache at https://cache.nixos.org/. When successful build and test criteria are met, the Nixpkgs expressions are distributed via Nix channels.
Contributing
Nixpkgs is among the most active projects on GitHub. While thousands of open issues and pull requests might seem a lot at first, it helps consider it in the context of the scope of the project. Nixpkgs describes how to build tens of thousands of pieces of software and implements a Linux distribution. The GitHub Insights page gives a sense of the project activity.
Community contributions are always welcome through GitHub Issues and Pull Requests.
For more information about contributing to the project, please visit the contributing page.
Donations
The infrastructure for NixOS and related projects is maintained by a nonprofit organization, the NixOS Foundation. To ensure the continuity and expansion of the NixOS infrastructure, we are looking for donations to our organization.
You can donate to the NixOS foundation through SEPA bank transfers or by using Open Collective:
License
Nixpkgs is licensed under the MIT License.
Note: MIT license does not apply to the packages built by Nixpkgs, merely to the files in this repository (the Nix expressions, build scripts, NixOS modules, etc.). It also might not apply to patches included in Nixpkgs, which may be derivative works of the packages to which they apply. The aforementioned artifacts are all covered by the licenses of the respective packages.