Commit Graph

5887 Commits

Author SHA1 Message Date
Jörg Thalheim
83c19c791d
Merge pull request #200757 from Flakebi/rocm-test
nixos/tests: add ROCm and amdvlk tests
2022-11-23 09:56:05 +01:00
figsoda
82fe76d1cd carnix,cratesIO: remove 2022-11-22 14:10:25 -05:00
Sebastian Neubauer
76a4adc19c makeImpureTest: init function for hardware tests
Add a function to create tests that use hardware from the host system
like GPUs.
2022-11-22 16:54:32 +01:00
Winter
009a234bdd prefetch-npm-deps: repack hosted git deps
Previously, we stored the tarballs from the hosted Git providers directly in the cache. However, as we've seen with `fetchFromGitHub` etc, these files may change subtly.

Given this, this commit repacks the dependencies before storing them in the cache.
2022-11-21 15:00:08 -05:00
Winter
091d039b12 prefetch-npm-deps: deduplicate dependencies when parsing lockfile
A `package-lock.json` file can contain multiple instances of the same dependency, which caused unnecessary downloads and duplicate index entries in the generated cache.
2022-11-21 15:00:08 -05:00
Winter
b117b359db prefetch-npm-deps: move tests to separate file 2022-11-21 15:00:08 -05:00
Winter
3d5f77f496 npmHooks.npmConfigHook: sugget specifying legacy peer deps option for all commands 2022-11-21 15:00:08 -05:00
Winter
93334f5234 npmHooks.npmInstallHook: pass install flags to prune 2022-11-21 15:00:08 -05:00
Winter
80ecb954e9 npmHooks.npmConfigHook: add diagnostic for when dependencies aren't provided 2022-11-21 15:00:08 -05:00
Winter
c0e7124bcc npmHooks.npmConfigHook: add some missing quotes around variable assignments 2022-11-21 15:00:08 -05:00
Winter
68d5b19fca npmHooks.npmConfigHook: also patch shebangs after rebuilding 2022-11-21 15:00:08 -05:00
Winter
8e651111b7 npmHooks.npmBuildHook: mention dontNpmBuild if script fails to run 2022-11-21 15:00:08 -05:00
Winter
c6903a01e7 prefetch-npm-deps: add etherpad-lite v1.8.18 as a test 2022-11-21 15:00:08 -05:00
Winter
03a7739d75 prefetch-npm-deps: skip bundled dependencies in v1 lockfiles 2022-11-21 15:00:08 -05:00
Winter
276982bd5e prefetch-npm-deps: add support for hosted git shorthands 2022-11-21 15:00:08 -05:00
Winter
cc5756b171 prefetch-npm-deps: add support for link dependencies in v2 lockfiles 2022-11-21 15:00:08 -05:00
Sandro
3a05360e53
Merge pull request #200082 from panicgh/fetchgit-sparse-checkout 2022-11-21 20:00:56 +01:00
Sandro
702768bc28
Merge pull request #202174 from mupdt/containers-prefer-local-builds 2022-11-21 19:49:54 +01:00
Elis Hirwing
566793db55
Merge pull request #198037 from SuperSandro2000/php-cleanup
phpExtensions: little cleanup
2022-11-21 08:25:31 +01:00
Robert Hensing
29352751cf
Merge pull request #196251 from hercules-ci/testers-build-failure-and-equal-contents
`testers`: Add `testBuildFailure` and `testEqualContents`
2022-11-21 06:17:00 +01:00
Sandro Jäckel
023798d172 build-pecl: fetch via https 2022-11-21 00:27:34 +01:00
Artturin
341e6fd558 splice.nix: start deprecating nativeDrv and crossDrv 2022-11-19 00:04:54 +02:00
mupdt
bb5827a692 dockerTools: prefer local builds 2022-11-18 09:31:53 -05:00
figsoda
7c6353c884
Merge pull request #201472 from ncfavier/remove-dead-make-symlinks
remove dead `make-symlinks` builder
2022-11-16 14:39:44 -05:00
Aaron Andersen
adbe2f532c
Merge pull request #199300 from Fuuzetsu/escape-rust-exports
rustBuildCrate: properly handle cargo env pragmas with spaces
2022-11-16 07:15:25 -05:00
Naïm Favier
4d68c2f0ba
remove dead make-symlinks builder
Introduced in 5808ac7148, never used again
after b06335a835.
2022-11-16 11:20:50 +01:00
Nicolas Benes
c95d7d5a8c treewide: make sparseCheckout a list of strings 2022-11-15 19:46:12 +01:00
Nicolas Benes
f6b07f0e2f fetchgit: make sparseCheckout a list of strings
The `sparseCheckout` argument allows the user to specify directories or
patterns of files, which Git uses to filter files it should check-out.

Git expects a multi-line string on stdin ("newline-delimited list", see
`git-sparse-checkout(1)`), but within nixpkgs it is more consistent to
use a list of strings instead. The list elements are joined to a
multi-line string only before passing it to the builder script.

A deprecation warning is emitted if a (multi-line) string is passed to
`sparseCheckout`, but for the time being it is still accepted.
2022-11-15 19:45:33 +01:00
Bernardo Meurer
43bf542ccd
tests.trivial-builders.linkFarm: init 2022-11-15 11:47:36 -05:00
Bernardo Meurer
6382598677
linkFarm: make last entry win in case of list repeats 2022-11-15 11:45:20 -05:00
Robert Hensing
449ae7573b
Merge pull request #170048 from lovesegfault/link-farm-passthru
linkFarm: add entries to passthru
2022-11-15 17:39:43 +01:00
Bernardo Meurer
a93aed5bab
linkFarm: allow entries to be an attrset 2022-11-15 11:01:52 -05:00
Bernardo Meurer
b54257fb36
linkFarm: add entries to passthru 2022-11-15 10:59:31 -05:00
github-actions[bot]
f3a93620b1
Merge master into staging-next 2022-11-10 12:01:27 +00:00
Shamrock Lee
6386d79326 patchRcPath hooks: use the passthru argument 2022-11-10 09:27:40 +08:00
Shamrock Lee
da31bd5673 patchRcPathBash, patchRcPathCsh, patchRcPathFish, patchRcPathPosix: init
Init patchRcPath hooks, which provides
utilities to patch shell scripts to be sourced by users.

Add test cases and documentation.
2022-11-10 09:23:52 +08:00
github-actions[bot]
020ea91183
Merge master into staging-next 2022-11-10 00:02:27 +00:00
John Ericson
e94d54dd86 build-support/rust/lib: Add toTargetFamily
Taken from https://github.com/kolloch/crate2nix/pull/255/files, it
belongs in Nixpkgs not crate2nix.

I have been using that P.R. for a few months without incident.
2022-11-09 15:52:01 -05:00
github-actions[bot]
d99020653d
Merge master into staging-next 2022-11-09 18:01:11 +00:00
Winter
16722903aa buildNpmPackage: init 2022-11-09 10:35:59 -05:00
github-actions[bot]
e51ba60580
Merge master into staging-next 2022-11-07 00:03:30 +00:00
Yarny0
335a9083b0 patchPpdFilesHook: new setup hook for absolute executable paths
PostScript Printer Description (ppd) files
describe printer features and capabilities.
They are usually evaluated by CUPS to convert
print jobs into a format suitable for a printer.
The conversion is often accomplished by commands
or even short shell scripts inside the ppd files.

ppd files are included in many printer driver packages.
Their scripts sometimes refer to other executables;
some of them are more common (like `perl`),
others are more exotic (like `rastertohp`).
If an executable is called with its name alone,
the effects of the ppd file depend on whether
the executable is in the PATH of CUPS,
and on the executable's version.
If an executable is called with an absolut path
(like `/usr/bin/perl`), it won't work at all in NixOS.

The commit at hand adds a setup hook that uses
the `fixupPhase` to substitute certain executable's
invocations in pdd files with absolute paths.
To use it, add the hook to `nativeBuildInputs` and
provide a list of executable names in `ppdFileCommands`.
Each executable must be available in the
package that is being built, or in `buildInputs`.
The setup hook's script then looks for ppd files in
`share/cups/model` and `share/ppds` in each output,
and replaces executable names with their absolute paths.
If ppd files need to be patched in unorthodox locations or
the setup hook needs to be invoked manually for other reasons,
one may leave the list `ppdFileCommands` empty to
avoid automatic processing of ppd files, then call
the shell function `patchPpdFileCommands` directly.
Details are described in the file `patch-ppd-hook.sh`.

Notes on the motivation for this setup hook:

Most packages in nixpkgs that provide
ppd files do not patch those ppd files at all.
This is not fatal when the executables are just called
with their names since the user can add packages
with the executables to `services.printing.drivers`.
E.g. if the user adds `pkgs.perl`, then all ppd
files that invoke `perl` will work as expected.
Nevertheless, to make these ppd files independent of
their execution environment, command invocations should
be substituted with absolut paths into the nix store.
This is similar to patching shebang lines so scripts can be
called independently of having the interpreter in the PATH.

The hook script in this commit is meant to support new packages
`foomatic-db*` which will generate several thousands of
ppd files referencing a plethora of different executables.
During development of these packages, I realized that
it's quite hard to patch ppd files in a robust way.
While binary names like `rastertokpsl` seem to be sufficiently
unique to be patched with `sed`, names like `date` or `gs`
are hard to patch without producing "false positives",
i.e., coincidental occurences of the executable's name that do
*not* refer to the executable and should not be patched at all.
As this problem also affects other packages,
it seems reasonable to put a robust implementation
in its own setup hook so that other
packages can use it without much effort.

Notes on the implementation:

The ppd file format is far from trivial.
The basic structure are key-value pairs;
keys may occur multiple times.
Only a small subset of keys may contain
executable names or shell scripts in their values.
Some values may span multiple lines;
a linebreak might even occur in the middle of a token.
Some executable names also occur in other keys by accident
where they must not be substituted (e.g. `gs` or `date`).

It is necessary to provide the list of command
names that will be patched for two reasons:
ppd files often contain "tokens" that might look
like commands (e.g. "file" or "host") but aren't;
these would erroneously get patched.
Also, looking for everything that might be a command
would slow down the patching process considerably.

The implementation uses `awk` to detect
keys that might contain executable names;
only their values are treated for substitution.
This avoids most cases of "overzealous" substitutions.
Since values may span multiple lines,
`sed` alone (while faster than `awk`) cannot focus
its substitution capabilities on relevant keys.
An elaborate set of regular expressions further helps
to minimize the probability of "false positives".
Several tricks are employed to speed up `awk`.
Notably, relevant files are identified with
`grep` before `awk` is applied to those files only.
Note that the script probably cannot handle fancy command
names (like spaces or backslashes as part of the name).
Also, there are still edge cases that the script would
mistakenly skip, e.g. if a shell script contains a
line break in the middle of an executable's name;
although ppd files permit such constellations,
I have yet to see one.

ppd files may be gzipped.
The setup hook accepts gzipped ppd files:
It decompresses them, substitutes paths, then recompresses them.
However, Nix cannot detect substituted paths as
runtime dependencies in compressed ppd files.
To ensure substituted paths are propagated as
runtime dependencies, the script adds each substituted
path to the variable `propagatedBuildInputs`.
Since this might not be enough for multi-output packages,
those paths are also written directly to
`nix-support/propagated-build-inputs`.
See the comment in `patch-ppd-hook.sh` for details.

Finally, the setup hook comes with a small test that
probes some edge cases with an artificial ppd file.

References:

* https://www.cups.org/doc/spec-ppd.html
  * general ppd file specification
  * lists some keys that may contain
    executable names or shell scripts

* https://refspecs.linuxfoundation.org/LSB_4.0.0/LSB-Printing/LSB-Printing/ppdext.html
  * lists some keys that may contain
    executable names or shell scripts

* https://en.wikipedia.org/wiki/PostScript_Printer_Description#CUPS
  * lists the usual locations of ppd files
2022-11-06 16:19:22 +01:00
github-actions[bot]
cfd5f0bddc
Merge staging-next into staging 2022-11-05 00:02:45 +00:00
Artturi
1cdecacb79
Merge pull request #198489 from Artturin/fetchgitlabremoteavvt
fetchFromGitLab: don't passthruAttrs fetchGit specific attrs to fetchUrl
2022-11-04 20:29:38 +02:00
Mateusz Kowalczyk
233205c464
rustBuildCrate: properly handle cargo env pragmas with spaces
There are two problems: first that we end up splitting on spaces in the
loop. Even when that is fixed, we still would split on spaces in the
`export` inside the loop. We need to guard against both.

Fixes #199298

Confirmed that it fixes the case mentioned in the ticket:

```console
[nix-develop]$ $(nix-build -I nixpkgs=/home/shana/programming/nixpkgs Cargo.nix -A rootCrate.build  --no-out-link)/bin/nix-rustc-env-escape-repro
Expecting three words, got: first second third
```

I think this is going to cause a rebuild of every Rust package even if
they were unaffected, not much we can do here.
2022-11-03 14:49:24 +09:00
Yureka
c2eaaae50d cargoSetupHook: pass host config flags 2022-10-30 18:55:47 +01:00
Artturin
19d2bb4863 fetchFromGitLab: don't passthruAttrs fetchGit specific attrs to fetchUrl
if `fetchSubmodules = false` to 'fetchFromGitLab' then theres the
following error

error: anonymous function at /nix/store/9m8drnpifyl5qsx93g6ll2xw6wkps03z-source/pkgs/build-support/fetchurl/default.nix:41:1 called with unexpected argument 'fetchSubmodules'

       at /nix/store/9m8drnpifyl5qsx93g6ll2xw6wkps03z-source/pkgs/build-support/fetchzip/default.nix:36:1:

           35|
           36| fetchurl ((
             | ^
           37|   if (pname != "" && version != "") then
2022-10-29 19:12:07 +03:00
github-actions[bot]
c45b1c2226
Merge staging-next into staging 2022-10-29 00:02:59 +00:00
github-actions[bot]
f349a716a3
Merge master into staging-next 2022-10-29 00:02:26 +00:00
Anderson Torres
f367ac90aa
Merge pull request #197997 from atorres1985-contrib/hummingbird
skawarePackages: reformat
2022-10-28 17:10:53 -03:00