Commit Graph

445 Commits

Author SHA1 Message Date
github-actions[bot]
20df8d00cc
Merge master into staging-next 2023-01-25 12:01:34 +00:00
Nick Cao
0ea2d480e4
postgresqlTestHook: run postgresqlTestSetupPost hook after setting up 2023-01-25 18:46:27 +08:00
Guillaume Girol
33afbf39f6 treewide: switch to nativeCheckInputs
checkInputs used to be added to nativeBuildInputs. Now we have
nativeCheckInputs to do that instead. Doing this treewide change allows
to keep hashes identical to before the introduction of
nativeCheckInputs.
2023-01-21 12:00:00 +00:00
Guillaume Girol
25193e2732 postgresqlTestHook: adapt comment to nativeCheckInputs 2023-01-21 16:42:10 +01:00
Guillaume Girol
90c78aee6c Merge branch 'nativeCheckInputs' into staging-nativeCheckInputs 2023-01-21 12:00:00 +00:00
Robert Hensing
a1cd06f900
Merge pull request #211082 from hercules-ci/fix-issue-16182
Fix issue 16182  `Error: _assignFirst found no valid variant!`
2023-01-19 19:22:04 +01:00
Alyssa Ross
0ae87d514f treewide: add names to all setup hooks 2023-01-19 15:00:36 +00:00
Robert Hensing
c696a19bfa multiple-outputs.sh: Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-01-17 21:18:59 +01:00
Robert Hensing
01d7f19346 multi-outputs.sh: Improve _assignFirst error message
Closes #16182

This improves the error message

    Error: _assignFirst found no valid variant!

which occurred when the set of outputs was not sufficient to set
the various outputDev, outputBin, etc variables. Specifically, this
would mean that "out" is not among the outputs, which is valid for
a derivation.

This changes the message to something like

    error: _assignFirst: could not find a non-empty variable to assign to outputDev. The following variables were all unset or empty: dev out.
          If you did not define an "out" output, make sure to define all the specific required outputs: define an output for one of the unset variables.

While this isn't a full explanation of what stdenv can and can not do,
I think it's vast improvement over the 0 bits of information that it
used to provide. This at least gives a clue as to what's going on, and
even suggests a fix, although probably multiple such fixes are required
in an instance where someone starts with a no-out derivation from scratch
(and decide to persist).
2023-01-16 12:57:01 +01:00
Artturin
fe1c7a1945 treewide: remove usages of header and stopNest
they're obsolete
2023-01-16 00:08:12 +02:00
Guillaume Maudoux
aa7cfc3aed
Merge pull request #210908 from Yarny0/auto-patchelf
autoPatchelfHook: fix symlink handling
2023-01-15 17:18:01 +01:00
Yarny0
818d0f8cf1 auto-patchelf: don't resolve symlinks if basenames don't match
The auto-patchelf python script assembles a list of
library (so=shared object) file names and their paths.
This helps speed up the discovery of
library files later when patching elf files.
As further optimization, if a symlink points to a library file,
the script uses the resolved path and file name.
However, this produces a broken list entry if the
symlink's target name doesn't match the symlink's name.

A symptom of the bug, affecting the `tsm-client` package,
is fixed in https://github.com/NixOS/nixpkgs/pull/172372 .

The commit at hand stops resolving symlinks if
the target name differs from the symlink's name.
The commit has been authored by
layus (Guillaume Maudoux <layus.on@gmail.com>)
in pull request comment

https://github.com/NixOS/nixpkgs/pull/172372#issuecomment-1194687183
2023-01-15 10:57:22 +01:00
Artturin
033ec05960 setup-hooks/strip.sh: redirect stdout to dev/null
to not unnecessarily print

```
hello> ++ type -f strip
hello> strip is /nix/store/xzqzq99kbbkad23s2bf4bbrjppg6x0vj-bootstrap-stage2-gcc-wrapper-/bin/strip
```

printing what the strip command is is already done in stripDirs

`echo "stripping (with command $cmd and flags $stripFlags) in $paths"`
2023-01-08 21:02:22 +02:00
Artturin
ba11c6f123 setup-hooks/make-symlinks-relative.sh: match what other hooks do so
the hook doesn't output the following if $prefix doesn't exist

find: '/nix/store/...': No such file or directory

some other hooks that do this: prune-libtool-files.sh, audit-tmpdir.sh separate-debug-info.sh
2023-01-06 18:56:54 +02:00
Artturin
3b3ef7b766 audit-tmpdir.sh: fix on darwin
compress-man-pages.sh uses `find ... | while ...; do...` so this works
on darwin

vcunat says the following in https://github.com/NixOS/nixpkgs/commit/94d164dd7#r22030369

> Unfortunately, process substitution doesn't work during early darwin bootstrap. http://hydra.nixos.org/build/52433342 We've run into that before.
> I think it should work to have `find ... | while ...; do...`
2022-12-13 00:57:35 +02:00
Artturin
af70ba03d3 build-support/setup-hooks: change shebang to shellcheck directive 2022-12-13 00:24:33 +02:00
Artturin
daab80e08d move-docs.sh: update comment
_multioutDocs is in multiple-outputs.sh
2022-12-12 21:39:56 +02:00
Robert Hensing
16f5747575
Merge pull request #175649 from Artturin/opt-in-structured-attrs
stdenv: support opt-in __structuredAttrs
2022-12-10 21:12:43 +01:00
Artturin
238a6053c4 stdenv: support opt-in __structuredAttrs
Co-authored-by: Robin Gloster <mail@glob.in>

stdenv: print message if structuredAttrs is enabled

stdenv: add _append

reduces the chance of a user doing it wrong

fix nix develop issue

output hooks don't work yet in nix develop though

making $outputs be the same on non-structuredAttrs and structuredAttrs
is too much trouble.

lets instead make a function that gets the output names

reading environment file '/nix/store/2x7m69a2sm2kh0r6v0q5s9z1dh41m4xf-xz-5.2.5-env-bin'
nix: src/nix/develop.cc:299: std::string Common::makeRcScript(nix::ref<nix::Store>, const BuildEnvironment&, const Path&): Assertion `outputs != buildEnvironment.vars.end()' failed.

use a function to get all output names instead of using $outputs

copy env functionality from https://github.com/NixOS/nixpkgs/pull/76732/commits
2022-12-08 06:13:19 +02:00
Artturin
6c01681679 make-symlinks-relative: fix no such file or directory if output is created in postFixup
wlroots(and others) have
```
wlroots> +++ find /nix/store/3a0xwszw8n5dzzhsgfnilvsqi4hk565s-wlroots-0.15.1-examples -type l -print0
wlroots> find: '/nix/store/3a0xwszw8n5dzzhsgfnilvsqi4hk565s-wlroots-0.15.1-examples': No such file or directory
```

because the examples output is created in postFixup while this hook runs in fixupPhase
2022-12-06 00:08:11 +02:00
Sergei Trofimovich
1238d0ed39
Merge pull request #182055 from wentasah/moveToOutput-comment-fix
multiple-output.sh: fix moveToOutput comment
2022-11-18 22:40:50 +00:00
Shamrock Lee
6386d79326 patchRcPath hooks: use the passthru argument 2022-11-10 09:27:40 +08:00
Shamrock Lee
da31bd5673 patchRcPathBash, patchRcPathCsh, patchRcPathFish, patchRcPathPosix: init
Init patchRcPath hooks, which provides
utilities to patch shell scripts to be sourced by users.

Add test cases and documentation.
2022-11-10 09:23:52 +08:00
Yarny0
335a9083b0 patchPpdFilesHook: new setup hook for absolute executable paths
PostScript Printer Description (ppd) files
describe printer features and capabilities.
They are usually evaluated by CUPS to convert
print jobs into a format suitable for a printer.
The conversion is often accomplished by commands
or even short shell scripts inside the ppd files.

ppd files are included in many printer driver packages.
Their scripts sometimes refer to other executables;
some of them are more common (like `perl`),
others are more exotic (like `rastertohp`).
If an executable is called with its name alone,
the effects of the ppd file depend on whether
the executable is in the PATH of CUPS,
and on the executable's version.
If an executable is called with an absolut path
(like `/usr/bin/perl`), it won't work at all in NixOS.

The commit at hand adds a setup hook that uses
the `fixupPhase` to substitute certain executable's
invocations in pdd files with absolute paths.
To use it, add the hook to `nativeBuildInputs` and
provide a list of executable names in `ppdFileCommands`.
Each executable must be available in the
package that is being built, or in `buildInputs`.
The setup hook's script then looks for ppd files in
`share/cups/model` and `share/ppds` in each output,
and replaces executable names with their absolute paths.
If ppd files need to be patched in unorthodox locations or
the setup hook needs to be invoked manually for other reasons,
one may leave the list `ppdFileCommands` empty to
avoid automatic processing of ppd files, then call
the shell function `patchPpdFileCommands` directly.
Details are described in the file `patch-ppd-hook.sh`.

Notes on the motivation for this setup hook:

Most packages in nixpkgs that provide
ppd files do not patch those ppd files at all.
This is not fatal when the executables are just called
with their names since the user can add packages
with the executables to `services.printing.drivers`.
E.g. if the user adds `pkgs.perl`, then all ppd
files that invoke `perl` will work as expected.
Nevertheless, to make these ppd files independent of
their execution environment, command invocations should
be substituted with absolut paths into the nix store.
This is similar to patching shebang lines so scripts can be
called independently of having the interpreter in the PATH.

The hook script in this commit is meant to support new packages
`foomatic-db*` which will generate several thousands of
ppd files referencing a plethora of different executables.
During development of these packages, I realized that
it's quite hard to patch ppd files in a robust way.
While binary names like `rastertokpsl` seem to be sufficiently
unique to be patched with `sed`, names like `date` or `gs`
are hard to patch without producing "false positives",
i.e., coincidental occurences of the executable's name that do
*not* refer to the executable and should not be patched at all.
As this problem also affects other packages,
it seems reasonable to put a robust implementation
in its own setup hook so that other
packages can use it without much effort.

Notes on the implementation:

The ppd file format is far from trivial.
The basic structure are key-value pairs;
keys may occur multiple times.
Only a small subset of keys may contain
executable names or shell scripts in their values.
Some values may span multiple lines;
a linebreak might even occur in the middle of a token.
Some executable names also occur in other keys by accident
where they must not be substituted (e.g. `gs` or `date`).

It is necessary to provide the list of command
names that will be patched for two reasons:
ppd files often contain "tokens" that might look
like commands (e.g. "file" or "host") but aren't;
these would erroneously get patched.
Also, looking for everything that might be a command
would slow down the patching process considerably.

The implementation uses `awk` to detect
keys that might contain executable names;
only their values are treated for substitution.
This avoids most cases of "overzealous" substitutions.
Since values may span multiple lines,
`sed` alone (while faster than `awk`) cannot focus
its substitution capabilities on relevant keys.
An elaborate set of regular expressions further helps
to minimize the probability of "false positives".
Several tricks are employed to speed up `awk`.
Notably, relevant files are identified with
`grep` before `awk` is applied to those files only.
Note that the script probably cannot handle fancy command
names (like spaces or backslashes as part of the name).
Also, there are still edge cases that the script would
mistakenly skip, e.g. if a shell script contains a
line break in the middle of an executable's name;
although ppd files permit such constellations,
I have yet to see one.

ppd files may be gzipped.
The setup hook accepts gzipped ppd files:
It decompresses them, substitutes paths, then recompresses them.
However, Nix cannot detect substituted paths as
runtime dependencies in compressed ppd files.
To ensure substituted paths are propagated as
runtime dependencies, the script adds each substituted
path to the variable `propagatedBuildInputs`.
Since this might not be enough for multi-output packages,
those paths are also written directly to
`nix-support/propagated-build-inputs`.
See the comment in `patch-ppd-hook.sh` for details.

Finally, the setup hook comes with a small test that
probes some edge cases with an artificial ppd file.

References:

* https://www.cups.org/doc/spec-ppd.html
  * general ppd file specification
  * lists some keys that may contain
    executable names or shell scripts

* https://refspecs.linuxfoundation.org/LSB_4.0.0/LSB-Printing/LSB-Printing/ppdext.html
  * lists some keys that may contain
    executable names or shell scripts

* https://en.wikipedia.org/wiki/PostScript_Printer_Description#CUPS
  * lists the usual locations of ppd files
2022-11-06 16:19:22 +01:00
adisbladis
01535ff0b0 autoPatchelfHook: support glob patterns
so for example cuda could be ignored by setting:
``` nix
autoPatchelfIgnoreMissingDeps = [ "*cuda*.so*" ];
```
2022-10-10 17:55:11 +13:00
adisbladis
41c09640e7 autoPatchelfHook: fix turning [ "*" ] into bash array
Previously globs were incorrectly handled and expanded by bash into the files in the temporary build directory.
2022-10-10 17:55:11 +13:00
Robert Hensing
8deb17a36e
Merge pull request #186323 from ShamrockLee/make-setuphook-passthru
trivial-builders.nix: Add input argument `passthru` to makeSetupHook
2022-09-19 11:30:37 +01:00
Robert Hensing
c078d552fe wrapGAppsHook: Set name
The previous commit, removing substitutions, causes a mass rebuild,
so we use the opportunity to set the name; also a mass rebuild.
2022-08-13 11:53:06 +02:00
Robert Hensing
4dc28e0057 setup hooks: substitutions.passthru -> passthru 2022-08-13 10:29:10 +02:00
Shamrock Lee
ba895a7da8 trivial-builders.nix: Add input argument passthru to makeSetupHook
One significant use case is adding `passthru.tests` to setup-hooks,
and help increase test coverage for mission-critical setup-hooks.

As `meta`, `passthru` doesn't go into the build script directly.
However, passing an empty set to `passthru` breaks nixpkgs-review
and OfBorg tests, so pass it only when specified.
2022-08-13 04:20:12 +08:00
Sergei Trofimovich
b3b672d5a1 setup-hooks/separate-debug-info.sh: don't inhibit strip hook
Before the change separate-debug-info.sh did the stripping itself.
This scheme has a few problems:
1. Stripping happens only on ELF files. *.a and *.o files are skipped.
   Derivations have to do it manually. Usually incorrectly
   as they don't run $RANLIB (true for `glibc` and `musl`).
2. Stripping happens on all paths. Ideally only `stripDebugList` paths
   should be considered.
3. Host strip is called on Target files.

This change offloads stripping logic to strip hook. This strips more
files for `glibc` and `musl`. Now we can remove most $STRIP calls
from individual derivations.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2022-08-07 12:49:37 +01:00
Sergei Trofimovich
c817efe660 gcc: extend stripping of .a libraries and .o objects
The initial intent was to strip .a and .o files, not .a.o files.
While at it expanded stripping for $lib output as well.

Without the change `libgcc.a` was not stripped and `.debug*` sections
made into final binaries. It's not a problem on it's own, but it's an
unintended side-effect. Noticed on `crystal_1_0` test failure where
`crystal` was not able to handle `dwarf-5`.

While at it allowed absolute file names to be passed to stripDebugList
and friends.
2022-08-03 22:28:47 +01:00
Bernardo Meurer
88c63ca65a
Merge pull request #182513 from trofi/strip-for-host-and-target
gcc: enable stripping for cross-compilers
2022-07-28 00:30:49 -07:00
Sergei Trofimovich
0507725061 setup-hooks/strip.sh: run RANLIB on static archives after stripping
'strip' does not normally preserve archive index in .a files.
This usually causes linking failures against static libs like:

    $ nix build --no-link -f. pkgsCross.mingw32.re2c
    > ...-i686-w64-mingw32-binutils-2.38/bin/i686-w64-mingw32-ld:
      /nix/store/...-i686-w64-mingw32-stage-final-gcc-13.0.0-lib/i686-w64-mingw32/lib/libstdc++.dll.a:
        error adding symbols: archive has no index; run ranlib to add one

We restore the index by running ranlib explicitly.
2022-07-25 11:06:58 +01:00
Sergei Trofimovich
0f45ce6e77 setup-hooks/strip.sh: add strip{All,Debug}ListTarget variables
This change mimics existing strip{All,Debug}List variables to
allow special stripping directories just for Target.

The primary use case in mind is gcc where package has to install
both host and target ELFs. They have to be stripped by their own
strip tools accordingly.

Co-authored-by: Rick van Schijndel <Mindavi@users.noreply.github.com>
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2022-07-25 11:06:45 +01:00
Sergei Trofimovich
17f413f293 setup-hooks/strip.sh: use STRIP_FOR_TARGET, not TARGET_STRIP
Since 1ac53985 "*-wrapper; Switch from `infixSalt` to `suffixSalt`"
(2020) 'TARGET_' prefix (and infix) is no more. '_FOR_TARGET' suffix
is the only used suffix for target-specific tools and flags.

Use that in stip instead of always-empty variable.
2022-07-22 18:38:02 +01:00
John Ericson
c38de9b3f2
Merge pull request #181525 from alyssais/rust-debug
separateDebugInfo: enable full Rust debug info
2022-07-21 13:09:19 -04:00
Artturin
7249b8a2f3 makePkgconfigItem: init new function to generate pc files
A function to generate pkg-config files for Nix packages that need to create them ad hoc,
like blas and lapack.

Inspiration taken from `makeDesktopItem`.
2022-07-20 06:22:39 +03:00
Michal Sojka
ae45c235fa multiple-output.sh: fix moveToOutput comment
The comment suggested that "{foo,bar}" is a supported pattern, which
is not true. "{foo,bar}" is only understood by brace expansion but the
code performs only globbing. We replace the comment with "[abc]",
which is a correct example of globbing.
2022-07-19 08:23:35 +02:00
Alyssa Ross
d7fff81159
separateDebugInfo: enable full Rust debug info
By default, Cargo will only enable line tables.  -g enables full debug
info.  The RUSTFLAGS environment variable is examined by Cargo,
similar to how the NIX_*FLAGS* variables are examined by our compiler
wrappers.
2022-07-14 21:44:12 +00:00
Thiago Kenji Okada
299538e834
Merge pull request #172769 from ncfavier/wrappers-append-args
makeWrapper,makeBinaryWrapper: implement `--append-flags`
2022-06-12 11:46:03 +01:00
Naïm Favier
eef2c762ce
makeBinaryWrapper: fix cross-compilation and add test
Fixes https://github.com/NixOS/nixpkgs/issues/175045
2022-05-31 13:55:41 +02:00
Naïm Favier
1c70b694fe
makeWrapper,makeBinaryWrapper: implement --append-flags 2022-05-30 23:07:58 +02:00
Naïm Favier
1cc6f08cdd
makeBinaryWrapper: fix codesign on aarch64-darwin
Reverts 8b79ef2c on aarch64-darwin, no-op on other platforms.
2022-05-22 21:46:06 +02:00
Naïm Favier
3e385d9a82
makeBinaryWrapper: add comment 2022-05-12 15:23:38 +02:00
Naïm Favier
88369997e1
makeBinaryWrapper: add extractCmd
A small shell script that can be used to extract a binary wrapper's
makeCWrapper call from its embedded docstring, without depending on
makeBinaryWrapper.
2022-05-12 01:44:02 +02:00
Naïm Favier
2ae69114a1
makeWrapper: implement --inherit-argv0
For symmetry/interoperability with makeBinaryWrapper. Implemented as
--argv0 '$0'
2022-05-10 22:07:57 +02:00
Naïm Favier
8b79ef2cb6
makeBinaryWrapper: remove cc from deps
Fixes https://github.com/NixOS/nixpkgs/issues/172249
2022-05-10 22:07:57 +02:00
Naïm Favier
81b9c712ce
wrapGAppsHook: rename argument to makeWrapper
...and pass it makeBinaryWrapper in all-packages.nix
2022-05-10 22:07:56 +02:00
Naïm Favier
62245943aa
makeWrapper,makeBinaryWrapper: introduce explicitly named functions
Because both versions might end up in a derivation's build inputs, it
might be useful to be able to explicitly select which function to use.
2022-05-10 22:07:56 +02:00