From bf17d6dacf2408436ffb5a2f8c87a31c88ec28e2 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 17 Dec 2016 23:51:18 -0800 Subject: [PATCH 01/14] top-level: Introduce `buildPackages` for resolving build-time deps [N.B., this package also applies to the commits that follow it in the same PR.] In most cases, buildPackages = pkgs so things work just as before. For cross compiling, however, buildPackages is resolved as the previous bootstrapping stage. This allows us to avoid the mkDerivation hacks cross compiling currently uses today. To avoid a massive refactor, callPackage will splice together both package sets. Again to avoid churn, it uses the old `nativeDrv` vs `crossDrv` to do so. So now, whether cross compiling or not, packages with get a `nativeDrv` and `crossDrv`---in the non-cross-compiling case they are simply the same derivation. This is good because it reduces the divergence between the cross and non-cross dataflow. See `pkgs/top-level/splice.nix` for a comment along the lines of the preceding paragraph, and the code that does this splicing. Also, `forceNativeDrv` is replaced with `forceNativePackages`. The latter resolves `pkgs` unless the host platform is different from the build platform, in which case it resolves to `buildPackages`. Note that the target platform is not important here---it will not prevent `forcedNativePackages` from resolving to `pkgs`. -------- Temporarily, we make preserve some dubious decisions in the name of preserving hashes: Most importantly, we don't distinguish between "host" and "target" in the autoconf sense. This leads to the proliferation of *Cross derivations currently used. What we ought to is resolve native deps of the cross "build packages" (build = host != target) package set against the "vanilla packages" (build = host = target) package set. Instead, "build packages" uses itself, with (informally) target != build in all cases. This is wrong because it violates the "sliding window" principle of bootstrapping stages that shifting the platform triple of one stage to the left coincides with the next stage's platform triple. Only because we don't explicitly distinguish between "host" and "target" does it appear that the "sliding window" principle is preserved--indeed it is over the reductionary "platform double" of just "build" and "host/target". Additionally, we build libc, libgcc, etc in the same stage as the compilers themselves, which is wrong because they are used at runtime, not build time. Fixing this is somewhat subtle, and the solution and problem will be better explained in the commit that does fix it. Commits after this will solve both these issues, at the expense of breaking cross hashes. Native hashes won't be broken, thankfully. -------- Did the temporary ugliness pan out? Of the packages that currently build in `release-cross.nix`, the only ones that have their hash changed are `*.gcc.crossDrv` and `bootstrapTools.*.coreutilsMinimal`. In both cases I think it doesn't matter. 1. GCC when doing a `build = host = target = foreign` build (maximally cross), still defines environment variables like `CPATH`[1] with packages. This seems assuredly wrong because whether gcc dynamically links those, or the programs built by gcc dynamically link those---I have no idea which case is reality---they should be foreign. Therefore, in all likelihood, I just made the gcc less broken. 2. Coreutils (ab)used the old cross-compiling infrastructure to depend on a native version of itself. When coreutils was overwritten to be built with fewer features, the native version it used would also be overwritten because the binding was tight. Now it uses the much looser `BuildPackages.coreutils` which is just fine as a richer build dep doesn't cause any problems and avoids a rebuild. So, in conclusion I'd say the conservatism payed off. Onward to actually raking the muck in the next PR! [1]: https://gcc.gnu.org/onlinedocs/gcc/Environment-Variables.html --- pkgs/os-specific/gnu/default.nix | 14 ++-- pkgs/stdenv/adapters.nix | 18 ++--- pkgs/stdenv/booter.nix | 18 +++-- pkgs/stdenv/cross/default.nix | 11 ++- pkgs/stdenv/generic/default.nix | 14 +++- .../linux/make-bootstrap-tools-cross.nix | 21 ++--- pkgs/top-level/all-packages.nix | 70 +++++++---------- pkgs/top-level/release-cross.nix | 14 +++- pkgs/top-level/release-lib.nix | 5 +- pkgs/top-level/splice.nix | 76 +++++++++++++++++++ pkgs/top-level/stage.nix | 7 ++ 11 files changed, 176 insertions(+), 92 deletions(-) create mode 100644 pkgs/top-level/splice.nix diff --git a/pkgs/os-specific/gnu/default.nix b/pkgs/os-specific/gnu/default.nix index 457b670319e0..c002447d64b1 100644 --- a/pkgs/os-specific/gnu/default.nix +++ b/pkgs/os-specific/gnu/default.nix @@ -3,7 +3,7 @@ args@{ fetchgit, stdenv, autoconf, automake, automake111x, libtool , texinfo, glibcCross, hurdPartedCross, libuuid, samba , gccCrossStageStatic, gccCrossStageFinal -, forceNativeDrv, forceSystem, newScope, platform, config, crossSystem +, forcedNativePackages, forceSystem, newScope, platform, config, crossSystem , overrides ? {} }: with args; @@ -12,7 +12,7 @@ let callPackage = newScope gnu; gnu = { - hurdCross = forceNativeDrv (callPackage ./hurd { + hurdCross = forcedNativePackages.callPackage ./hurd { inherit fetchgit stdenv autoconf libtool texinfo glibcCross hurdPartedCross; inherit (gnu) machHeaders mig; @@ -21,9 +21,9 @@ let headersOnly = false; cross = assert crossSystem != null; crossSystem; gccCross = gccCrossStageFinal; - }); + }; - hurdCrossIntermediate = forceNativeDrv (callPackage ./hurd { + hurdCrossIntermediate = forcedNativePackages.callPackage ./hurd { inherit fetchgit stdenv autoconf libtool texinfo glibcCross; inherit (gnu) machHeaders mig; hurdPartedCross = null; @@ -42,7 +42,7 @@ let # libshouldbeinlibc. buildTarget = "libihash libstore libshouldbeinlibc"; installTarget = "libihash-install libstore-install libshouldbeinlibc-install"; - }); + }; hurdHeaders = callPackage ./hurd { automake = automake111x; @@ -58,13 +58,13 @@ let hurd = null; }; - libpthreadCross = forceNativeDrv (callPackage ./libpthread { + libpthreadCross = forcedNativePackages.callPackage ./libpthread { inherit fetchgit stdenv autoconf automake libtool glibcCross; inherit (gnu) machHeaders hurdHeaders; hurd = gnu.hurdCrossIntermediate; gccCross = gccCrossStageStatic; cross = assert crossSystem != null; crossSystem; - }); + }; # In theory GNU Mach doesn't have to be cross-compiled. However, since it # has to be built for i586 (it doesn't work on x86_64), one needs a cross diff --git a/pkgs/stdenv/adapters.nix b/pkgs/stdenv/adapters.nix index 11f9a43c035e..daa180c644fa 100644 --- a/pkgs/stdenv/adapters.nix +++ b/pkgs/stdenv/adapters.nix @@ -66,12 +66,10 @@ rec { # In nixpkgs, sometimes 'null' gets in as a buildInputs element, # and we handle that through isAttrs. - getNativeDrv = drv: drv.nativeDrv or drv; - getCrossDrv = drv: drv.crossDrv or drv; - nativeBuildInputsDrvs = map getNativeDrv nativeBuildInputs; - buildInputsDrvs = map getCrossDrv buildInputs; - propagatedBuildInputsDrvs = map getCrossDrv propagatedBuildInputs; - propagatedNativeBuildInputsDrvs = map getNativeDrv propagatedNativeBuildInputs; + nativeBuildInputsDrvs = nativeBuildInputs; + buildInputsDrvs = buildInputs; + propagatedBuildInputsDrvs = propagatedBuildInputs; + propagatedNativeBuildInputsDrvs = propagatedNativeBuildInputs; # The base stdenv already knows that nativeBuildInputs and # buildInputs should be built with the usual gcc-wrapper @@ -88,10 +86,7 @@ rec { (drv: builtins.isAttrs drv && drv ? nativeDrv) buildInputs; nativeInputsFromBuildInputs = stdenv.lib.filter hostAsNativeDrv buildInputsNotNull; - # We should overwrite the input attributes in crossDrv, to overwrite - # the defaults for only-native builds in the base stdenv - crossDrv = if cross == null then nativeDrv else - stdenv.mkDerivation (args // { + in stdenv.mkDerivation (args // { name = name + "-" + cross.config; nativeBuildInputs = nativeBuildInputsDrvs ++ nativeInputsFromBuildInputs @@ -112,9 +107,6 @@ rec { crossConfig = cross.config; } // args.crossAttrs or {}); - in nativeDrv // { - inherit crossDrv nativeDrv; - }; } // { inherit cross gccCross binutilsCross; ccCross = gccCross; diff --git a/pkgs/stdenv/booter.nix b/pkgs/stdenv/booter.nix index 11ca8e1440e1..6e5d073e55a7 100644 --- a/pkgs/stdenv/booter.nix +++ b/pkgs/stdenv/booter.nix @@ -57,12 +57,18 @@ stageFuns: let # debugging purposes. folder = stageFun: finalSoFar: let args = stageFun finalSoFar; - stdenv = args.stdenv // { - # For debugging - __bootPackages = finalSoFar; + args' = args // { + stdenv = args.stdenv // { + # For debugging + __bootPackages = finalSoFar; + }; }; - args' = args // { inherit stdenv; }; - in - (if args.__raw or false then lib.id else allPackages) args'; + self = + if args.__raw or false + then args' + else allPackages ((builtins.removeAttrs args' ["selfBuild"]) // { + buildPackages = if args.selfBuild or true then self else finalSoFar; + }); + in self; in lib.lists.fold folder {} withAllowCustomOverrides diff --git a/pkgs/stdenv/cross/default.nix b/pkgs/stdenv/cross/default.nix index 16f41671b768..e684c14da4a5 100644 --- a/pkgs/stdenv/cross/default.nix +++ b/pkgs/stdenv/cross/default.nix @@ -12,13 +12,11 @@ let in bootStages ++ [ - # Build Packages. - # - # For now, this is just used to build the native stdenv. Eventually, it - # should be used to build compilers and other such tools targeting the cross - # platform. Then, `forceNativeDrv` can be removed. + # Build Packages (vanillaPackages: { inherit system platform crossSystem config overlays; + # Should be false, but we're trying to preserve hashes for now + selfBuild = true; # It's OK to change the built-time dependencies allowCustomOverrides = true; stdenv = vanillaPackages.stdenv // { @@ -28,9 +26,10 @@ in bootStages ++ [ }; }) - # Run packages + # Run Packages (buildPackages: { inherit system platform crossSystem config overlays; + selfBuild = false; stdenv = if crossSystem.useiOSCross or false then let inherit (buildPackages.darwin.ios-cross { diff --git a/pkgs/stdenv/generic/default.nix b/pkgs/stdenv/generic/default.nix index 32e0d8948188..269d7ef893a1 100644 --- a/pkgs/stdenv/generic/default.nix +++ b/pkgs/stdenv/generic/default.nix @@ -115,7 +115,19 @@ let , sandboxProfile ? "" , propagatedSandboxProfile ? "" , ... } @ attrs: - let + let # Rename argumemnts to avoid cycles + buildInputs__ = buildInputs; + nativeBuildInputs__ = nativeBuildInputs; + propagatedBuildInputs__ = propagatedBuildInputs; + propagatedNativeBuildInputs__ = propagatedNativeBuildInputs; + in let + getNativeDrv = drv: drv.nativeDrv or drv; + getCrossDrv = drv: drv.crossDrv or drv; + nativeBuildInputs = map getNativeDrv nativeBuildInputs__; + buildInputs = map getCrossDrv buildInputs__; + propagatedBuildInputs = map getCrossDrv propagatedBuildInputs__; + propagatedNativeBuildInputs = map getNativeDrv propagatedNativeBuildInputs__; + in let pos' = if pos != null then pos diff --git a/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix b/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix index 9f4a4517627e..a1b1f02d83d8 100644 --- a/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix +++ b/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix @@ -55,11 +55,12 @@ let if toolsArch == "armv6l" then raspberrypiCrossSystem else if toolsArch == "armv7l" then armv7l-hf-multiplatform-crossSystem else null; - pkgs = pkgsFun ({inherit system;} // selectedCrossSystem); + pkgsUnspliced = pkgsFun ({inherit system;} // selectedCrossSystem); + pkgs = pkgsUnspliced.splicedPackages; - inherit (pkgs) stdenv nukeReferences cpio binutilsCross; + inherit (pkgsUnspliced.buildPackages) stdenv nukeReferences cpio binutilsCross; - glibc = pkgs.libcCross; + glibc = pkgs.libcCross.nativeDrv; bash = pkgs.bash.crossDrv; findutils = pkgs.findutils.crossDrv; diffutils = pkgs.diffutils.crossDrv; @@ -71,7 +72,7 @@ let gnumake = pkgs.gnumake.crossDrv; patch = pkgs.patch.crossDrv; patchelf = pkgs.patchelf.crossDrv; - gcc = pkgs.gcc.cc.crossDrv; + gcc = pkgs.gcc.crossDrv.cc; gmpxx = pkgs.gmpxx.crossDrv; mpfr = pkgs.mpfr.crossDrv; zlib = pkgs.zlib.crossDrv; @@ -86,17 +87,17 @@ in rec { - coreutilsMinimal = (pkgs.coreutils.override (args: { + coreutilsMinimal = pkgs.coreutils.override (args: { # We want coreutils without ACL/attr support. aclSupport = false; attrSupport = false; # Our tooling currently can't handle scripts in bin/, only ELFs and symlinks. singleBinary = "symlinks"; - })).crossDrv; + }); - tarMinimal = (pkgs.gnutar.override { acl = null; }).crossDrv; + tarMinimal = pkgs.gnutar.override { acl = null; }; - busyboxMinimal = (pkgs.busybox.override { + busyboxMinimal = pkgs.busybox.override { useMusl = true; enableStatic = true; enableMinimal = true; @@ -109,13 +110,13 @@ rec { CONFIG_TAR y CONFIG_UNXZ y ''; - }).crossDrv; + }; build = stdenv.mkDerivation { name = "stdenv-bootstrap-tools-cross"; - crossConfig = stdenv.cross.config; + crossConfig = pkgsUnspliced.crossSystem.config; buildInputs = [nukeReferences cpio binutilsCross]; diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix index 8d2b4cf52014..bafe502da385 100644 --- a/pkgs/top-level/all-packages.nix +++ b/pkgs/top-level/all-packages.nix @@ -10,26 +10,12 @@ self: pkgs: with pkgs; -let - defaultScope = pkgs // pkgs.xorg; -in - { # Allow callPackage to fill in the pkgs argument inherit pkgs; - # We use `callPackage' to be able to omit function arguments that - # can be obtained from `pkgs' or `pkgs.xorg' (i.e. `defaultScope'). - # Use `newScope' for sets of packages in `pkgs' (see e.g. `gnome' - # below). - callPackage = newScope {}; - - callPackages = lib.callPackagesWith defaultScope; - - newScope = extra: lib.callPackageWith (defaultScope // extra); - # Override system. This is useful to build i686 packages on x86_64-linux. forceSystem = system: kernel: nixpkgsFun { inherit system; @@ -39,15 +25,9 @@ in # Used by wine, firefox with debugging version of Flash, ... pkgsi686Linux = forceSystem "i686-linux" "i386"; - callPackage_i686 = lib.callPackageWith (pkgsi686Linux // pkgsi686Linux.xorg); + callPackage_i686 = pkgsi686Linux.callPackage; - forceNativeDrv = drv: - # Even when cross compiling, some packages come from the stdenv's - # bootstrapping package set. Those packages are only built for the native - # platform. - if crossSystem != null && drv ? crossDrv - then drv // { crossDrv = drv.nativeDrv; } - else drv; + forcedNativePackages = if crossSystem == null then pkgs else buildPackages; # A stdenv capable of building 32-bit binaries. On x86_64-linux, # it uses GCC compiled with multilib support; on i686-linux, it's @@ -4771,41 +4751,45 @@ in gccApple = throw "gccApple is no longer supported"; - gccCrossStageStatic = let + gccCrossStageStatic = assert crossSystem != null; let libcCross1 = if stdenv.cross.libc == "msvcrt" then windows.mingw_w64_headers else if stdenv.cross.libc == "libSystem" then darwin.xcode else null; in wrapGCCCross { - gcc = forceNativeDrv (gcc.cc.override { + gcc = forcedNativePackages.gcc.cc.override { cross = crossSystem; crossStageStatic = true; langCC = false; libcCross = libcCross1; enableShared = false; - }); + # Why is this needed? + inherit (forcedNativePackages) binutilsCross; + }; libc = libcCross1; binutils = binutilsCross; cross = crossSystem; }; # Only needed for mingw builds - gccCrossMingw2 = wrapGCCCross { + gccCrossMingw2 = assert crossSystem != null; wrapGCCCross { gcc = gccCrossStageStatic.gcc; libc = windows.mingw_headers2; binutils = binutilsCross; cross = assert crossSystem != null; crossSystem; }; - gccCrossStageFinal = wrapGCCCross { - gcc = forceNativeDrv (gcc.cc.override { + gccCrossStageFinal = assert crossSystem != null; wrapGCCCross { + gcc = forcedNativePackages.gcc.cc.override { cross = crossSystem; crossStageStatic = false; # XXX: We have troubles cross-compiling libstdc++ on MinGW (see # ), so don't even try. langCC = crossSystem.config != "i686-pc-mingw32"; - }); + # Why is this needed? + inherit (forcedNativePackages) binutilsCross; + }; libc = libcCross; binutils = binutilsCross; cross = crossSystem; @@ -5499,12 +5483,12 @@ in wrapGCCCross = {gcc, libc, binutils, cross, shell ? "", name ? "gcc-cross-wrapper"}: - forceNativeDrv (callPackage ../build-support/gcc-cross-wrapper { + forcedNativePackages.callPackage ../build-support/gcc-cross-wrapper { nativeTools = false; nativeLibc = false; noLibc = (libc == null); inherit gcc binutils libc shell name cross; - }); + }; # prolog yap = callPackage ../development/compilers/yap { }; @@ -6084,12 +6068,12 @@ in gold = false; }); - binutilsCross = assert crossSystem != null; lowPrio (forceNativeDrv ( + binutilsCross = assert crossSystem != null; lowPrio ( if crossSystem.libc == "libSystem" then darwin.cctools_cross - else binutils.override { + else forcedNativePackages.binutils.override { noSysDirs = true; cross = crossSystem; - })); + }); bison2 = callPackage ../development/tools/parsing/bison/2.x.nix { }; bison3 = callPackage ../development/tools/parsing/bison/3.x.nix { }; @@ -6558,9 +6542,9 @@ in cross_renaming: we should make all programs use pkgconfig as nativeBuildInput after the renaming. */ - pkgconfig = forceNativeDrv (callPackage ../development/tools/misc/pkgconfig { + pkgconfig = forcedNativePackages.callPackage ../development/tools/misc/pkgconfig { fetchurl = fetchurlBoot; - }); + }; pkgconfigUpstream = lowPrio (pkgconfig.override { vanilla = true; }); postiats-utilities = callPackage ../development/tools/postiats-utilities {}; @@ -7336,10 +7320,10 @@ in withGd = true; }; - glibcCross = forceNativeDrv (glibc.override { + glibcCross = forcedNativePackages.glibc.override { gccCross = gccCrossStageStatic; linuxHeaders = linuxHeadersCross; - }); + }; # We can choose: libcCrossChooser = name: if name == "glibc" then glibcCross @@ -10910,7 +10894,7 @@ in cmdline = callPackage ../os-specific/darwin/command-line-tools {}; apple-source-releases = callPackage ../os-specific/darwin/apple-source-releases { }; in apple-source-releases // rec { - cctools_cross = callPackage (forceNativeDrv (callPackage ../os-specific/darwin/cctools/port.nix {}).cross) { + cctools_cross = callPackage (forcedNativePackages.callPackage ../os-specific/darwin/cctools/port.nix {}).cross { cross = assert crossSystem != null; crossSystem; inherit maloader; xctoolchain = xcode.toolchain; @@ -11148,13 +11132,13 @@ in linuxHeaders = linuxHeaders_4_4; - linuxHeaders24Cross = forceNativeDrv (callPackage ../os-specific/linux/kernel-headers/2.4.nix { + linuxHeaders24Cross = forcedNativePackages.callPackage ../os-specific/linux/kernel-headers/2.4.nix { cross = assert crossSystem != null; crossSystem; - }); + }; - linuxHeaders26Cross = forceNativeDrv (callPackage ../os-specific/linux/kernel-headers/4.4.nix { + linuxHeaders26Cross = forcedNativePackages.callPackage ../os-specific/linux/kernel-headers/4.4.nix { cross = assert crossSystem != null; crossSystem; - }); + }; linuxHeaders_3_18 = callPackage ../os-specific/linux/kernel-headers/3.18.nix { }; diff --git a/pkgs/top-level/release-cross.nix b/pkgs/top-level/release-cross.nix index f582bcf3b323..48f183162cf8 100644 --- a/pkgs/top-level/release-cross.nix +++ b/pkgs/top-level/release-cross.nix @@ -32,8 +32,10 @@ let in { - # These `nativeDrv`s should be identical to their vanilla ones --- cross - # compiling should not affect the native derivation. + # These derivations from a cross package set's `buildPackages` should be + # identical to their vanilla equivalents --- none of these package should + # observe the target platform which is the only difference between those + # package sets. ensureUnaffected = let # Absurd values are fine here, as we are not building anything. In fact, # there probably a good idea to try to be "more parametric" --- i.e. avoid @@ -47,8 +49,12 @@ in # good idea lest there be some irrelevant pass-through debug attrs that # cause false negatives. testEqualOne = path: system: let - f = attrs: builtins.toString (lib.getAttrFromPath path (allPackages attrs)); - in assert f { inherit system; } == f { inherit system crossSystem; }; true; + f = path: attrs: builtins.toString (lib.getAttrFromPath path (allPackages attrs)); + in assert + f path { inherit system; } + == + f (["buildPackages"] ++ path) { inherit system crossSystem; }; + true; testEqual = path: systems: forAllSupportedSystems systems (testEqualOne path); diff --git a/pkgs/top-level/release-lib.nix b/pkgs/top-level/release-lib.nix index 5fe87711c016..04a57ef8063b 100644 --- a/pkgs/top-level/release-lib.nix +++ b/pkgs/top-level/release-lib.nix @@ -76,8 +76,9 @@ rec { * parameter for allPackages, defining the target platform for cross builds, * and triggering the build of the host derivation (cross built - crossDrv). */ mapTestOnCross = crossSystem: mapAttrsRecursive - (path: systems: testOnCross crossSystem systems - (pkgs: addMetaAttrs { maintainers = crossMaintainers; } (getAttrFromPath path pkgs))); + (path: systems: testOnCross crossSystem systems (pkgs: addMetaAttrs + { maintainers = crossMaintainers; } + (getAttrFromPath path pkgs.splicedPackages))); /* Recursively map a (nested) set of derivations to an isomorphic diff --git a/pkgs/top-level/splice.nix b/pkgs/top-level/splice.nix new file mode 100644 index 000000000000..f57f42020c27 --- /dev/null +++ b/pkgs/top-level/splice.nix @@ -0,0 +1,76 @@ +# The `splicedPackages' package set, and its use by `callPackage` +# +# The `buildPackages` pkg set is a new concept, and the vast majority package +# expression (the other *.nix files) are not designed with it in mind. This +# presents us with a problem with how to get the right version (build-time vs +# run-time) of a package to a consumer that isn't used to thinking so cleverly. +# +# The solution is to splice the package sets together as we do below, so every +# `callPackage`d expression in fact gets both versions. Each# derivation (and +# each derivation's outputs) consists of the run-time version, augmented with a +# `nativeDrv` field for the build-time version, and `crossDrv` field for the +# run-time version. +# +# We could have used any names we want for the disambiguated versions, but +# `crossDrv` and `nativeDrv` were somewhat similarly used for the old +# cross-compiling infrastructure. The names are mostly invisible as +# `mkDerivation` knows how to pull out the right ones for `buildDepends` and +# friends, but a few packages use them directly, so it seemed efficient (to +# @Ericson2314) to reuse those names, at least initially, to minimize breakage. +lib: pkgs: + +let + defaultBuildScope = pkgs.buildPackages // pkgs.buildPackages.xorg; + # TODO(@Ericson2314): we shouldn't preclude run-time fetching by removing + # these attributes. We should have a more general solution for selecting + # whether `nativeDrv` or `crossDrv` is the default in `defaultScope`. + pkgsWithoutFetchers = lib.filterAttrs (n: _: !lib.hasPrefix "fetch" n) pkgs; + defaultRunScope = pkgsWithoutFetchers // pkgs.xorg; + + splicer = buildPkgs: runPkgs: let + mash = buildPkgs // runPkgs; + merge = name: { + inherit name; + value = let + defaultValue = mash.${name}; + buildValue = buildPkgs.${name} or {}; + runValue = runPkgs.${name} or {}; + augmentedValue = defaultValue + // (lib.optionalAttrs (buildPkgs ? ${name}) { nativeDrv = buildValue; }) + // (lib.optionalAttrs (runPkgs ? ${name}) { crossDrv = runValue; }); + # Get the set of outputs of a derivation + getOutputs = value: + lib.genAttrs (value.outputs or []) (output: value.${output}); + in + # Certain *Cross derivations will fail assertions, but we need their + # nativeDrv. We are assuming anything that fails to evaluate is an + # attrset (including derivation) and thus can be unioned. + if !(builtins.tryEval defaultValue).success then augmentedValue + # The derivation along with its outputs, which we recur + # on to splice them together. + else if lib.isDerivation defaultValue then augmentedValue + // splicer (getOutputs buildValue) (getOutputs runValue) + # Just recur on plain attrsets + else if lib.isAttrs defaultValue then splicer buildValue runValue + # Don't be fancy about non-derivations. But we could have used used + # `__functor__` for functions instead. + else defaultValue; + }; + in lib.listToAttrs (map merge (lib.attrNames mash)); + + splicedPackages = splicer defaultBuildScope defaultRunScope; + +in + +{ + splicedPackages = splicedPackages // { recurseForDerivations = false; }; + + # We use `callPackage' to be able to omit function arguments that can be + # obtained `pkgs` or `buildPackages` and their `xorg` package sets. Use + # `newScope' for sets of packages in `pkgs' (see e.g. `gnome' below). + callPackage = pkgs.newScope {}; + + callPackages = lib.callPackagesWith splicedPackages; + + newScope = extra: lib.callPackageWith (splicedPackages // extra); +} diff --git a/pkgs/top-level/stage.nix b/pkgs/top-level/stage.nix index cbf65870eb7e..c100e21473cd 100644 --- a/pkgs/top-level/stage.nix +++ b/pkgs/top-level/stage.nix @@ -12,6 +12,9 @@ { # The system (e.g., `i686-linux') for which to build the packages. system +, # the package set used at build-time + buildPackages + , # The standard environment to use for building packages. stdenv @@ -51,10 +54,13 @@ let stdenvBootstappingAndPlatforms = self: super: { stdenv = stdenv // { inherit platform; }; + buildPackages = buildPackages // { recurseForDerivations = false; }; inherit system platform crossSystem; }; + splice = self: super: import ./splice.nix lib self; + allPackages = self: super: let res = import ./all-packages.nix { inherit lib nixpkgsFun noSysDirs config; } @@ -85,6 +91,7 @@ let stdenvBootstappingAndPlatforms stdenvAdapters trivialBuilders + splice allPackages aliases stdenvOverrides From eee7cafaf7961b762cb74edd1670b42e5a7547a9 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 15 Jan 2017 13:28:35 -0500 Subject: [PATCH 02/14] coreutils: Use `buildPackages` instead of `self` hack for native version --- pkgs/tools/misc/coreutils/default.nix | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/pkgs/tools/misc/coreutils/default.nix b/pkgs/tools/misc/coreutils/default.nix index 9e66c6ba9181..b65c20e7808d 100644 --- a/pkgs/tools/misc/coreutils/default.nix +++ b/pkgs/tools/misc/coreutils/default.nix @@ -1,4 +1,4 @@ -{ lib, stdenv, fetchurl, perl, xz, gmp ? null +{ lib, stdenv, buildPackages, fetchurl, perl, xz, gmp ? null , aclSupport ? false, acl ? null , attrSupport ? false, attr ? null , selinuxSupport? false, libselinux ? null, libsepol ? null @@ -12,8 +12,7 @@ assert selinuxSupport -> libselinux != null && libsepol != null; with lib; -let - self = stdenv.mkDerivation rec { +stdenv.mkDerivation rec { name = "coreutils-8.26"; src = fetchurl { @@ -63,12 +62,12 @@ let # Works around a bug with 8.26: # Makefile:3440: *** Recursive variable 'INSTALL' references itself (eventually). Stop. preInstall = '' - sed -i Makefile -e 's|^INSTALL =.*|INSTALL = ${self}/bin/install -c|' + sed -i Makefile -e 's|^INSTALL =.*|INSTALL = ${buildPackages.coreutils}/bin/install -c|' ''; postInstall = '' rm $out/share/man/man1/* - cp ${self}/share/man/man1/* $out/share/man/man1 + cp ${buildPackages.coreutils}/share/man/man1/* $out/share/man/man1 ''; # Needed for fstatfs() @@ -110,6 +109,4 @@ let maintainers = [ maintainers.eelco ]; }; - }; -in - self + } From 94df8e7e4d9024abe9883f310dd6c2c7929f4ce0 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 14 Jan 2017 14:54:53 -0500 Subject: [PATCH 03/14] openssl: Output-santizing hack properly uses native perl again --- pkgs/development/libraries/openssl/default.nix | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pkgs/development/libraries/openssl/default.nix b/pkgs/development/libraries/openssl/default.nix index cd4a696b1d98..7cce57a753fa 100644 --- a/pkgs/development/libraries/openssl/default.nix +++ b/pkgs/development/libraries/openssl/default.nix @@ -1,4 +1,4 @@ -{ stdenv, fetchurl, perl +{ stdenv, fetchurl, buildPackages, perl , withCryptodev ? false, cryptodevHeaders , enableSSL2 ? false }: @@ -76,7 +76,7 @@ let postFixup = '' # Check to make sure the main output doesn't depend on perl - if grep -r '${perl}' $out; then + if grep -r '${buildPackages.perl}' $out; then echo "Found an erroneous dependency on perl ^^^" >&2 exit 1 fi From d27403b17b9e7baa902f90be25391d65c22fe19d Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 14 Jan 2017 16:22:50 -0500 Subject: [PATCH 04/14] libiconv: TEMP force building library improperly for target plat like libc --- pkgs/top-level/all-packages.nix | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix index bafe502da385..47c565bd2ee7 100644 --- a/pkgs/top-level/all-packages.nix +++ b/pkgs/top-level/all-packages.nix @@ -8241,9 +8241,9 @@ with pkgs; # glibc provides libiconv so systems with glibc don't need to build libiconv # separately, but we also provide libiconvReal, which will always be a # standalone libiconv, just in case you want it - libiconv = if crossSystem != null then - (if crossSystem.libc == "glibc" then libcCross - else if crossSystem.libc == "libSystem" then darwin.libiconv + libiconv = if stdenv ? cross then + (if stdenv.cross.libc == "glibc" then libcCross + else if stdenv.cross.libc == "libSystem" then darwin.libiconv else libiconvReal) else if stdenv.isGlibc then glibcIconv stdenv.cc.libc else if stdenv.isDarwin then darwin.libiconv From 5b88f09ec4d6b3ac953cd9d252ebfd6663205c57 Mon Sep 17 00:00:00 2001 From: David Grayson Date: Fri, 23 Dec 2016 21:26:54 -0800 Subject: [PATCH 05/14] coreutils: Fix indentation after removing `self` in `buildPackages` commit --- pkgs/tools/misc/coreutils/default.nix | 162 +++++++++++++------------- 1 file changed, 81 insertions(+), 81 deletions(-) diff --git a/pkgs/tools/misc/coreutils/default.nix b/pkgs/tools/misc/coreutils/default.nix index b65c20e7808d..2c435881f8c5 100644 --- a/pkgs/tools/misc/coreutils/default.nix +++ b/pkgs/tools/misc/coreutils/default.nix @@ -13,100 +13,100 @@ assert selinuxSupport -> libselinux != null && libsepol != null; with lib; stdenv.mkDerivation rec { - name = "coreutils-8.26"; + name = "coreutils-8.26"; - src = fetchurl { - url = "mirror://gnu/coreutils/${name}.tar.xz"; - sha256 = "13lspazc7xkviy93qz7ks9jv4sldvgmwpq36ghrbrqpq93br8phm"; - }; + src = fetchurl { + url = "mirror://gnu/coreutils/${name}.tar.xz"; + sha256 = "13lspazc7xkviy93qz7ks9jv4sldvgmwpq36ghrbrqpq93br8phm"; + }; - # FIXME needs gcc 4.9 in bootstrap tools - hardeningDisable = [ "stackprotector" ]; + # FIXME needs gcc 4.9 in bootstrap tools + hardeningDisable = [ "stackprotector" ]; - patches = optional stdenv.isCygwin ./coreutils-8.23-4.cygwin.patch; + patches = optional stdenv.isCygwin ./coreutils-8.23-4.cygwin.patch; - # The test tends to fail on btrfs and maybe other unusual filesystems. - postPatch = optionalString (!stdenv.isDarwin) '' - sed '2i echo Skipping dd sparse test && exit 0' -i ./tests/dd/sparse.sh - sed '2i echo Skipping cp sparse test && exit 0' -i ./tests/cp/sparse.sh - sed '2i echo Skipping rm deep-2 test && exit 0' -i ./tests/rm/deep-2.sh - sed '2i echo Skipping du long-from-unreadable test && exit 0' -i ./tests/du/long-from-unreadable.sh + # The test tends to fail on btrfs and maybe other unusual filesystems. + postPatch = optionalString (!stdenv.isDarwin) '' + sed '2i echo Skipping dd sparse test && exit 0' -i ./tests/dd/sparse.sh + sed '2i echo Skipping cp sparse test && exit 0' -i ./tests/cp/sparse.sh + sed '2i echo Skipping rm deep-2 test && exit 0' -i ./tests/rm/deep-2.sh + sed '2i echo Skipping du long-from-unreadable test && exit 0' -i ./tests/du/long-from-unreadable.sh + ''; + + outputs = [ "out" "info" ]; + + nativeBuildInputs = [ perl xz.bin ]; + configureFlags = + optional (singleBinary != false) + ("--enable-single-binary" + optionalString (isString singleBinary) "=${singleBinary}") + ++ optional stdenv.isSunOS "ac_cv_func_inotify_init=no" + ++ optional withPrefix "--program-prefix=g"; + + buildInputs = [ gmp ] + ++ optional aclSupport acl + ++ optional attrSupport attr + ++ optionals stdenv.isCygwin [ autoconf automake114x texinfo ] # due to patch + ++ optionals selinuxSupport [ libselinux libsepol ]; + + crossAttrs = { + buildInputs = [ gmp.crossDrv ] + ++ optional aclSupport acl.crossDrv + ++ optional attrSupport attr.crossDrv + ++ optionals selinuxSupport [ libselinux.crossDrv libsepol.crossDrv ] + ++ optional (stdenv.ccCross.libc ? libiconv) + stdenv.ccCross.libc.libiconv.crossDrv; + + # Prevents attempts of running 'help2man' on cross-built binaries. + PERL = "missing"; + + # Works around a bug with 8.26: + # Makefile:3440: *** Recursive variable 'INSTALL' references itself (eventually). Stop. + preInstall = '' + sed -i Makefile -e 's|^INSTALL =.*|INSTALL = ${buildPackages.coreutils}/bin/install -c|' ''; - outputs = [ "out" "info" ]; + postInstall = '' + rm $out/share/man/man1/* + cp ${buildPackages.coreutils}/share/man/man1/* $out/share/man/man1 + ''; - nativeBuildInputs = [ perl xz.bin ]; - configureFlags = - optional (singleBinary != false) - ("--enable-single-binary" + optionalString (isString singleBinary) "=${singleBinary}") - ++ optional stdenv.isSunOS "ac_cv_func_inotify_init=no" - ++ optional withPrefix "--program-prefix=g"; + # Needed for fstatfs() + # I don't know why it is not properly detected cross building with glibc. + configureFlags = [ "fu_cv_sys_stat_statfs2_bsize=yes" ]; + doCheck = false; + }; - buildInputs = [ gmp ] - ++ optional aclSupport acl - ++ optional attrSupport attr - ++ optionals stdenv.isCygwin [ autoconf automake114x texinfo ] # due to patch - ++ optionals selinuxSupport [ libselinux libsepol ]; + # The tests are known broken on Cygwin + # (http://thread.gmane.org/gmane.comp.gnu.core-utils.bugs/19025), + # Darwin (http://thread.gmane.org/gmane.comp.gnu.core-utils.bugs/19351), + # and {Open,Free}BSD. + # With non-standard storeDir: https://github.com/NixOS/nix/issues/512 + doCheck = stdenv ? glibc && builtins.storeDir == "/nix/store"; - crossAttrs = { - buildInputs = [ gmp.crossDrv ] - ++ optional aclSupport acl.crossDrv - ++ optional attrSupport attr.crossDrv - ++ optionals selinuxSupport [ libselinux.crossDrv libsepol.crossDrv ] - ++ optional (stdenv.ccCross.libc ? libiconv) - stdenv.ccCross.libc.libiconv.crossDrv; + # Saw random failures like ‘help2man: can't get '--help' info from + # man/sha512sum.td/sha512sum’. + enableParallelBuilding = false; - # Prevents attempts of running 'help2man' on cross-built binaries. - PERL = "missing"; + NIX_LDFLAGS = optionalString selinuxSupport "-lsepol"; + FORCE_UNSAFE_CONFIGURE = optionalString stdenv.isSunOS "1"; - # Works around a bug with 8.26: - # Makefile:3440: *** Recursive variable 'INSTALL' references itself (eventually). Stop. - preInstall = '' - sed -i Makefile -e 's|^INSTALL =.*|INSTALL = ${buildPackages.coreutils}/bin/install -c|' - ''; + makeFlags = optionalString stdenv.isDarwin "CFLAGS=-D_FORTIFY_SOURCE=0"; - postInstall = '' - rm $out/share/man/man1/* - cp ${buildPackages.coreutils}/share/man/man1/* $out/share/man/man1 - ''; + meta = { + homepage = http://www.gnu.org/software/coreutils/; + description = "The basic file, shell and text manipulation utilities of the GNU operating system"; - # Needed for fstatfs() - # I don't know why it is not properly detected cross building with glibc. - configureFlags = [ "fu_cv_sys_stat_statfs2_bsize=yes" ]; - doCheck = false; - }; + longDescription = '' + The GNU Core Utilities are the basic file, shell and text + manipulation utilities of the GNU operating system. These are + the core utilities which are expected to exist on every + operating system. + ''; - # The tests are known broken on Cygwin - # (http://thread.gmane.org/gmane.comp.gnu.core-utils.bugs/19025), - # Darwin (http://thread.gmane.org/gmane.comp.gnu.core-utils.bugs/19351), - # and {Open,Free}BSD. - # With non-standard storeDir: https://github.com/NixOS/nix/issues/512 - doCheck = stdenv ? glibc && builtins.storeDir == "/nix/store"; + license = licenses.gpl3Plus; - # Saw random failures like ‘help2man: can't get '--help' info from - # man/sha512sum.td/sha512sum’. - enableParallelBuilding = false; + platforms = platforms.all; - NIX_LDFLAGS = optionalString selinuxSupport "-lsepol"; - FORCE_UNSAFE_CONFIGURE = optionalString stdenv.isSunOS "1"; - - makeFlags = optionalString stdenv.isDarwin "CFLAGS=-D_FORTIFY_SOURCE=0"; - - meta = { - homepage = http://www.gnu.org/software/coreutils/; - description = "The basic file, shell and text manipulation utilities of the GNU operating system"; - - longDescription = '' - The GNU Core Utilities are the basic file, shell and text - manipulation utilities of the GNU operating system. These are - the core utilities which are expected to exist on every - operating system. - ''; - - license = licenses.gpl3Plus; - - platforms = platforms.all; - - maintainers = [ maintainers.eelco ]; - }; - } + maintainers = [ maintainers.eelco ]; + }; +} From 92edcb7ebbf5b4b324288ec62bebbc58a3f96ef6 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sat, 24 Dec 2016 10:55:11 -0800 Subject: [PATCH 06/14] top-level: Lay the groundwork for `{build,host,target}Platform` The long term goal is a big replace: { inherit system platform; } => buildPlatform crossSystem => hostPlatform stdenv.cross => targetPlatform And additionally making sure each is defined even when not cross compiling. This commit refactors the bootstrapping code along that vision, but leaves the old identifiers with their null semantics in place so packages can be modernized incrementally. --- pkgs/stdenv/adapters.nix | 2 +- pkgs/stdenv/cross/default.nix | 16 ++++--- pkgs/stdenv/custom/default.nix | 9 ++-- pkgs/stdenv/darwin/default.nix | 17 +++++-- pkgs/stdenv/default.nix | 4 +- pkgs/stdenv/freebsd/default.nix | 8 +++- pkgs/stdenv/linux/default.nix | 30 ++++++++----- pkgs/stdenv/native/default.nix | 13 ++++-- pkgs/stdenv/nix/default.nix | 4 +- pkgs/top-level/default.nix | 3 +- pkgs/top-level/stage.nix | 80 ++++++++++++++++++++++++++------- 11 files changed, 134 insertions(+), 52 deletions(-) diff --git a/pkgs/stdenv/adapters.nix b/pkgs/stdenv/adapters.nix index daa180c644fa..5d66db266c75 100644 --- a/pkgs/stdenv/adapters.nix +++ b/pkgs/stdenv/adapters.nix @@ -108,7 +108,7 @@ rec { crossConfig = cross.config; } // args.crossAttrs or {}); } // { - inherit cross gccCross binutilsCross; + inherit gccCross binutilsCross; ccCross = gccCross; }; diff --git a/pkgs/stdenv/cross/default.nix b/pkgs/stdenv/cross/default.nix index e684c14da4a5..37f403acee9e 100644 --- a/pkgs/stdenv/cross/default.nix +++ b/pkgs/stdenv/cross/default.nix @@ -1,10 +1,10 @@ { lib -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays }: let bootStages = import ../. { - inherit lib system platform overlays; + inherit lib localSystem overlays; crossSystem = null; # Ignore custom stdenvs when cross compiling for compatability config = builtins.removeAttrs config [ "replaceStdenv" ]; @@ -14,21 +14,25 @@ in bootStages ++ [ # Build Packages (vanillaPackages: { - inherit system platform crossSystem config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = crossSystem; + inherit config overlays; # Should be false, but we're trying to preserve hashes for now selfBuild = true; # It's OK to change the built-time dependencies allowCustomOverrides = true; stdenv = vanillaPackages.stdenv // { - # Needed elsewhere as a hacky way to pass the target - cross = crossSystem; overrides = _: _: {}; }; }) # Run Packages (buildPackages: { - inherit system platform crossSystem config overlays; + buildPlatform = localSystem; + hostPlatform = crossSystem; + targetPlatform = crossSystem; + inherit config overlays; selfBuild = false; stdenv = if crossSystem.useiOSCross or false then let diff --git a/pkgs/stdenv/custom/default.nix b/pkgs/stdenv/custom/default.nix index d7e9bf53bed1..d5dc977b37a7 100644 --- a/pkgs/stdenv/custom/default.nix +++ b/pkgs/stdenv/custom/default.nix @@ -1,12 +1,12 @@ { lib -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays }: assert crossSystem == null; let bootStages = import ../. { - inherit lib system platform crossSystem overlays; + inherit lib localSystem crossSystem overlays; # Remove config.replaceStdenv to ensure termination. config = builtins.removeAttrs config [ "replaceStdenv" ]; }; @@ -15,7 +15,10 @@ in bootStages ++ [ # Additional stage, built using custom stdenv (vanillaPackages: { - inherit system platform crossSystem config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = config.replaceStdenv { pkgs = vanillaPackages; }; }) diff --git a/pkgs/stdenv/darwin/default.nix b/pkgs/stdenv/darwin/default.nix index e3a87ea078fc..e647c81890e1 100644 --- a/pkgs/stdenv/darwin/default.nix +++ b/pkgs/stdenv/darwin/default.nix @@ -1,11 +1,12 @@ { lib -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays # Allow passing in bootstrap files directly so we can test the stdenv bootstrap process when changing the bootstrap tools , bootstrapFiles ? let fetch = { file, sha256, executable ? true }: import { url = "http://tarballs.nixos.org/stdenv-darwin/x86_64/33f59c9d11b8d5014dfd18cc11a425f6393c884a/${file}"; - inherit sha256 system executable; + inherit (localSystem) system; + inherit sha256 executable; }; in { sh = fetch { file = "sh"; sha256 = "1rx4kg6358xdj05z0m139a0zn4f4zfmq4n4vimlmnwyfiyn4x7wk"; }; bzip2 = fetch { file = "bzip2"; sha256 = "104qnhzk79vkbp2yi0kci6lszgfppvrwk3rgxhry842ly1xz2r7l"; }; @@ -18,6 +19,8 @@ assert crossSystem == null; let + inherit (localSystem) system platform; + libSystemProfile = '' (import "${./standard-sandbox.sb}") ''; @@ -98,7 +101,10 @@ in rec { }; in { - inherit system platform crossSystem config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = thisStdenv; }; @@ -316,7 +322,10 @@ in rec { stage3 stage4 (prevStage: { - inherit system crossSystem platform config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = stdenvDarwin prevStage; }) ]; diff --git a/pkgs/stdenv/default.nix b/pkgs/stdenv/default.nix index f60ffec4b564..78dbde13b89d 100644 --- a/pkgs/stdenv/default.nix +++ b/pkgs/stdenv/default.nix @@ -7,7 +7,7 @@ { # Args just for stdenvs' usage lib # Args to pass on to the pkgset builder, too -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays } @ args: let @@ -51,4 +51,4 @@ in "i686-cygwin" = stagesNative; "x86_64-cygwin" = stagesNative; "x86_64-freebsd" = stagesFreeBSD; - }.${system} or stagesNative + }.${localSystem.system} or stagesNative diff --git a/pkgs/stdenv/freebsd/default.nix b/pkgs/stdenv/freebsd/default.nix index 2cb059deb34b..b926c6bdd901 100644 --- a/pkgs/stdenv/freebsd/default.nix +++ b/pkgs/stdenv/freebsd/default.nix @@ -1,8 +1,9 @@ { lib -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays }: assert crossSystem == null; +let inherit (localSystem) system; in [ @@ -58,7 +59,10 @@ assert crossSystem == null; }) (prevStage: { - inherit system crossSystem platform config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = import ../generic { name = "stdenv-freebsd-boot-3"; inherit system config; diff --git a/pkgs/stdenv/linux/default.nix b/pkgs/stdenv/linux/default.nix index 12da007f2a76..611628b35aba 100644 --- a/pkgs/stdenv/linux/default.nix +++ b/pkgs/stdenv/linux/default.nix @@ -4,21 +4,23 @@ # compiler and linker that do not search in default locations, # ensuring purity of components produced by it. { lib -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays -, bootstrapFiles ? - if system == "i686-linux" then import ./bootstrap-files/i686.nix - else if system == "x86_64-linux" then import ./bootstrap-files/x86_64.nix - else if system == "armv5tel-linux" then import ./bootstrap-files/armv5tel.nix - else if system == "armv6l-linux" then import ./bootstrap-files/armv6l.nix - else if system == "armv7l-linux" then import ./bootstrap-files/armv7l.nix - else if system == "mips64el-linux" then import ./bootstrap-files/loongson2f.nix - else abort "unsupported platform for the pure Linux stdenv" +, bootstrapFiles ? { # switch + "i686-linux" = import ./bootstrap-files/i686.nix; + "x86_64-linux" = import ./bootstrap-files/x86_64.nix; + "armv5tel-linux" = import ./bootstrap-files/armv5tel.nix; + "armv6l-linux" = import ./bootstrap-files/armv6l.nix; + "armv7l-linux" = import ./bootstrap-files/armv7l.nix; + "mips64el-linux" = import ./bootstrap-files/loongson2f.nix; + }.${localSystem.system} + or (abort "unsupported platform for the pure Linux stdenv") }: assert crossSystem == null; let + inherit (localSystem) system platform; commonPreHook = '' @@ -91,7 +93,10 @@ let }; in { - inherit system platform crossSystem config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = thisStdenv; }; @@ -246,7 +251,10 @@ in # dependency (`nix-store -qR') on bootstrapTools or the first # binutils built. (prevStage: { - inherit system crossSystem platform config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = import ../generic rec { inherit system config; diff --git a/pkgs/stdenv/native/default.nix b/pkgs/stdenv/native/default.nix index 4028638009e1..f5c0976bf931 100644 --- a/pkgs/stdenv/native/default.nix +++ b/pkgs/stdenv/native/default.nix @@ -1,10 +1,11 @@ { lib -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays }: assert crossSystem == null; let + inherit (localSystem) system platform; shell = if system == "i686-freebsd" || system == "x86_64-freebsd" then "/usr/local/bin/bash" @@ -134,7 +135,10 @@ in # First build a stdenv based only on tools outside the store. (prevStage: { - inherit system crossSystem platform config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = makeStdenv { inherit (prevStage) cc fetchurl; } // { inherit (prevStage) fetchurl; }; @@ -143,7 +147,10 @@ in # Using that, build a stdenv that adds the ‘xz’ command (which most systems # don't have, so we mustn't rely on the native environment providing it). (prevStage: { - inherit system crossSystem platform config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = makeStdenv { inherit (prevStage.stdenv) cc fetchurl; extraPath = [ prevStage.xz ]; diff --git a/pkgs/stdenv/nix/default.nix b/pkgs/stdenv/nix/default.nix index a5f0a18464c1..9aece3ce829d 100644 --- a/pkgs/stdenv/nix/default.nix +++ b/pkgs/stdenv/nix/default.nix @@ -9,9 +9,9 @@ assert crossSystem == null; bootStages ++ [ (prevStage: let inherit (prevStage) stdenv; - inherit (stdenv) system platform; in { - inherit system platform crossSystem config; + inherit (prevStage) buildPlatform hostPlatform targetPlatform; + inherit config overlays; stdenv = import ../generic rec { inherit config; diff --git a/pkgs/top-level/default.nix b/pkgs/top-level/default.nix index a146dad63bc8..b7a4273acff0 100644 --- a/pkgs/top-level/default.nix +++ b/pkgs/top-level/default.nix @@ -83,7 +83,8 @@ in let boot = import ../stdenv/booter.nix { inherit lib allPackages; }; stages = stdenvStages { - inherit lib system platform crossSystem config overlays; + localSystem = { inherit system platform; }; + inherit lib crossSystem config overlays; }; pkgs = boot stages; diff --git a/pkgs/top-level/stage.nix b/pkgs/top-level/stage.nix index c100e21473cd..780be566436c 100644 --- a/pkgs/top-level/stage.nix +++ b/pkgs/top-level/stage.nix @@ -9,10 +9,43 @@ import `pkgs/default.nix` or `default.nix`. */ -{ # The system (e.g., `i686-linux') for which to build the packages. - system +{ ## Misc parameters kept the same for all stages + ## -, # the package set used at build-time + # Utility functions, could just import but passing in for efficiency + lib + +, # Use to reevaluate Nixpkgs; a dirty hack that should be removed + nixpkgsFun + + ## Platform parameters + ## + ## The "build" "host" "target" terminology below comes from GNU Autotools. See + ## its documentation for more information on what those words mean. Note that + ## each should always be defined, even when not cross compiling. + ## + ## For purposes of bootstrapping, think of each stage as a "sliding window" + ## over a list of platforms. Specifically, the host platform of the previous + ## stage becomes the build platform of the current one, and likewise the + ## target platform of the previous stage becomes the host platform of the + ## current one. + ## + +, # The platform on which packages are built. Consists of `system`, a + # string (e.g.,`i686-linux') identifying the most import attributes of the + # build platform, and `platform` a set of other details. + buildPlatform + +, # The platform on which packages run. + hostPlatform + +, # The platform which build tools (especially compilers) build for in this stage, + targetPlatform + + ## Other parameters + ## + +, # The package set used at build-time buildPackages , # The standard environment to use for building packages. @@ -24,21 +57,19 @@ allowCustomOverrides , # Non-GNU/Linux OSes are currently "impure" platforms, with their libc - # outside of the store. Thus, GCC, GFortran, & co. must always look for - # files in standard system directories (/usr/include, etc.) - noSysDirs ? (system != "x86_64-freebsd" && system != "i686-freebsd" - && system != "x86_64-solaris" - && system != "x86_64-kfreebsd-gnu") + # outside of the store. Thus, GCC, GFortran, & co. must always look for files + # in standard system directories (/usr/include, etc.) + noSysDirs ? buildPlatform.system != "x86_64-freebsd" + && buildPlatform.system != "i686-freebsd" + && buildPlatform.system != "x86_64-solaris" + && buildPlatform.system != "x86_64-kfreebsd-gnu" , # The configuration attribute set config -, overlays # List of overlays to use in the fix-point. - -, crossSystem -, platform -, lib -, nixpkgsFun +, # A list of overlays (Additional `self: super: { .. }` customization + # functions) to be fixed together in the produced package set + overlays }: let @@ -53,10 +84,24 @@ let }; stdenvBootstappingAndPlatforms = self: super: { - stdenv = stdenv // { inherit platform; }; buildPackages = buildPackages // { recurseForDerivations = false; }; - inherit - system platform crossSystem; + inherit stdenv + buildPlatform hostPlatform targetPlatform; + }; + + # The old identifiers for cross-compiling. These should eventually be removed, + # and the packages that rely on them refactored accordingly. + platformCompat = self: super: let + # TODO(@Ericson2314) this causes infinite recursion + #inherit (self) buildPlatform hostPlatform targetPlatform; + in { + stdenv = super.stdenv // { + inherit (buildPlatform) platform; + } // lib.optionalAttrs (targetPlatform != buildPlatform) { + cross = targetPlatform; + }; + inherit (buildPlatform) system platform; + crossSystem = if targetPlatform != buildPlatform then targetPlatform else null; }; splice = self: super: import ./splice.nix lib self; @@ -89,6 +134,7 @@ let # The complete chain of package set builders, applied from top to bottom toFix = lib.foldl' (lib.flip lib.extends) (self: {}) ([ stdenvBootstappingAndPlatforms + platformCompat stdenvAdapters trivialBuilders splice From 09401d44eb402a368b1eb6908a68e5b5a3e64e05 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 15 Jan 2017 13:40:32 -0500 Subject: [PATCH 07/14] cc-wrapper: Remove unneeded crossAttrs --- pkgs/build-support/cc-wrapper/default.nix | 3 --- 1 file changed, 3 deletions(-) diff --git a/pkgs/build-support/cc-wrapper/default.nix b/pkgs/build-support/cc-wrapper/default.nix index c8e3d8b4cc82..95e0b360937a 100644 --- a/pkgs/build-support/cc-wrapper/default.nix +++ b/pkgs/build-support/cc-wrapper/default.nix @@ -281,9 +281,6 @@ stdenv.mkDerivation { crossAttrs = { shell = shell.crossDrv + shell.crossDrv.shellPath; libc = stdenv.ccCross.libc; - coreutils = coreutils.crossDrv; - binutils = binutils.crossDrv; - cc = cc.crossDrv; # # This is not the best way to do this. I think the reference should be # the style in the gcc-cross-wrapper, but to keep a stable stdenv now I From 1c0365bd88b7d98377ff639b5ede5f725e05663b Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 15 Jan 2017 16:47:53 -0500 Subject: [PATCH 08/14] cross-stdenv: Inline useless bindings and reindent Semantics should be unchanged --- pkgs/stdenv/adapters.nix | 94 ++++++++++++++++++++-------------------- 1 file changed, 46 insertions(+), 48 deletions(-) diff --git a/pkgs/stdenv/adapters.nix b/pkgs/stdenv/adapters.nix index 5d66db266c75..7e0eaeddd2c4 100644 --- a/pkgs/stdenv/adapters.nix +++ b/pkgs/stdenv/adapters.nix @@ -56,61 +56,59 @@ rec { # Return a modified stdenv that adds a cross compiler to the # builds. - makeStdenvCross = stdenv: cross: binutilsCross: gccCross: stdenv // - { mkDerivation = {name ? "", buildInputs ? [], nativeBuildInputs ? [], - propagatedBuildInputs ? [], propagatedNativeBuildInputs ? [], - selfNativeBuildInput ? false, ...}@args: let + makeStdenvCross = stdenv: cross: binutilsCross: gccCross: stdenv // { - # *BuildInputs exists temporarily as another name for - # *HostInputs. + mkDerivation = + { name ? "", buildInputs ? [], nativeBuildInputs ? [] + , propagatedBuildInputs ? [], propagatedNativeBuildInputs ? [] + , selfNativeBuildInput ? false, ... + } @ args: - # In nixpkgs, sometimes 'null' gets in as a buildInputs element, - # and we handle that through isAttrs. - nativeBuildInputsDrvs = nativeBuildInputs; - buildInputsDrvs = buildInputs; - propagatedBuildInputsDrvs = propagatedBuildInputs; - propagatedNativeBuildInputsDrvs = propagatedNativeBuildInputs; + let + # *BuildInputs exists temporarily as another name for + # *HostInputs. - # The base stdenv already knows that nativeBuildInputs and - # buildInputs should be built with the usual gcc-wrapper - # And the same for propagatedBuildInputs. - nativeDrv = stdenv.mkDerivation args; + # The base stdenv already knows that nativeBuildInputs and + # buildInputs should be built with the usual gcc-wrapper + # And the same for propagatedBuildInputs. + nativeDrv = stdenv.mkDerivation args; - # Temporary expression until the cross_renaming, to handle the - # case of pkgconfig given as buildInput, but to be used as - # nativeBuildInput. - hostAsNativeDrv = drv: - builtins.unsafeDiscardStringContext drv.nativeDrv.drvPath - == builtins.unsafeDiscardStringContext drv.crossDrv.drvPath; - buildInputsNotNull = stdenv.lib.filter - (drv: builtins.isAttrs drv && drv ? nativeDrv) buildInputs; - nativeInputsFromBuildInputs = stdenv.lib.filter hostAsNativeDrv buildInputsNotNull; + # Temporary expression until the cross_renaming, to handle the + # case of pkgconfig given as buildInput, but to be used as + # nativeBuildInput. + hostAsNativeDrv = drv: + builtins.unsafeDiscardStringContext drv.nativeDrv.drvPath + == builtins.unsafeDiscardStringContext drv.crossDrv.drvPath; + buildInputsNotNull = stdenv.lib.filter + (drv: builtins.isAttrs drv && drv ? nativeDrv) buildInputs; + nativeInputsFromBuildInputs = stdenv.lib.filter hostAsNativeDrv buildInputsNotNull; + in + stdenv.mkDerivation (args // { + name = name + "-" + cross.config; + nativeBuildInputs = nativeBuildInputs + ++ nativeInputsFromBuildInputs + ++ [ gccCross binutilsCross ] + ++ stdenv.lib.optional selfNativeBuildInput nativeDrv + # without proper `file` command, libtool sometimes fails + # to recognize 64-bit DLLs + ++ stdenv.lib.optional (cross.config == "x86_64-w64-mingw32") pkgs.file + ; - in stdenv.mkDerivation (args // { - name = name + "-" + cross.config; - nativeBuildInputs = nativeBuildInputsDrvs - ++ nativeInputsFromBuildInputs - ++ [ gccCross binutilsCross ] - ++ stdenv.lib.optional selfNativeBuildInput nativeDrv - # without proper `file` command, libtool sometimes fails - # to recognize 64-bit DLLs - ++ stdenv.lib.optional (cross.config == "x86_64-w64-mingw32") pkgs.file - ; + # Cross-linking dynamic libraries, every buildInput should + # be propagated because ld needs the -rpath-link to find + # any library needed to link the program dynamically at + # loader time. ld(1) explains it. + buildInputs = []; + propagatedBuildInputs = propagatedBuildInputs ++ buildInputs; + propagatedNativeBuildInputs = propagatedNativeBuildInputs; - # Cross-linking dynamic libraries, every buildInput should - # be propagated because ld needs the -rpath-link to find - # any library needed to link the program dynamically at - # loader time. ld(1) explains it. - buildInputs = []; - propagatedBuildInputs = propagatedBuildInputsDrvs ++ buildInputsDrvs; - propagatedNativeBuildInputs = propagatedNativeBuildInputsDrvs; + crossConfig = cross.config; + } // args.crossAttrs or {}); - crossConfig = cross.config; - } // args.crossAttrs or {}); - } // { - inherit gccCross binutilsCross; - ccCross = gccCross; - }; + inherit gccCross binutilsCross; + ccCross = gccCross; + + }; /* Modify a stdenv so that the specified attributes are added to From a1a798f01788bc3efe4ca477f2c9e8545eeafddc Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 15 Jan 2017 17:31:52 -0500 Subject: [PATCH 09/14] top-level: `crossSystem` is no longer exposed to packages. Use `*Platform`. --- pkgs/development/compilers/ghc/head.nix | 7 ++- pkgs/os-specific/gnu/default.nix | 9 +-- .../linux/make-bootstrap-tools-cross.nix | 2 +- pkgs/top-level/all-packages.nix | 60 +++++++++---------- pkgs/top-level/haskell-packages.nix | 4 +- pkgs/top-level/stage.nix | 1 - 6 files changed, 42 insertions(+), 41 deletions(-) diff --git a/pkgs/development/compilers/ghc/head.nix b/pkgs/development/compilers/ghc/head.nix index 971365eda48f..e7f4335d6f6e 100644 --- a/pkgs/development/compilers/ghc/head.nix +++ b/pkgs/development/compilers/ghc/head.nix @@ -1,5 +1,6 @@ { stdenv, fetchgit, bootPkgs, perl, gmp, ncurses, libiconv, binutils, coreutils -, autoconf, automake, happy, alex, python3, crossSystem, selfPkgs, cross ? null +, autoconf, automake, happy, alex, python3, buildPlatform, targetPlatform +, selfPkgs, cross ? null }: let @@ -68,9 +69,9 @@ in stdenv.mkDerivation (rec { passthru = { inherit bootPkgs; - } // stdenv.lib.optionalAttrs (crossSystem != null) { + } // stdenv.lib.optionalAttrs (targetPlatform != buildPlatform) { crossCompiler = selfPkgs.ghc.override { - cross = crossSystem; + cross = targetPlatform; bootPkgs = selfPkgs; }; }; diff --git a/pkgs/os-specific/gnu/default.nix b/pkgs/os-specific/gnu/default.nix index c002447d64b1..247c73e468d0 100644 --- a/pkgs/os-specific/gnu/default.nix +++ b/pkgs/os-specific/gnu/default.nix @@ -3,7 +3,8 @@ args@{ fetchgit, stdenv, autoconf, automake, automake111x, libtool , texinfo, glibcCross, hurdPartedCross, libuuid, samba , gccCrossStageStatic, gccCrossStageFinal -, forcedNativePackages, forceSystem, newScope, platform, config, crossSystem +, forcedNativePackages, forceSystem, newScope, platform, config +, targetPlatform, buildPlatform , overrides ? {} }: with args; @@ -19,7 +20,7 @@ let libuuid = libuuid.crossDrv; automake = automake111x; headersOnly = false; - cross = assert crossSystem != null; crossSystem; + cross = assert targetPlatform != buildPlatform; targetPlatform; gccCross = gccCrossStageFinal; }; @@ -30,7 +31,7 @@ let libuuid = null; automake = automake111x; headersOnly = false; - cross = assert crossSystem != null; crossSystem; + cross = assert targetPlatform != buildPlatform; targetPlatform; # The "final" GCC needs glibc and the Hurd libraries (libpthread in # particular) so we first need an intermediate Hurd built with the @@ -63,7 +64,7 @@ let inherit (gnu) machHeaders hurdHeaders; hurd = gnu.hurdCrossIntermediate; gccCross = gccCrossStageStatic; - cross = assert crossSystem != null; crossSystem; + cross = assert targetPlatform != buildPlatform; targetPlatform; }; # In theory GNU Mach doesn't have to be cross-compiled. However, since it diff --git a/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix b/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix index a1b1f02d83d8..38b3e611bc28 100644 --- a/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix +++ b/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix @@ -116,7 +116,7 @@ rec { stdenv.mkDerivation { name = "stdenv-bootstrap-tools-cross"; - crossConfig = pkgsUnspliced.crossSystem.config; + crossConfig = pkgsUnspliced.hostPlatform.config; buildInputs = [nukeReferences cpio binutilsCross]; diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix index 47c565bd2ee7..1fe326e77627 100644 --- a/pkgs/top-level/all-packages.nix +++ b/pkgs/top-level/all-packages.nix @@ -27,7 +27,7 @@ with pkgs; callPackage_i686 = pkgsi686Linux.callPackage; - forcedNativePackages = if crossSystem == null then pkgs else buildPackages; + forcedNativePackages = if hostPlatform == buildPlatform then pkgs else buildPackages; # A stdenv capable of building 32-bit binaries. On x86_64-linux, # it uses GCC compiled with multilib support; on i686-linux, it's @@ -3246,7 +3246,7 @@ with pkgs; pngout = callPackage ../tools/graphics/pngout { }; hurdPartedCross = - if crossSystem != null && crossSystem.config == "i586-pc-gnu" + if targetPlatform != buildPlatform && targetPlatform.config == "i586-pc-gnu" then (makeOverridable ({ hurd }: (parted.override { @@ -4751,14 +4751,14 @@ with pkgs; gccApple = throw "gccApple is no longer supported"; - gccCrossStageStatic = assert crossSystem != null; let + gccCrossStageStatic = assert targetPlatform != buildPlatform; let libcCross1 = if stdenv.cross.libc == "msvcrt" then windows.mingw_w64_headers else if stdenv.cross.libc == "libSystem" then darwin.xcode else null; in wrapGCCCross { gcc = forcedNativePackages.gcc.cc.override { - cross = crossSystem; + cross = targetPlatform; crossStageStatic = true; langCC = false; libcCross = libcCross1; @@ -4768,31 +4768,31 @@ with pkgs; }; libc = libcCross1; binutils = binutilsCross; - cross = crossSystem; + cross = targetPlatform; }; # Only needed for mingw builds - gccCrossMingw2 = assert crossSystem != null; wrapGCCCross { + gccCrossMingw2 = assert targetPlatform != buildPlatform; wrapGCCCross { gcc = gccCrossStageStatic.gcc; libc = windows.mingw_headers2; binutils = binutilsCross; - cross = assert crossSystem != null; crossSystem; + cross = targetPlatform; }; - gccCrossStageFinal = assert crossSystem != null; wrapGCCCross { + gccCrossStageFinal = assert targetPlatform != buildPlatform; wrapGCCCross { gcc = forcedNativePackages.gcc.cc.override { - cross = crossSystem; + cross = targetPlatform; crossStageStatic = false; # XXX: We have troubles cross-compiling libstdc++ on MinGW (see # ), so don't even try. - langCC = crossSystem.config != "i686-pc-mingw32"; + langCC = targetPlatform.config != "i686-pc-mingw32"; # Why is this needed? inherit (forcedNativePackages) binutilsCross; }; libc = libcCross; binutils = binutilsCross; - cross = crossSystem; + cross = targetPlatform; }; gcc45 = lowPrio (wrapCC (callPackage ../development/compilers/gcc/4.5 { @@ -4810,7 +4810,7 @@ with pkgs; # and host != build), `cross' must be null but the cross-libc must still # be passed. cross = null; - libcCross = if crossSystem != null then libcCross else null; + libcCross = if targetPlatform != buildPlatform then libcCross else null; })); gcc48 = lowPrio (wrapCC (callPackage ../development/compilers/gcc/4.8 { @@ -4823,7 +4823,7 @@ with pkgs; # and host != build), `cross' must be null but the cross-libc must still # be passed. cross = null; - libcCross = if crossSystem != null then libcCross else null; + libcCross = if targetPlatform != buildPlatform then libcCross else null; isl = if !stdenv.isDarwin then isl_0_14 else null; cloog = if !stdenv.isDarwin then cloog else null; @@ -4840,7 +4840,7 @@ with pkgs; # and host != build), `cross' must be null but the cross-libc must still # be passed. cross = null; - libcCross = if crossSystem != null then libcCross else null; + libcCross = if targetPlatform != buildPlatform then libcCross else null; isl = if !stdenv.isDarwin then isl_0_11 else null; @@ -4857,7 +4857,7 @@ with pkgs; # and host != build), `cross' must be null but the cross-libc must still # be passed. cross = null; - libcCross = if crossSystem != null then libcCross else null; + libcCross = if targetPlatform != buildPlatform then libcCross else null; isl = if !stdenv.isDarwin then isl_0_14 else null; })); @@ -4872,7 +4872,7 @@ with pkgs; # and host != build), `cross' must be null but the cross-libc must still # be passed. cross = null; - libcCross = if crossSystem != null then libcCross else null; + libcCross = if targetPlatform != buildPlatform then libcCross else null; isl = if !stdenv.isDarwin then isl_0_14 else null; })); @@ -5005,7 +5005,7 @@ with pkgs; # Haskell and GHC - haskell = callPackage ./haskell-packages.nix { inherit crossSystem; }; + haskell = callPackage ./haskell-packages.nix { }; haskellPackages = haskell.packages.ghc801.override { overrides = config.haskellPackageOverrides or (self: super: {}); @@ -6068,11 +6068,11 @@ with pkgs; gold = false; }); - binutilsCross = assert crossSystem != null; lowPrio ( - if crossSystem.libc == "libSystem" then darwin.cctools_cross + binutilsCross = assert targetPlatform != buildPlatform; lowPrio ( + if targetPlatform.libc == "libSystem" then darwin.cctools_cross else forcedNativePackages.binutils.override { noSysDirs = true; - cross = crossSystem; + cross = targetPlatform; }); bison2 = callPackage ../development/tools/parsing/bison/2.x.nix { }; @@ -6718,7 +6718,7 @@ with pkgs; gdbGuile = lowPrio (gdb.override { inherit guile; }); gdbCross = lowPrio (callPackage ../development/tools/misc/gdb { - target = crossSystem; + target = if targetPlatform != buildPlatform then targetPlatform else null; }); gdb-multitarget = lowPrio (gdb.override { multitarget = true; }); @@ -7332,7 +7332,7 @@ with pkgs; else if name == "libSystem" then darwin.xcode else throw "Unknown libc"; - libcCross = assert crossSystem != null; libcCrossChooser crossSystem.libc; + libcCross = assert targetPlatform != buildPlatform; libcCrossChooser targetPlatform.libc; # Only supported on Linux glibcLocales = if stdenv.isLinux then callPackage ../development/libraries/glibc/locales.nix { } else null; @@ -10895,7 +10895,7 @@ with pkgs; apple-source-releases = callPackage ../os-specific/darwin/apple-source-releases { }; in apple-source-releases // rec { cctools_cross = callPackage (forcedNativePackages.callPackage ../os-specific/darwin/cctools/port.nix {}).cross { - cross = assert crossSystem != null; crossSystem; + cross = assert targetPlatform != buildPlatform; targetPlatform; inherit maloader; xctoolchain = xcode.toolchain; }; @@ -10968,7 +10968,7 @@ with pkgs; libossp_uuid = callPackage ../development/libraries/libossp-uuid { }; libuuid = - if crossSystem != null && crossSystem.config == "i586-pc-gnu" + if targetPlatform != buildPlatform && targetPlatform.config == "i586-pc-gnu" then (utillinuxMinimal // { crossDrv = lib.overrideDerivation utillinuxMinimal.crossDrv (args: { # `libblkid' fails to build on GNU/Hurd. @@ -11053,7 +11053,7 @@ with pkgs; # GNU/Hurd core packages. gnu = recurseIntoAttrs (callPackage ../os-specific/gnu { - inherit platform crossSystem; + inherit platform; }); hwdata = callPackage ../os-specific/linux/hwdata { }; @@ -11133,11 +11133,11 @@ with pkgs; linuxHeaders = linuxHeaders_4_4; linuxHeaders24Cross = forcedNativePackages.callPackage ../os-specific/linux/kernel-headers/2.4.nix { - cross = assert crossSystem != null; crossSystem; + cross = assert targetPlatform != buildPlatform; targetPlatform; }; linuxHeaders26Cross = forcedNativePackages.callPackage ../os-specific/linux/kernel-headers/4.4.nix { - cross = assert crossSystem != null; crossSystem; + cross = assert targetPlatform != buildPlatform; targetPlatform; }; linuxHeaders_3_18 = callPackage ../os-specific/linux/kernel-headers/3.18.nix { }; @@ -11149,8 +11149,8 @@ with pkgs; else if ver == "2.6" then linuxHeaders26Cross else throw "Unknown linux kernel version"; - linuxHeadersCross = assert crossSystem != null; - linuxHeadersCrossChooser crossSystem.platform.kernelMajor; + linuxHeadersCross = assert targetPlatform != buildPlatform; + linuxHeadersCrossChooser targetPlatform.platform.kernelMajor; kernelPatches = callPackage ../os-specific/linux/kernel/patches.nix { }; @@ -11812,7 +11812,7 @@ with pkgs; uclibcCross = lowPrio (callPackage ../os-specific/linux/uclibc { linuxHeaders = linuxHeadersCross; gccCross = gccCrossStageStatic; - cross = assert crossSystem != null; crossSystem; + cross = assert targetPlatform != buildPlatform; targetPlatform; }); udev = systemd; diff --git a/pkgs/top-level/haskell-packages.nix b/pkgs/top-level/haskell-packages.nix index 5b14af145e9f..7309121486e5 100644 --- a/pkgs/top-level/haskell-packages.nix +++ b/pkgs/top-level/haskell-packages.nix @@ -1,4 +1,4 @@ -{ pkgs, callPackage, stdenv, crossSystem }: +{ pkgs, callPackage, stdenv, buildPlatform, targetPlatform }: rec { @@ -55,7 +55,7 @@ rec { ghcHEAD = callPackage ../development/compilers/ghc/head.nix rec { bootPkgs = packages.ghc7103; inherit (bootPkgs) alex happy; - inherit crossSystem; + inherit buildPlatform targetPlatform; selfPkgs = packages.ghcHEAD; }; ghcjs = packages.ghc7103.callPackage ../development/compilers/ghcjs { diff --git a/pkgs/top-level/stage.nix b/pkgs/top-level/stage.nix index 780be566436c..0f3d967a0914 100644 --- a/pkgs/top-level/stage.nix +++ b/pkgs/top-level/stage.nix @@ -101,7 +101,6 @@ let cross = targetPlatform; }; inherit (buildPlatform) system platform; - crossSystem = if targetPlatform != buildPlatform then targetPlatform else null; }; splice = self: super: import ./splice.nix lib self; From 4c17cd555f6443207144da9af6e1c2b1304afd8b Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 15 Jan 2017 17:45:04 -0500 Subject: [PATCH 10/14] top-level: Document the `{local,cross}System, contrasting with `*Platform` This is an implementation detail of how the bootstrapping chain is chosen, and thus need not be in the manual. --- pkgs/top-level/default.nix | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/pkgs/top-level/default.nix b/pkgs/top-level/default.nix index b7a4273acff0..3c67d316f7c8 100644 --- a/pkgs/top-level/default.nix +++ b/pkgs/top-level/default.nix @@ -83,6 +83,22 @@ in let boot = import ../stdenv/booter.nix { inherit lib allPackages; }; stages = stdenvStages { + # One would think that `localSystem` and `crossSystem` overlap horribly with + # the three `*Platforms` (`buildPlatform`, `hostPlatform,` and + # `targetPlatform`; see `stage.nix` or the manual). Actually, those + # identifiers I, @Ericson2314, purposefully not used here to draw a subtle + # but important distinction: + # + # While the granularity of having 3 platforms is necessary to properly + # *build* packages, it is overkill for specifying the user's *intent* when + # making a build plan or package set. A simple "build vs deploy" dichotomy + # is adequate: the "sliding window" principle described in the manual shows + # how to interpolate between the these two "end points" to get the 3 + # platform triple for each bootstrapping stage. + # + # Also, less philosophically but quite practically, `crossSystem` should be + # null when one doesn't want to cross-compile, while the `*Platform`s are + # always non-null. `localSystem` is always non-null. localSystem = { inherit system platform; }; inherit lib crossSystem config overlays; }; From bfb147b6a8c4a3ddc581aab0b8a29b418db3b7a6 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 15 Jan 2017 18:03:59 -0500 Subject: [PATCH 11/14] top-level: Only splice as needed for performance --- pkgs/stdenv/booter.nix | 13 ++++++------- pkgs/top-level/splice.nix | 11 +++++++++-- pkgs/top-level/stage.nix | 8 +++++--- 3 files changed, 20 insertions(+), 12 deletions(-) diff --git a/pkgs/stdenv/booter.nix b/pkgs/stdenv/booter.nix index 6e5d073e55a7..2c82d12da95d 100644 --- a/pkgs/stdenv/booter.nix +++ b/pkgs/stdenv/booter.nix @@ -63,12 +63,11 @@ stageFuns: let __bootPackages = finalSoFar; }; }; - self = - if args.__raw or false - then args' - else allPackages ((builtins.removeAttrs args' ["selfBuild"]) // { - buildPackages = if args.selfBuild or true then self else finalSoFar; - }); - in self; + in + if args.__raw or false + then args' + else allPackages ((builtins.removeAttrs args' ["selfBuild"]) // { + buildPackages = if args.selfBuild or true then null else finalSoFar; + }); in lib.lists.fold folder {} withAllowCustomOverrides diff --git a/pkgs/top-level/splice.nix b/pkgs/top-level/splice.nix index f57f42020c27..7afbd956d5bb 100644 --- a/pkgs/top-level/splice.nix +++ b/pkgs/top-level/splice.nix @@ -17,7 +17,11 @@ # `mkDerivation` knows how to pull out the right ones for `buildDepends` and # friends, but a few packages use them directly, so it seemed efficient (to # @Ericson2314) to reuse those names, at least initially, to minimize breakage. -lib: pkgs: +# +# For performance reasons, rather than uniformally splice in all cases, we only +# do so when `pkgs` and `buildPackages` are distinct. The `actuallySplice` +# parameter there the boolean value of that equality check. +lib: pkgs: actuallySplice: let defaultBuildScope = pkgs.buildPackages // pkgs.buildPackages.xorg; @@ -58,7 +62,10 @@ let }; in lib.listToAttrs (map merge (lib.attrNames mash)); - splicedPackages = splicer defaultBuildScope defaultRunScope; + splicedPackages = + if actuallySplice + then splicer defaultBuildScope defaultRunScope + else pkgs // pkgs.xorg; in diff --git a/pkgs/top-level/stage.nix b/pkgs/top-level/stage.nix index 0f3d967a0914..6febedb79f3d 100644 --- a/pkgs/top-level/stage.nix +++ b/pkgs/top-level/stage.nix @@ -45,7 +45,8 @@ ## Other parameters ## -, # The package set used at build-time +, # The package set used at build-time. If null, `buildPackages` will + # be defined internally as the produced package set as itself. buildPackages , # The standard environment to use for building packages. @@ -84,7 +85,8 @@ let }; stdenvBootstappingAndPlatforms = self: super: { - buildPackages = buildPackages // { recurseForDerivations = false; }; + buildPackages = (if buildPackages == null then self else buildPackages) + // { recurseForDerivations = false; }; inherit stdenv buildPlatform hostPlatform targetPlatform; }; @@ -103,7 +105,7 @@ let inherit (buildPlatform) system platform; }; - splice = self: super: import ./splice.nix lib self; + splice = self: super: import ./splice.nix lib self (buildPackages != null); allPackages = self: super: let res = import ./all-packages.nix From 7dc4e43837a83d1b938b7eba0541b32f4e9bce72 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 22 Jan 2017 14:36:16 -0500 Subject: [PATCH 12/14] nixos doc: Mention cross overhaul in 17.03 release notes --- nixos/doc/manual/release-notes/rl-1703.xml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/nixos/doc/manual/release-notes/rl-1703.xml b/nixos/doc/manual/release-notes/rl-1703.xml index 6be2cd3af7f8..aa864b7a757a 100644 --- a/nixos/doc/manual/release-notes/rl-1703.xml +++ b/nixos/doc/manual/release-notes/rl-1703.xml @@ -30,6 +30,15 @@ has the following highlights: following incompatible changes: + + + Cross compilation has been rewritten. See the nixpkgs manual for + details. The most obvious breaking change is that derivations absent a + .nativeDrv or .crossDrv are now + cross by default, not native. + + + stdenv.overrides is now expected to take self From 76ea89aa78c9099a2e5c5121942cdaa8b2d52c89 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 22 Jan 2017 14:36:46 -0500 Subject: [PATCH 13/14] nixpkgs doc: Talk about nativeBuildInputs and propgatedNativeBuildInputs Do so in the stdenv section where the other two are discussed. This can be done without brining up cross-compilation by talking about build-time vs run-time. --- doc/stdenv.xml | 90 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 54 insertions(+), 36 deletions(-) diff --git a/doc/stdenv.xml b/doc/stdenv.xml index 44a0e4601fc1..6ec5c9f2814f 100644 --- a/doc/stdenv.xml +++ b/doc/stdenv.xml @@ -194,33 +194,52 @@ genericBuild tools. + + + + Variables specifying dependencies + + + nativeBuildInputs + + A list of dependencies used by the new derivation at build-time. + I.e. these dependencies should not make it into the package's runtime-closure, though this is currently not checked. + For each dependency dir, the directory dir/bin, if it exists, is added to the PATH environment variable. + Other environment variables are also set up via a pluggable mechanism. + For instance, if buildInputs contains Perl, then the lib/site_perl subdirectory of each input is added to the PERL5LIB environment variable. + See for details. + + + buildInputs - A list of dependencies used by - stdenv to set up the environment for the build. - For each dependency dir, the directory - dir/bin, if it - exists, is added to the PATH environment variable. - Other environment variables are also set up via a pluggable - mechanism. For instance, if buildInputs - contains Perl, then the lib/site_perl - subdirectory of each input is added to the PERL5LIB - environment variable. See for - details. + + A list of dependencies used by the new derivation at run-time. + Currently, the build-time environment is modified in the exact same way as with nativeBuildInputs. + This is problematic in that when cross-compiling, foreign executables can clobber native ones on the PATH. + Even more confusing is static-linking. + A statically-linked library should be listed here because ultimately that generated machine code will be used at run-time, even though a derivation containing the object files or static archives will only be used at build-time. + A less confusing solution to this would be nice. + - + + + + propagatedNativeBuildInputs + + Like nativeBuildInputs, but these dependencies are propagated: + that is, the dependencies listed here are added to the nativeBuildInputs of any package that uses this package as a dependency. + So if package Y has propagatedBuildInputs = [X], and package Z has buildInputs = [Y], then package X will appear in Z’s build environment automatically. + + + propagatedBuildInputs - Like buildInputs, but these - dependencies are propagated: that is, the - dependencies listed here are added to the - buildInputs of any package that uses - this package as a dependency. So if package - Y has propagatedBuildInputs = [X], and package - Z has buildInputs = [Y], then package X will - appear in Z’s build environment automatically. + + Like buildInputs, but propagated just like propagatedNativeBuildInputs. + This inherits buildInputs's flaws of clobbering native executables when cross-compiling and being confusing for static linking. + - @@ -322,7 +341,7 @@ executed and in what order: $preInstallPhases installPhase fixupPhase $preDistPhases distPhase $postPhases. - + Usually, if you just want to add a few phases, it’s more convenient to set one of the variables below (such as preInstallPhases), as you then don’t specify @@ -706,7 +725,7 @@ makeFlagsArray=(CFLAGS="-O0 -g" LDFLAGS="-lfoo -lbar") - + You can set flags for make through the makeFlags variable. @@ -773,7 +792,7 @@ doCheck = true; - + @@ -840,12 +859,12 @@ install phase. The default fixupPhase does the following: - + It moves the man/, doc/ and info/ subdirectories of $out to share/. - + It strips libraries and executables of debug information. @@ -1091,13 +1110,13 @@ functions. - + substitute infile outfile subs - + Performs string substitution on the contents of infile, writing the result to @@ -1125,7 +1144,7 @@ functions. @...@ in the template as placeholders. - + varName @@ -1134,7 +1153,7 @@ functions. @varName@ by the string s. - + @@ -1162,7 +1181,7 @@ substitute ./foo.in ./foo.out \ - + substituteInPlace @@ -1173,7 +1192,7 @@ substitute ./foo.in ./foo.out \ file. - + substituteAll infile @@ -1233,7 +1252,7 @@ echo @foo@ Strips the directory and hash part of a store path, outputting the name part to stdout. For example: - + # prints coreutils-8.24 stripHash "/nix/store/9s9r019176g7cvn2nvcw41gsp862y6b4-coreutils-8.24" @@ -1241,7 +1260,7 @@ stripHash "/nix/store/9s9r019176g7cvn2nvcw41gsp862y6b4-coreutils-8.24" If you wish to store the result in another variable, then the following idiom may be useful: - + name="/nix/store/9s9r019176g7cvn2nvcw41gsp862y6b4-coreutils-8.24" someVar=$(stripHash $name) @@ -1250,7 +1269,7 @@ someVar=$(stripHash $name) - + @@ -1607,4 +1626,3 @@ Arch Wiki. - From 39fb46f5384bd4d28f1b10580ac9227c4ed36da2 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 22 Jan 2017 15:52:35 -0500 Subject: [PATCH 14/14] nixpkgs docs: Cross compilation docs --- doc/cross-compilation.xml | 153 ++++++++++++++++++++++++++++++++++++++ doc/manual.xml | 1 + 2 files changed, 154 insertions(+) create mode 100644 doc/cross-compilation.xml diff --git a/doc/cross-compilation.xml b/doc/cross-compilation.xml new file mode 100644 index 000000000000..e93d1a98f7fd --- /dev/null +++ b/doc/cross-compilation.xml @@ -0,0 +1,153 @@ + + +Cross-compilation + +
+ Introduction + + "Cross-compilation" means compiling a program on one machine for another type of machine. + For example, a typical use of cross compilation is to compile programs for embedded devices. + These devices often don't have the computing power and memory to compile their own programs. + One might think that cross-compilation is a fairly niche concern, but there are advantages to being rigorous about distinguishing build-time vs run-time environments even when one is developing and deploying on the same machine. + Nixpkgs is increasingly adopting this opinion in that packages should be written with cross-compilation in mind, and nixpkgs should evaluate in a similar way (by minimizing cross-compilation-specific special cases) whether or not one is cross-compiling. + + + + This chapter will be organized in three parts. + First, it will describe the basics of how to package software in a way that supports cross-compilation. + Second, it will describe how to use Nixpkgs when cross-compiling. + Third, it will describe the internal infrastructure supporting cross-compilation. + +
+ + + +
+ Packing in a cross-friendly manner + +
+ Platform parameters + + The three GNU Autoconf platforms, build, host, and cross, are historically the result of much confusion. + clears this up somewhat but there is more to be said. + An important advice to get out the way is, unless you are packaging a compiler or other build tool, just worry about the build and host platforms. + Dealing with just two platforms usually better matches people's preconceptions, and in this case is completely correct. + + + In Nixpkgs, these three platforms are defined as attribute sets under the names buildPlatform, hostPlatform, and targetPlatform. + All are guaranteed to contain at least a platform field, which contains detailed information on the platform. + All three are always defined at the top level, so one can get at them just like a dependency in a function that is imported with callPackage: + { stdenv, buildPlatform, hostPlatform, fooDep, barDep, .. }: ... + + + These platforms should all have the same structure in all scenarios, but that is currently not the case. + When not cross-compiling, they will each contain a system field with a short 2-part, hyphen-separated summering string name for the platform. + But, when when cross compiling, hostPlatform and targetPlatform may instead contain config with a fuller 3- or 4-part string in the manner of LLVM. + We should have all 3 platforms always contain both, and maybe give config a better name while we are at it. + + + + buildPlatform + + The "build platform" is the platform on which a package is built. + Once someone has a built package, or pre-built binary package, the build platform should not matter and be safe to ignore. + + + + hostPlatform + + The "host platform" is the platform on which a package is run. + This is the simplest platform to understand, but also the one with the worst name. + + + + targetPlatform + + + The "target platform" is black sheep. + The other two intrinsically apply to all compiled software—or any build process with a notion of "build-time" followed by "run-time". + The target platform only applies to programming tools, and even then only is a good for for some of them. + Briefly, GCC, Binutils, GHC, and certain other tools are written in such a way such that a single build can only compiler code for a single platform. + Thus, when building them, one must think ahead about what platforms they wish to use the tool to produce machine code for, and build binaries for each. + + + There is no fundamental need to think about the target ahead of time like this. + LLVM, for example, was designed from the beginning with cross-compilation in mind, and so a normal LLVM binary will support every architecture that LLVM supports. + If the tool supports modular or pluggable backends, one might imagine specifying a set of target platforms / backends one wishes to support, rather than a single one. + + + The biggest reason for mess, if there is one, is that many compilers have the bad habit a build process that builds the compiler and standard library/runtime together. + Then the specifying target platform is essential, because it determines the host platform of the standard library/runtime. + Nixpkgs tries to avoid this where possible too, but still, because the concept of a target platform is so ingrained now in Autoconf and other tools, it is best to support it as is. + Tools like LLVM that don't need up-front target platforms can safely ignore it like normal packages, and it will do no harm. + + + + + + If you dig around nixpkgs, you may notice there is also stdenv.cross. + This field defined as hostPlatform when the host and build platforms differ, but otherwise not defined at all. + This field is obsolete and will soon disappear—please do not use it. + +
+ +
+ Specifying Dependencies + + As mentioned in the introduction to this chapter, one can think about a build time vs run time distinction whether cross-compiling or not. + In the case of cross-compilation, this corresponds with whether a derivation running on the native or foreign platform is produced. + An interesting thing to think about is how this corresponds with the three Autoconf platforms. + In the run-time case, the depending and depended-on package simply have matching build, host, and target platforms. + But in the build-time case, one can imagine "sliding" the platforms one over. + The depended-on package's host and target platforms (respectively) become the depending package's build and host platforms. + This is the most important guiding principle behind cross-compilation with Nixpkgs, and will be called the sliding window principle. + In this manner, given the 3 platforms for one package, we can determine the three platforms for all its transitive dependencies. + + + The depending package's target platform is unconstrained by the sliding window principle, which makes sense in that one can in principle build cross compilers targeting arbitrary platforms. + + + From the above, one would surmise that if a package is being built with a (build, host, target) platform triple of (foo, bar, bar), then its build-time dependencies would have a triple of (foo, foo, bar), and those packages' build-time dependencies would have triple of (foo, foo, foo). + In other words, it should take two "rounds" of following build-time dependency edges before one reaches a fixed point where, by the sliding window principle, the platform triple no longer changes. + Unfortunately, at the moment, we do not implement this correctly, and after only one round of following build-time dependencies is the fixed point reached, with target incorrectly kept different than the others. + + + How does this work in practice? Nixpkgs is now structured so that build-time dependencies are taken from from buildPackages, whereas run-time dependencies are taken from the top level attribute set. + For example, buildPackages.gcc should be used at build time, while gcc should be used at run time. + Now, for most of Nixpkgs's history, there was no buildPackages, and most packages have not been refactored to use it explicitly. + Instead, one can use the four attributes used for specifying dependencies as documented in . + We "splice" together the run-time and build-time package sets with callPackage, and then mkDerivation for each of four attributes pulls the right derivation out. + This splicing can be skipped when not cross compiling as the package sets are the same, but is a bit slow for cross compiling. + Because of this, a best-of-both-worlds solution is in the works with no splicing or explicit access of buildPackages needed. + For now, feel free to use either method. + +
+ +
+ + + +
+ Cross-building packages + + To be written. + This is basically unchanged so see the old wiki for now. + +
+ + + +
+ Cross-compilation infrastructure + To be written. + + If one explores nixpkgs, they will see derivations with names like gccCross. + Such *Cross derivations is a holdover from before we properly distinguished between the host and target platforms + —the derivation with "Cross" in the name covered the build = host != target case, while the other covered the host = target, with build platform the same or not based on whether one was using its .nativeDrv or .crossDrv. + This ugliness will disappear soon. + +
+ + diff --git a/doc/manual.xml b/doc/manual.xml index 1c0dac6e4df7..75bd21557fd1 100644 --- a/doc/manual.xml +++ b/doc/manual.xml @@ -13,6 +13,7 @@ +