* Add script for fixing materialized files
This is useful in two ways:
- We reference it in the error message, so users can more easily
copy-paste the correct call, rather than several lines of commands.
- We add it to `passthru`, so users can reference it directly when they
know they need to update the files, without having to first run a build
and find it on stderr.
* Add script for calculating the sha
This is useful in two ways:
- It makes it easier for users to copy from the message.
- We add it to `passthru` so users can call it directly if they know
they need to update the sha, without having to run a build and copy it
from stderr.
* Refactor unchecked a bit
Put the positive logic first, use the materialized files by preference
if they're there.
* Allow the updater script to take an argument
* Use double brackets
* Fix a dependency
* Separate generating and updating scripts
* ghc 8.10
* Adds/Updates/Removes materialization files
* Disable one-shot-kqueue-on-macos patch on ghc-8.10
* Test ghc883 and ghc8101 but only with nix 20.03
* Various smaller fixes.
This should fix the infinite recursion issue with #654
Co-authored-by: Lennart Spitzner <hexagoxel@hexagoxel.de>
Running `cabal v2-configure` overwrites the `cabal.project.local` file
so this change adds the contents onto the end of the `cabal.project`
file before running `cabal v2-configure`.
You can override this behavior passing `cabalProjectLocal = ""`
Treats cabal.project.feeze the same way as `cabal.project` and
`cabal.project.local` in that it will still use it if it is filtered by cleanGit.
This change also adds `cabalProjectFileName` arg so that
we can override the default `cabal.project` name.
This change improves the dummy ghc scripts by adding a bang pattern,
supporting `--version` and using bash case instead of lots of `elif`
Includes the same fixes for dummy-ghc-pkg
Some times it is handy to temporarily use a relative path between git
repos. If the repos are individually cleaned this is not possible
(since the cleaned version of one repo will never include the files
of the other).
`cleanGits` allows us to specify a root directory and any number of
sub directories containing git repos.
When the `license-file` is not at the top-level, e.g. in the LICENSES
subdirectory, it didn’t have any chance of being included, as the
subdirectory was filtered out right away.
Include `licenseFiles` in the list of subdirs used for determining
“needed” subdirectories (the ones that will be kept).
Although the default ghc used by haskell.nix is ghc 8.6.5 many of
the tools used in haskell.nix are still built with the boot compiler
ghc 8.4.4. These include
* haskell-nix.cabal-install
* haskell-nix.alex
* haskell-nix.happy
This change updates those to ghc 8.6.5 and includes materializations
for the new versions.
When cabal-install is built it is careful to disable materialization
checks on the version of itself used during the build to avoid
infinite recursion.
There was a version of nix-tools built with the boot ghc which was
only used when `checkMaterialization = true`. It was used for
the boot versions of alex, happy and hscolour. These have been update
to use the default (ghc 8.6.5) version of nix-tools and
checkMaterialization is forced off when they are being used to build
ghc. This means the materialization will only be checked for these
when they are built independently (they are included in the test set
via haskellNixRoots).
Three new arguments are added to `default.nix`:
* `defaultCompilerNixName` if not specified "ghc865" is used
* `checkMaterialization` makes it easier to switch on materialization checks
* `system` defaults to `builtins.currentSystem`
This change also moves the work needed for hydra eval to the eval
system using a new `evalPackages` feature. This includes:
* Fetching from git with `fetchgit`
* Building scripts with `runCommand` and `writeTextFile`
* `git ls-files` in `cleanGit`
* running `cabal v2-configure`
* copying materialized files (we are not sure why this is necessary but
if we do not `cp -r` the files nix will not load them on hydra)
Reduce size of `make-config-files.nix` strings by around 80%.
These are unlikely to be the cause of hydra eval time memory
issues in the GB range, but were still quite large (around 10MB for the
`cabal-simple` test case).
There was issue causing excessive builds of the `git` package when
cross compiling. Gory details are a comment in `lib/defaults.nix`
but in short if you use `git` you need an extra `.buildPackages`
one is not enough because it depends on `gdb` and that will
be different in `buildPackages` compared to
`buildPackages.buildPackages`.
Adds missing materialization files for ghc 8.4.4 (only needed
when `checkMaterialization` is on because of other
materialiazations, but good to have).
* Add compiler-nix-name to cabalProject
Setting the compiler explicitly does not make it easy for cross
compilation to also choose the correct compiler when compiling
build tools for the build platform. Also it is easy to pass the
wrong compiler type. Specifying just the nix-name of the compiler
makes that less likely.
Co-authored-by: Moritz Angermann <moritz.angermann@gmail.com>
The `cabalProject` function works by running `cabal v2-configure` followed by `plan-to-nix`. However for `v2-configure` to correctly calculate the plan (including flag settings and buildable components) it needs to know what GHC we want to use.
This causes significant problems:
* For hydra to assemble a list of jobs from `components.tests` it must first have GHC that will be used. If a patch has been applied to the GHC to be used it must be rebuilt before the list of jobs can be assembled. If a lot of different GHCs are being tests that can be a lot of work all happening in the eval stage where little feedback is available.
* Once the jobs are running the compilation of the GHC needed (the eval stage already must have done it, but the outputs there are apparently not added to the cache) happens inside the IFD part of `cabalProject`. This causes a very large amount of work to be done in the IFD and our understanding is that this can cause problems on nix and/or hydra.
* When using `cabalProject` we cannot examine the properties of the project without building or downloading the GHC (less of an issue as we would normally need it soon anyway).
The solution here is to:
* Create a dummy `ghc` and `ghc-pkg` from the real ones by capturing the `ghc` and `ghc-pkg` outputs that `cabal v2-configure` requests.
* The dummy `ghc` and `ghc-pkg` is used instead of the real one in `cabalProject` when running `cabal v2-configure`.
* The captured outputs are materialized into the haskell.nix repo so that the real GHC is only needed when `checkMaterialization` is set.
* Add native musl support
- adds haskellLibl.isNaviveMusl and related fixes (previously it was
assumed that for musl we always use cross-compilation)
- fixed some issues for musl intoduced in nixpkgs 20.03 (ssh/fetchcvs
infinite recursion, busybox pkgsMusl.pkgsStatic chain, etc.)
- cross-compilation fixes for ghc (absolute path for ar, always pass
target to configure, libffi and gmp packages for target platform) -
this needs more testing
* Better explanation for targetPlatform.{libffi, gmp}
* Ghc configure script now respect AR env variable
Co-authored-by: Moritz Angermann <moritz.angermann@gmail.com>
* Add support for external Hackage repositories
Currently haskell.nix is not able to build Cabal projects that depend on
packages from private Hackage repositories, as it make only main Hackage
available to cabal. This is unfortunate.
This commit adds this functionality, by allowing the user to pass
`extra-hackages` and `extra-hackage-tarballs` to `mkPkgSet` and
`callCabalToNix` respectively, to add as much extra repositories as
needed.
This repositories are first made available to Cabal when calling
`v2-configure`, resulting in correct plans. Later they are combined with
global Hackage when building dependencies of the local packages.
* Use cabal.project.freeze if available
Currently callCabalProjectToNix does not copy `cabal.project.freeze`
from source directory, leading to different build plans when building
components with nix and when building project with `cabal new-build`
inside `nix-shell`.
This behavior is undesired, so this commits fixes it.
* Add tests for extra-hackages functionality
Co-authored-by: Moritz Angermann <moritz.angermann@gmail.com>
This overlay helps accessing common executable components.
Typically we want to make these available in a nix-shell
created with shellFor. In most cases the package name
will be the same as the executable, but we have a
`toolPackageName` mapping to help when it is not.
```
# To get a single tool:
haskell-nix.tool "cabal" "3.2.0.0"
# This does the same thing as:
(haskell-nix.hackage-package {
name = "cabal-install"
version = "3.2.0.0"
}).components.exes.cabal
# To get an attr set containing multiple tools:
haskell-nix.tools { cabal = "3.2.0.0"; hlint = "2.2.11"; }
# To add tools to a shell:
shellFor { tools = { cabal = "3.2.0.0"; hlint = "2.2.11"; }; }
```
When used in shellFor the tools will be compiled with the same version
of ghc used in the shell (the build ghc in the case of cross
compilation).
Instead of a version string we can use an attr set containing
arguments that will be passed to `cabalProject`.
For instance to specify the ghc used to compile. Use:
```
haskell-nix.tool "cabal" {
version = "3.2.0.0";
ghc = haskell-nix.compiler.ghc883;
}
```
* Downloads using `builtins.fetchGit` when no sha is found.
* Adds a `lookupSha256` arg for when we can't change `cabal.project`.
* Outputs a message when the sha256 is missing that includes advice on how add one.
* Include `tests` in path for cross compiled tests
* Add ghc-boot-packages-nix to haskellNixRoots
* Fix naming warnings in tests sources
* Uses a single cleanGit call for all the tests
* Add memoize code for ghc-boot and ghc-extra pkgs
* Memoize ghc-boot and ghc-extra pkgs nix
* Fix release.nix being more strict than ci.nix by updating filterAttrsOnlyRecursive in ci-lib.nix
* Nicer errors when materialized path does not exist
* Updated materialization docs
* Add internalHackageIndexState to set the index-state used within haskell.nix
This fires at least 3 times for every haskell.nix project (bootstrap
happy, alex, and hscolor), and then again for every `cabalProject` and
`hackage-package`. Moreover, it isn't actually informative: it just says
`ghc` and `ghc-pkg` for the executables being used.
When no name is provided to `cleanSourceWith` or `cleanGit` we
currently use `baseNameOf src` as a default.
This was cute, but it lead to cache misses. For instance if
`x = cleanSourceWith { src = ./.; }` then `baseName src`
will be different when `src` resolves to "/nix/store/X"
than when it is in a local directory. If people use
git worktrees they also may wind up with different
values for `name`. Anything that depends on `x.name` will
propagate the issue.
This change uses `subDir` if present or if it is not it encourages
adding a suitable `name` with a warning message.
* Enables tests for Musl and AArch64 (also rasbery pi, but leaves them disabled for now) and includes many fixes for the tests that were broken.
* Makes Musl cross compiler more like native (including making `ghci` and `runghc` work)
* Updates selection of enableShared
* Works around missing configFiles attribute issue
* Use ghc-extra-packages and compiler.nix-name to pick the right extra packages
* Leaves out --cross-compile on hsc2hs for musl
* Fixes haddock and hoogle for musl and disables them for other cross compilers
* Adds ghc 8.8.3
* Static link components (except libraries and `all`) on musl
* Use qemu for the arm cross compile `testWrapper`
* Add isCrossHost and isCrossTarget functions
* Fixes profiling on AArch64
* Disable split sections when compiling for windows (fixes GHC 8.8 cross compile to windows)
* Disable hoogle in cross compiler shells for now
Fixes#457.
- Components that don't have a library component will no longer break
`collectComponents "library"`.
- Avoid having lots of empty attribute sets.
- Don't set `recurseForDerivations` excessively.
Repos that have submodules with submodules will have a `.git` file
that references the relative location of the submodules dir in the
`.git/modules`. Without this file the contents of submodules of
submodules are excluded from the `cleanGit` output.
If the .nix files listed in the cache are full paths pointing at the
store we get errors like this in restricted eval mode:
error: access to path '/nix/store/...-cabal-simple.nix' is forbidden
in restricted mode
This fixes and issue with TH functions like `makeRelativeToProject`
that can be used to keep a reference to source files. If they are in
a temporary location that can result in surprising `file not found`
errors at run time.
It may also be useful for making debug symbols and the like refer to
real files.
There is a risk that keeping the source for all packages could expand
closure sizes (if for instance debug info is wanted, but you do not
want to keep the source in the store). For that reason we have
made this an option that must be enabled.
This change adds a `materialized` argument to the functions that
generate nix files with `plan-to-nix` and `stack-to-nix`.
* Provides `builtins.trace` messages with pointers on how to do so.
* Includes a `checkMaterialized` option that can be used
to verify the materialized files.
* Documents how to script updates with `plan-nix` or `stack-nix`.
* Outputs warnings if materialized files are used without hash
(or without an `index-state` for cabal projects).
* Provides materialized nix files for `alex`, `happy` and `hscolour`
Because we `meta.platforms` was not set in `nix-tools/default.nix` we
were not testing or caching `nix-tools` for macOS. As a result
we missed that there was a need for the ranlib fix on `macOS`.
This enables hydra builds of nix-tools on all platforms and fixes
the resulting issue with `ranlib` in the `old-ghc-nix` version
of ghc 8.4.4.
Also enables hydra builds of the plan-nix to make sure dependencies
of plan-nix are cached for darwin. And fixes the default `nix-tools`
arg for `hackageTarball`.
This change replaces `passthru.run` and `collectRunComponents` with
`haskellLib.check` and `$pkg.checks`.
Uses `ln -s` instead of `cp` for DLLs to avoid duplicating them.
Adds features to help debugging eval issues with haskell.nix/release.nix.
The ifdInputLevel can be set to limit the jobs included to control
how much work is done at the eval stage when looking for issues.
Reduce CI times by excluding older ghc versions from haskellNixRoots
(GHC 8.6.3 and 8.6.4)
Sets `preferLocalBuild=false` on `callCabalProjectToNix` and `callStackToNix`
to ensure they are cached on hydra for both Linux and macOS.
Fix `data-dir:` when cleaning components.
`normalizeRelativeDir` adds a slash on to the end of `dataDir`.
Adding another one here results in `//` and files are left out by
mistake.
Changes to the haskell.nix code to fix broken tests:
* Add missing `then` to `call-cabal-project-to-nix.nix`.
* Fix default `hsSourceDirs` so that `.` gets included for `.all` component if one of the components does not have a `hsSourceDir` set.
* Fix `haskellNixRoots` so it works when cross compiling to windows.
Improvements to the haskell.nix tests:
* Run haskell.nix tests of nixpkgs 19.03 and 19.09.
* Run haskell.nix tests cross compiled to Windows under Wine (when possible).
* Add nix used as IFD inputs as tests to ensure they are cached.
* Use `haskell-nix.cabal-install` instead of `nixpkgs.cabal-install` in tests.
When a project's source directory is empty (i.e. no files have been
added to the project's corresponding git repo), a more friendly error
message will now be printed
* cleanSourceHaskell: Add more filters
* cleanSourceHaskell: Add a "name" parameter, split away the filter
* cleanSourceHaskell: Make it compatible with nixpkgs-19.03
* Adjust usage of cleanSourceHaskell in nix-tools
It is easy to mistakenly pass in a `haskell.compiler.ghcXXX` compiler
instead of a `haskell-nix.compiler.ghcXXX` one. This leads to subtle
bugs (for instance missing Cabal library patch).
When we are not in a worktree the submodule info is in the
`.git` dirs scattered throughout the repo itself. In that situation
we need to copy the `.git` dirs into the temp location we are using
to run `git ls-files`.
We added a setup component to allow overriding of the config used to
to build Setup.hs. The code that deals with library components
was misidentifying this component as a library.
Currently when a dependency of cabal-install or plan-to-nix changes
the cabal new-configure and plan-to-nix have to be rebuilt in order
to recalculate the plan-nix for bootstrap packages alex, happy and
hscolour. We have pinned the inputs such as hackage index-state
so that the plan generated should be the same. Unfortunately it can
still result in a long wait.
Haskell.nix does cache the result via haskellNixRoots when built on
iohk's hydra server. But if you use a different version of nixpkgs
it is unlikely to work. If you have access to a CI that populates a
nix cache you can add haskellNixRoots to your CI build, but if not
you are kind of stuck.
One option to help with this would be to include the plan-to-nix output
in the haskell.nix git repository. But intermediate files in git is
not nice.
Instead this PR adds just the hashes of the output to the derivations
that runs plan-to-nix. Since the inputs are fixed or should have
no affect on the result (GCC version for instance), we should
be able to rely on them remaining constant.
* Pin the index-state used by ghc dependencies
Currently when a new hackage index-state is available (every day on
haskell.nix/master branch) the index-state used alex, happy and hscolour
is in the bootPkgs is updates to that latest one. This results in a
rebuild of GHC (even though the build plans are likely unchanged).
By pinning the index state for these bootPkgs nix will be able to
reuse GHC. The only down side is that if we need to get new versions
of alex, happy and hscolour into bootPkgs we will likely need to bump
the index-state attributes as well as the version.
The Overlays branch
This is a major reorganization in how haskell.nix is used, and marks our 1.0 release.
The branch doesn't build due to numerous issues that we believe to be with the CI
and not this branch. We expect only very minor adjustments prior to calling this the
official 1.0 release.
* Move iohk-nix patches into haskell.nix
This moves the customizations we did in iohk-nix
into haskell.nix via overlays and config.
Add bootPkgs logic
this moves the nuking of bootPkgs where it belongs. This should eventually still be
removed and replaced by a proper solution, that doesn't require the nuking of
bootPkgs.
Allow us to bootstrap binary ghcs and a cabal-install
With this we can do the following:
```
$ nix repl compiler/old-ghc-nix
nix-repl> :b (let pkgs = import <nixpkgs> {}; in with import ./. {}; import ./compiler/bootstrap/cabal-install.nix { inherit (pkgs) fetchurl stdenv zlib; inherit hackage
; ghc = ghc844; src = pkgs.fetchurl { url = "https://github.com/haskell/cabal/archive/Cabal-v3.0.0.0-rc3.tar.gz"; sha256 = "1zl2mgg8307ykq3v8nmafc6zdhhj1cw7w8ffpap16dsm6
5lbnx33"; }; })
```
which wile it may look daunting, will allow us to bootstrap a
cabal-install with a ghc. From that point onwards, we should be able to
build any hackage package via haskell.nix.
Pass through cabal-install version
Better threading of arguments.
Add bootstrap overlay
Allow alex + happy to be built
This still has a wart: we need nix-tools, and for that we use the ghc865 from
nixpkgs. Ideally we'd build nix-tools against a specific ghc, but then
we'd need a build expression for that.
Make ghcjs work
Building something like this:
```
nix build '(with import ./. { nixpkgs = ../nixpkgs; nixpkgsArgs = { crossSystem = { config = "js-unknown-ghcjs"; }; }; }; (haskell-nix.hackage-package { name = "hello"; version = "1.0.0.2"; })).components.exes.hello'
```
will now work. Assuming `nixpkgs` has been appropriately patched to support the `js-unknown-ghcjs` triple.
Also: this will need an additional `Cabal` patch, to make `Cabal` understand what it needs to do with: `dist/build/hello/hello: copyFile: does not exist (No such file or directory)`
It needs to learn that `dist/build/hello/hello.jsexe` is what it wants to copy and that that is a directory. Luckily we do have some code in Cabal already that does this for `.exe` on windows.
Build `js-unknown-ghcjs` packages with haskell.nix
Using the following expression:
```
with import ./. { nixpkgs = ../nixpkgs; nixpkgsArgs = { crossSystem = { config = "js-unknown-ghcjs"; }; }; };
let Cabal = buildPackages.haskell-nix.hackage-package {
name = "Cabal"; version = "2.4.1.0";
modules = [
{ packages.Cabal.patches = [ ./Cabal-install-folder.diff ]; }
];
}; in
(haskell-nix.hackage-package {
name = "hello"; version = "1.0.0.2";
modules = [
({config, ... }:{ packages.hello.package.setup-depends = [ Cabal ]; })
];}).components.exes.hello
```
in a `test.nix` file. And running
```
nix build -f ./test.nix
```
on it, will produce
```
./result
├── bin
│ └── hello.jsexe
│ ├── all.js
│ ├── all.js.externs
│ ├── index.html
│ ├── lib.js
│ ├── manifest.webapp
│ ├── out.frefs.js
│ ├── out.frefs.json
│ ├── out.js
│ ├── out.stats
│ ├── rts.js
│ └── runmain.js
└── share
└── doc
└── x86_64-linux-ghc-8.6.5
└── hello-1.0.0.2
└── LICENSE
6 directories, 12 files
```
Uses the `isLocal` for property set by `plan-to-nix` and `stack-to-nix`
on packages where the source is referenced in the `cabal.project`
or `stack.yaml` file.
* Filter components using buildable flag
* Bump nix-tools
* Include components without buildable flag
* Test buildable components are filtered correctly
When trying to build haskell packages from local source using nix (
source code you have used `git clone` to retrieve and have modified)
it is often annoying that small changes can trigger a lot of components
to be built unnecessarily. For instance if you change the code for
a test in a package it will often trigger builds of the library and
all its dependencies.
To avoid this problem we could manually set up `cleanSourceWith` calls
in nix to filter out files that are not need, but to do this every
component would be difficult and error prone.
This change automates this process using the information in the
packages `.cabal` file to filter the source of each component
automatically when using `callCabalPlanToNix` (we should be able to
add it to `callStackToNix` too).
## How does it work?
* A new version on `cleanSourceWith` is used that adds a `subDir`
argument. It allows descending into a subdirectory to compose with
cleaning the source.
* A new `--full` argument is passed to `plan-to-nix` by
`callCabalPlanToNix` telling it to output extra information about
the cabal packages in the plan (directories and filenames).
* `callCabalProjectToNix` uses the new `subDir` argument of
`cleanSourceWith` when setting the packageSrc.
* `comp-builder.nix` and `setup-builder.nix` use a new
`cleanCabalComponent` function to further clean the packageSrc
using the new information in the `package` and `component`.
## hpack wildcards
`package.yaml` files can include wild cards and these are not handled
yet (they do not map to anything in the `.cabal` file and so are lost).
They will need to be added manually to the appropriate field
(eg. `packages.xyz.package.dataFiles`)
If your hpack file does not contain wildcards or if you are willing to
add the wildcards entries manually in nix it is still useful to
enable cleaning on hpack.
For example:
```
packages.ghc-toolkit.package.cleanHpack = true;
packages.ghc-toolkit.components.library.extraSrcFiles = [
"genapply/**/**"
"boot-libs/**/**"
"ghc-libdir/**/**"
];
```
* Add sha256 to source-repository-packages for hydra
This allows source-repository-package to be downloaded and replaced in
with `package: /nix/store/...` by callCabalPackageToNix so that `cabal`
and `plan-to-nix` do not need to download them.
* Adds `stackage-lts-set` and `hackage-package`.
This allows us to build a full stackage-lts-set, or a package from
hackage.
```
nix build '(with import ./. {}; (hackage-package { name = "lens"; version = "4.17.1"; })).components.library'
```
would build the lens library.
```
nix build '(with import ./. {}; (stackage-lts-set "lts-13.28")).lens.components.library'
```
would build the lens library component from the lts-13.28 stackage set.
* Drop lts-stackage-set
* Adds some docs
* Cross comp. & callCabalProjectToNix/callStackToNix
This fixes an issue where the these IFD functions fail because they try
to build and run a cross compiled nix-tools on the build system.
It may also help if there are cross compilation issues with
setup-depends (they also access the buildPackages).
* Align stack-to-nix ifd with plan-to-nix ifd
This also fixes the restricted/pure eval issues with callStackToNix.
I've also made the tests run in restricted mode to catch these kinds of
issues.
* Add a custom error message for a missing stack resolver
* Make mkCabalProjectPkgSet robust to missing Hackage patches
* Add error for component prefix lookup
* Add a custom error for a missing GHC
* Run the nightly scripts to also update the index-state-hashes
This moves the index-state-hashes to `hackage.nix` as well. Thus
we need to have a least one hackage.nix update in.
* fix import
* Update update-index-state-hashes.nix
Add comments.
* Fix eval on Hydra
* Bump nixpkgs to latest 19.03
* Change unit tests into a derivation
Prevents Hydra from complaining about them.
* Add unpacked source pins to Hydra jobset
So that they are cached.
* Fix callCabalProjectToNix plans with source repos
* nix-tools: wrap executables to provide dependencies
* cabalProjectToNix needs git for cabal source repos
When resolving source repos the `cabal` executable uses `git` from
the `PATH` (not using nix store reference).
* Fix callCabalProjectToNix when using nix sandboxes
I am not sure why this works on hydra, but on default install of nix
on ubuntu it fails to rename the plan-and-src $out because it is made
read only by rsync.
* Add comment regarding the nix sanbox issue
