We still seem to be getting some OOM errors. This is likely to be
because some of the required jobs are still very large (with multiple
cross compilation target platforms). Hopefully breaking these out will
help.
This requires explicitly identifying the things tht don't work and
removing them. I used our own `disabled` attribute, because `broken` is
weird and breaks evaluation, which isn't really what we want.
Cuts the number of jobs from over 2000 to 216. This seems to greatly
speeds up the time it takes small changes (where most stuff remains the
same) to be processed by hydra.
Uses https://github.com/NixOS/hydra/issues/715 to reference the
jobs in required by name avoiding OOM issues.
Co-authored-by: Hamish Mackenzie <Hamish.Mackenzie@iohk.io>
* Update default nixpkgs to 20.09
* Move more CI to work on 20.09 by default
* IFD level 1
* Use nixpkgs 20.03 emscripten (needed for ghcjs)
* Add missing materialized files
* Add missing materialized files
* Fix for ghc 8.8 build on darwin
* Fix for ghcjs 8.8 build on darwin
* Fix hls stack test eval for ghc810220201118
* ifdLevel 2
* Oops: accidentally turned on tests for experimental GHC
* Revert "Fix hls stack test eval for ghc810220201118"
This reverts commit b405cd7ee8bd62214bd2d0a61e0317673b5bf1b5.
* ifdLevel 3
* Disable GHC 8.6.5 windows cross on nixpkgs 20.09
* Update supported GHC doc with a table
* Add emscripten comment
Co-authored-by: Hamish Mackenzie <Hamish.Mackenzie@iohk.io>
Changes to the interface of haskell.nix (from the changelog.md file):
* Removed `sources.nixpkgs-default`, use `sources.nixpkgs` instead.
* Removed `./nixpkgs` directory, use `(import ./. {}).sources`
or `./nix/sources.nix` instead.
* Removes V1 interface for details on how to fix old code see:
https://github.com/input-output-hk/haskell.nix/issues/709
* Removed defaultCompilerNixName.
* cabalProject, cabalProject', hackage-project and hackage-package
now require a `compiler-nix-name` argument.
* `haskell-nix.tool` and `.tools` now require a `compiler-nix-name` argument.
New functions `p.tool` and `p.tools` (where p is a project) do not.
Like `shellFor { tools = ... }` they will use the compiler nix name
from the project (including stack projects where it is derived from
the resolver).
* `haskell-nix.alex` and `haskell-nix.happy` have been removed. Use
`p.tool "alex" "3.2.5"` or `shellFor { tools = { alex = "3.2.5"; } }`.
* `haskell-nix.nix-tools` -> `haskell-nix.nix-tools.ghc883` (it includes
the hpack exe now).
* `haskell-nix.cabal-install` ->
`p.tool "cabal" "3.2.0.0"` or `shellFor { tools = { cabal = "3.2.0.0"; } }`
* `haskell-nix.haskellNixRoots` -> `haskell-nix.roots ghc883` or `p.roots`
Other changes:
Adds hpack executable to the nix-tools derivations.
Adds a `cabal-hpack` test to make sure `hpack` works with
`cabalProject`.
Reduces the number of calls to `cabalProject` (particularly when
checking materialization), by giving internal tools a per-compiler
attribute.
Uses happy 1.19.12 when building newer ghc versions.
Updates cabal-install 3.2.0.0 to use the source from github that
is compatible with ghc 8.10.1.
Updates the docs for callCabalProjectToNix.
Adds a license mapping to fix a common warning.
* ghc 8.10
* Adds/Updates/Removes materialization files
* Disable one-shot-kqueue-on-macos patch on ghc-8.10
* Test ghc883 and ghc8101 but only with nix 20.03
* Various smaller fixes.
This should fix the infinite recursion issue with #654
Co-authored-by: Lennart Spitzner <hexagoxel@hexagoxel.de>
Now that the defaultCompilerNixName is used to choose which GHC to use
for tools we need to update haskellNixRoots to make sure the common
tools wind up in the cache.
To do this we add the compilerNixName as a dimension in ci.nix.
This has the added bonus of ensuring the tests a run with the
supported versions of ghc (not just the default one).
haskellNixRoots is also updated to include the tools built with the
defaultCompilerNixName (in particular cabal-install was not pinned
and this was the likely cause of significant builds due to cache
misses).
* Update nixpkgs pins to include 20.03, wobble
- Add 20.03
- Remove 19.03 (and old 18.09 file)
- Update 19.09 to latest
- Update github pin to 20.03
- Remove workarounds for 19.03
* Mark openssl as allowedInsecure
* Force python3 for mkdocs
* Remove unused (and broken) test files
* Make default.nix expose pinned nixpkgs
Also make `nixpkgs/default.nix` just an attribute set of sources.
This is simpler, and consistent with what e.g. `niv` does.
* Add backwards compatibility shim, and version argument to allow evolution in future
* Fix some missed things
* Fix and improve quickstart
* Define release.nix in terms of ci.nix
Beef up ci.nix to handle the same things as release.nix, then implement
the one in terms of the other.
* Fix infinite loop in evaluating release.nix with high ifdLevel
* Add tests to cross-compile
* Remove obsolete fix
* Do generic platform filtering
* Undefined variable
* Extract ci-lib.nix
* More undefined variables
* Use filterAttrsOnlyRecursive to avoid forcing all the drv attributes
* Set ifdLevel 0 to see if it fixes hydra eval
* Turn off tests on aarch64 cross
* Ifd level 1
* Ifd level 2
* Ifd level 3
* Maybe it's required that's too big?
Co-authored-by: Hamish Mackenzie <Hamish.Mackenzie@iohk.io>
* Enables tests for Musl and AArch64 (also rasbery pi, but leaves them disabled for now) and includes many fixes for the tests that were broken.
* Makes Musl cross compiler more like native (including making `ghci` and `runghc` work)
* Updates selection of enableShared
* Works around missing configFiles attribute issue
* Use ghc-extra-packages and compiler.nix-name to pick the right extra packages
* Leaves out --cross-compile on hsc2hs for musl
* Fixes haddock and hoogle for musl and disables them for other cross compilers
* Adds ghc 8.8.3
* Static link components (except libraries and `all`) on musl
* Use qemu for the arm cross compile `testWrapper`
* Add isCrossHost and isCrossTarget functions
* Fixes profiling on AArch64
* Disable split sections when compiling for windows (fixes GHC 8.8 cross compile to windows)
* Disable hoogle in cross compiler shells for now
This change replaces `passthru.run` and `collectRunComponents` with
`haskellLib.check` and `$pkg.checks`.
Uses `ln -s` instead of `cp` for DLLs to avoid duplicating them.
Adds features to help debugging eval issues with haskell.nix/release.nix.
The ifdInputLevel can be set to limit the jobs included to control
how much work is done at the eval stage when looking for issues.
Reduce CI times by excluding older ghc versions from haskellNixRoots
(GHC 8.6.3 and 8.6.4)
Sets `preferLocalBuild=false` on `callCabalProjectToNix` and `callStackToNix`
to ensure they are cached on hydra for both Linux and macOS.
Fix `data-dir:` when cleaning components.
`normalizeRelativeDir` adds a slash on to the end of `dataDir`.
Adding another one here results in `//` and files are left out by
mistake.
Changes to the haskell.nix code to fix broken tests:
* Add missing `then` to `call-cabal-project-to-nix.nix`.
* Fix default `hsSourceDirs` so that `.` gets included for `.all` component if one of the components does not have a `hsSourceDir` set.
* Fix `haskellNixRoots` so it works when cross compiling to windows.
Improvements to the haskell.nix tests:
* Run haskell.nix tests of nixpkgs 19.03 and 19.09.
* Run haskell.nix tests cross compiled to Windows under Wine (when possible).
* Add nix used as IFD inputs as tests to ensure they are cached.
* Use `haskell-nix.cabal-install` instead of `nixpkgs.cabal-install` in tests.
Projects like asterius and leksah that include dependencies on ghc. Sometimes the are needed by a setup.hs file (as is the case for the binaryen package used by asterius). The recent overlays merge broke this and #279 was needed to allow reinstallableLibGhc to be used to fix it. This PR adds a test case to make sure it is not inadvertently broken again. Also gives us a place to try out better ways to make this type of package work.* Test setup deps issue depending on ghc and Cabal
* Uses nixpkgs 19.09 + macOS wine fix on hydra/buildkite
* Enables macOS hydra tests
* disables musl tests for now
* Adds haskellNixRoots to tests
* Uses reinstallableLibGhc to fix the new setup-depends test
* Adds meta.platforms to haskell.nix components
* Set meta.platforms on cabal-install
* Builds some maintainer-scripts on hydra
* Pins hackage index-state for ghc-extra-packages
* Fix eval on Hydra
* Bump nixpkgs to latest 19.03
* Change unit tests into a derivation
Prevents Hydra from complaining about them.
* Add unpacked source pins to Hydra jobset
So that they are cached.
