* Include dependencies of haskell.nix that were tracked in `nix/sources.json`
as flake inputs (`flake.lock` replaces `nix/sources.json`).
* Use `flake-compat` to continue to provide a compatible interface for non
flake projects.
Co-authored-by: Alexander Bantyev <balsoft@balsoft.ru>
Co-authored-by: Shea Levy <shea@shealevy.com>
* Remove flake specific sourceOverride that mess-up caching.
* Reduce evalPackages discrepency between flake/non-flake.
* Remove dependency on <nixpkgs> when importing sources.nix
The `hix` tools are wrappers for the various `nix` tools that
use `haskell.nix` without the need to add any `.nix` files.
This is useful for:
* A quick way to try out haskell.nix for new users.
* Using haskell.nix to work on projects that do not have
`.nix` files.
* Testing to see if `haskell.nix` can build a project.
* Making `flake` and `non flake` configurations to check `haskell.nix`
treats them the same.
The configuration arguments for `Hix` can be (from highest precedence to lowest):
* Passed on the command line with `--arg` (or `--argstr` for string args).
* Placed in `nix/hix.nix` file in the project dir.
* Placed in `~/.config/hix/hix.conf`
Boilerplate `default.nix`, `shell.nix` and `flake.nix` files can be added to a
a project with a `nix/hix.nix` file to make it work with the standard `Nix`
tools.
Changes to the interface of haskell.nix (from the changelog.md file):
* Removed `sources.nixpkgs-default`, use `sources.nixpkgs` instead.
* Removed `./nixpkgs` directory, use `(import ./. {}).sources`
or `./nix/sources.nix` instead.
* Removes V1 interface for details on how to fix old code see:
https://github.com/input-output-hk/haskell.nix/issues/709
* Removed defaultCompilerNixName.
* cabalProject, cabalProject', hackage-project and hackage-package
now require a `compiler-nix-name` argument.
* `haskell-nix.tool` and `.tools` now require a `compiler-nix-name` argument.
New functions `p.tool` and `p.tools` (where p is a project) do not.
Like `shellFor { tools = ... }` they will use the compiler nix name
from the project (including stack projects where it is derived from
the resolver).
* `haskell-nix.alex` and `haskell-nix.happy` have been removed. Use
`p.tool "alex" "3.2.5"` or `shellFor { tools = { alex = "3.2.5"; } }`.
* `haskell-nix.nix-tools` -> `haskell-nix.nix-tools.ghc883` (it includes
the hpack exe now).
* `haskell-nix.cabal-install` ->
`p.tool "cabal" "3.2.0.0"` or `shellFor { tools = { cabal = "3.2.0.0"; } }`
* `haskell-nix.haskellNixRoots` -> `haskell-nix.roots ghc883` or `p.roots`
Other changes:
Adds hpack executable to the nix-tools derivations.
Adds a `cabal-hpack` test to make sure `hpack` works with
`cabalProject`.
Reduces the number of calls to `cabalProject` (particularly when
checking materialization), by giving internal tools a per-compiler
attribute.
Uses happy 1.19.12 when building newer ghc versions.
Updates cabal-install 3.2.0.0 to use the source from github that
is compatible with ghc 8.10.1.
Updates the docs for callCabalProjectToNix.
Adds a license mapping to fix a common warning.
This was a nice idea, but in practice it did not work unless you
were explicitly extracting just `config`, `overlays` and `system`
(see #542).
```
~/iohk/haskell.nix$ nix-build -E 'import <nixpkgs> (import ./.)'
error: anonymous function at /nix/store/f7mwvx7gl8lwjiqmpv60xapc3crvc474-nixpkgs-20.03pre194824.1f56d679f40/nixpkgs/pkgs/top-level/default.nix:20:1 called with unexpected argument '__functor', at /nix/store/f7mwvx7gl8lwjiqmpv60xapc3crvc474-nixpkgs-20.03pre194824.1f56d679f40/nixpkgs/pkgs/top-level/impure.nix:84:1
```
This change will replace #542 errors with #709 errors.
Although the default ghc used by haskell.nix is ghc 8.6.5 many of
the tools used in haskell.nix are still built with the boot compiler
ghc 8.4.4. These include
* haskell-nix.cabal-install
* haskell-nix.alex
* haskell-nix.happy
This change updates those to ghc 8.6.5 and includes materializations
for the new versions.
When cabal-install is built it is careful to disable materialization
checks on the version of itself used during the build to avoid
infinite recursion.
There was a version of nix-tools built with the boot ghc which was
only used when `checkMaterialization = true`. It was used for
the boot versions of alex, happy and hscolour. These have been update
to use the default (ghc 8.6.5) version of nix-tools and
checkMaterialization is forced off when they are being used to build
ghc. This means the materialization will only be checked for these
when they are built independently (they are included in the test set
via haskellNixRoots).
Three new arguments are added to `default.nix`:
* `defaultCompilerNixName` if not specified "ghc865" is used
* `checkMaterialization` makes it easier to switch on materialization checks
* `system` defaults to `builtins.currentSystem`
This change also moves the work needed for hydra eval to the eval
system using a new `evalPackages` feature. This includes:
* Fetching from git with `fetchgit`
* Building scripts with `runCommand` and `writeTextFile`
* `git ls-files` in `cleanGit`
* running `cabal v2-configure`
* copying materialized files (we are not sure why this is necessary but
if we do not `cp -r` the files nix will not load them on hydra)
Reduce size of `make-config-files.nix` strings by around 80%.
These are unlikely to be the cause of hydra eval time memory
issues in the GB range, but were still quite large (around 10MB for the
`cabal-simple` test case).
There was issue causing excessive builds of the `git` package when
cross compiling. Gory details are a comment in `lib/defaults.nix`
but in short if you use `git` you need an extra `.buildPackages`
one is not enough because it depends on `gdb` and that will
be different in `buildPackages` compared to
`buildPackages.buildPackages`.
Adds missing materialization files for ghc 8.4.4 (only needed
when `checkMaterialization` is on because of other
materialiazations, but good to have).
Currently if we want to use nixpkgs-default and nixpkgsAgs it is
a bit painful. With this change:
```
nix-build -E 'let h = import ./. {}; in (import h.sources.nixpkgs-default h.nixpkgsArgs).haskell-nix.tool "hlint" "3.1"'
```
becomes
```
nix-build -E 'pkgs.haskell-nix.tool "hlint" "3.1"'
```
* Update nixpkgs pins to include 20.03, wobble
- Add 20.03
- Remove 19.03 (and old 18.09 file)
- Update 19.09 to latest
- Update github pin to 20.03
- Remove workarounds for 19.03
* Mark openssl as allowedInsecure
* Force python3 for mkdocs
* Remove unused (and broken) test files
* Make default.nix expose pinned nixpkgs
Also make `nixpkgs/default.nix` just an attribute set of sources.
This is simpler, and consistent with what e.g. `niv` does.
* Add backwards compatibility shim, and version argument to allow evolution in future
* Fix some missed things
* Fix and improve quickstart
The Overlays branch
This is a major reorganization in how haskell.nix is used, and marks our 1.0 release.
The branch doesn't build due to numerous issues that we believe to be with the CI
and not this branch. We expect only very minor adjustments prior to calling this the
official 1.0 release.
* Move iohk-nix patches into haskell.nix
This moves the customizations we did in iohk-nix
into haskell.nix via overlays and config.
Add bootPkgs logic
this moves the nuking of bootPkgs where it belongs. This should eventually still be
removed and replaced by a proper solution, that doesn't require the nuking of
bootPkgs.
Allow us to bootstrap binary ghcs and a cabal-install
With this we can do the following:
```
$ nix repl compiler/old-ghc-nix
nix-repl> :b (let pkgs = import <nixpkgs> {}; in with import ./. {}; import ./compiler/bootstrap/cabal-install.nix { inherit (pkgs) fetchurl stdenv zlib; inherit hackage
; ghc = ghc844; src = pkgs.fetchurl { url = "https://github.com/haskell/cabal/archive/Cabal-v3.0.0.0-rc3.tar.gz"; sha256 = "1zl2mgg8307ykq3v8nmafc6zdhhj1cw7w8ffpap16dsm6
5lbnx33"; }; })
```
which wile it may look daunting, will allow us to bootstrap a
cabal-install with a ghc. From that point onwards, we should be able to
build any hackage package via haskell.nix.
Pass through cabal-install version
Better threading of arguments.
Add bootstrap overlay
Allow alex + happy to be built
This still has a wart: we need nix-tools, and for that we use the ghc865 from
nixpkgs. Ideally we'd build nix-tools against a specific ghc, but then
we'd need a build expression for that.
Make ghcjs work
Building something like this:
```
nix build '(with import ./. { nixpkgs = ../nixpkgs; nixpkgsArgs = { crossSystem = { config = "js-unknown-ghcjs"; }; }; }; (haskell-nix.hackage-package { name = "hello"; version = "1.0.0.2"; })).components.exes.hello'
```
will now work. Assuming `nixpkgs` has been appropriately patched to support the `js-unknown-ghcjs` triple.
Also: this will need an additional `Cabal` patch, to make `Cabal` understand what it needs to do with: `dist/build/hello/hello: copyFile: does not exist (No such file or directory)`
It needs to learn that `dist/build/hello/hello.jsexe` is what it wants to copy and that that is a directory. Luckily we do have some code in Cabal already that does this for `.exe` on windows.
Build `js-unknown-ghcjs` packages with haskell.nix
Using the following expression:
```
with import ./. { nixpkgs = ../nixpkgs; nixpkgsArgs = { crossSystem = { config = "js-unknown-ghcjs"; }; }; };
let Cabal = buildPackages.haskell-nix.hackage-package {
name = "Cabal"; version = "2.4.1.0";
modules = [
{ packages.Cabal.patches = [ ./Cabal-install-folder.diff ]; }
];
}; in
(haskell-nix.hackage-package {
name = "hello"; version = "1.0.0.2";
modules = [
({config, ... }:{ packages.hello.package.setup-depends = [ Cabal ]; })
];}).components.exes.hello
```
in a `test.nix` file. And running
```
nix build -f ./test.nix
```
on it, will produce
```
./result
├── bin
│ └── hello.jsexe
│ ├── all.js
│ ├── all.js.externs
│ ├── index.html
│ ├── lib.js
│ ├── manifest.webapp
│ ├── out.frefs.js
│ ├── out.frefs.json
│ ├── out.js
│ ├── out.stats
│ ├── rts.js
│ └── runmain.js
└── share
└── doc
└── x86_64-linux-ghc-8.6.5
└── hello-1.0.0.2
└── LICENSE
6 directories, 12 files
```
Project flags (from stack.yaml and plan.json) are exported in a modules attribute by stack-to-nix and plan-to-nix, but are not currently used. This change updates `mkStackPkgSet` and `mkCabalProjectPkgSet` so that the modules attribute is used (if present) and includes tests to check they are.
This commit makes `stdenv.lib.mkOverride` necessary for setting flags in `mkCabalProjectPkgSet` modules, however it also means that you can set them in `cabal.project` instead (see #254).
When trying to build haskell packages from local source using nix (
source code you have used `git clone` to retrieve and have modified)
it is often annoying that small changes can trigger a lot of components
to be built unnecessarily. For instance if you change the code for
a test in a package it will often trigger builds of the library and
all its dependencies.
To avoid this problem we could manually set up `cleanSourceWith` calls
in nix to filter out files that are not need, but to do this every
component would be difficult and error prone.
This change automates this process using the information in the
packages `.cabal` file to filter the source of each component
automatically when using `callCabalPlanToNix` (we should be able to
add it to `callStackToNix` too).
## How does it work?
* A new version on `cleanSourceWith` is used that adds a `subDir`
argument. It allows descending into a subdirectory to compose with
cleaning the source.
* A new `--full` argument is passed to `plan-to-nix` by
`callCabalPlanToNix` telling it to output extra information about
the cabal packages in the plan (directories and filenames).
* `callCabalProjectToNix` uses the new `subDir` argument of
`cleanSourceWith` when setting the packageSrc.
* `comp-builder.nix` and `setup-builder.nix` use a new
`cleanCabalComponent` function to further clean the packageSrc
using the new information in the `package` and `component`.
## hpack wildcards
`package.yaml` files can include wild cards and these are not handled
yet (they do not map to anything in the `.cabal` file and so are lost).
They will need to be added manually to the appropriate field
(eg. `packages.xyz.package.dataFiles`)
If your hpack file does not contain wildcards or if you are willing to
add the wildcards entries manually in nix it is still useful to
enable cleaning on hpack.
For example:
```
packages.ghc-toolkit.package.cleanHpack = true;
packages.ghc-toolkit.components.library.extraSrcFiles = [
"genapply/**/**"
"boot-libs/**/**"
"ghc-libdir/**/**"
];
```
* Better defaults (doExactConfig = true; non-reinstallable lib:ghc)
Disabling haddocks by default has unintended consequences, as such we'll wait for the `.doc` derivation to solve this.
Fixes#201, #191
* Adds `stackage-lts-set` and `hackage-package`.
This allows us to build a full stackage-lts-set, or a package from
hackage.
```
nix build '(with import ./. {}; (hackage-package { name = "lens"; version = "4.17.1"; })).components.library'
```
would build the lens library.
```
nix build '(with import ./. {}; (stackage-lts-set "lts-13.28")).lens.components.library'
```
would build the lens library component from the lts-13.28 stackage set.
* Drop lts-stackage-set
* Adds some docs
* Cross comp. & callCabalProjectToNix/callStackToNix
This fixes an issue where the these IFD functions fail because they try
to build and run a cross compiled nix-tools on the build system.
It may also help if there are cross compilation issues with
setup-depends (they also access the buildPackages).
* Align stack-to-nix ifd with plan-to-nix ifd
This also fixes the restricted/pure eval issues with callStackToNix.
I've also made the tests run in restricted mode to catch these kinds of
issues.
* Add a custom error message for a missing stack resolver
* Make mkCabalProjectPkgSet robust to missing Hackage patches
* Add error for component prefix lookup
* Add a custom error for a missing GHC
* Buildkite: Add a test for nix-tools closure size
The limit is a rather generous 500MB.
* nix-tools: Reduce closure size
1. Run justStackExecutables on hpack to remove dependency on ghc and
libraries.
Ideally we would be using the component builder on hpack.
2. Replace nix-prefetch-scripts with nix-prefetch-git. stack.yaml
supports mercurial repositories but I've never seen it used.
* Run the nightly scripts to also update the index-state-hashes
This moves the index-state-hashes to `hackage.nix` as well. Thus
we need to have a least one hackage.nix update in.
* fix import
* Update update-index-state-hashes.nix
Add comments.
* Fix eval on Hydra
* Bump nixpkgs to latest 19.03
* Change unit tests into a derivation
Prevents Hydra from complaining about them.
* Add unpacked source pins to Hydra jobset
So that they are cached.
* Fix callCabalProjectToNix plans with source repos
* nix-tools: wrap executables to provide dependencies
* cabalProjectToNix needs git for cabal source repos
When resolving source repos the `cabal` executable uses `git` from
the `PATH` (not using nix store reference).
* Add a snapshots attribute
These contain package sets for all of the Stackage snapshots.
Relates to #22
* tests: Add stackage snapshots test
* Add a stackage fix for "No attribute: hsc2hs"
* Add comment to snapshots.nix and fix version comparison function
* Use pkgs.fetchurl to build a fixed output derivation
This should hopefully get us around the restricted limitations.
* Adds asserts
* Fix index-state-hashes file
* Fix index-state-hashes
Also adds a regeneration tool:
```
nix build -f ./. generateHackageIndexStateHashes
cp result lib/index-state-hashes.nix
```
* Fix off-by-one
* better errors
Currently if you override nix-tools using `-I nix-tools` to point to
a local directory that has build output directories like `dist-newstyle`
those directories are hashed and copied to the nix store if they have
changes. This change adds the `cleanHaskellSource` function from
`iohk-nix` and runs it on nix-tools.
* This adds more logic for the cabalProjectToNix idf
That is, something like this:
```
nix-build --expr 'with import ./. {}; callCabalProjectToNix { index-state = "2019-04-30T00:00:00Z"; src = /some/path; }'
```
should produce something that can be build with mkCabalProjectPkgSet.
* Make sure the hackageTarball's store path doesn't change
previously the fetchurl would produce a different store path each
and every time as hackage's index is a moving target. With this
impurity setup, we can ignore this.
* Fix test
* Proper name
* Re-enable test
* Allow to parameterize over the `system` for the test default.nix
* Copy instead of link for ifds.
This makes me really sad.
This would be useful if the project needs to stay with an old version
of haskell.nix but use recent version of hackage.nix/stackage.nix.
Normally latest haskell.nix and latest hackage/stackage are best.
