all-hies/Design.org

Assumptions:

• The stack build does not depend on which nixpkgs version it uses, it only needs a matching ghc version

~/.cache/all-hies/ layout:

• haskell-ide-engine: HIE git repository
• nixpkgs: nixpkgs git repository
• history-nixpkgs-unstable: nixpkgs-unstable channel history from https://channels.nix.gsc.io/

• per-ghcMinor: Files for specific minor GHC versions

  • <major><minor>: File with the list of base libraries for that ghc version

• per-nixpkgs: Files for specific nixpkgs versions

  • ghcVersions: Which GHC versions a nixpkgs version has

    • <nixpkgsrev>: List of GHC versions nixpkgs revision <nixpkgsrev> has

  • sha256: The sha256 of the nixpkgs version tarball

    • <nixpkgsrev>: sha256

• per-hie: Files for specific hie versions

  • <hie-rev>: Files for hie version <hie-rev>

    • <ghcVersion-stack2nix>.nix: File generated by stack2nix for <hie-rev> and its stack file for <ghcVersion>

./ (repository) layout:

• generated: Files that can be deleted without worry, they'll be regenerated, if possible from the caches in ~/.cache/all-hies

  • stack2nix: Files generated by stack2nix

    • revision: HIE revision (tag if possible) for the generated files
    • <ghcVersion>.nix: stack2nix generated expression for HIE's <ghcVersion> stack file

  • nixpkgsHashes

    • <nixpkgsrev>: File containing the sha256 hash for nixpkgs version <nixpkgsrev>. This is to allow the Nix builds to download these versions.

  • ghcBaseLibraries:

    • <ghcVersion>: List of base libraries for that ghc version, used by Nix to set all of those to null. Hack around https://github.com/input-output-hk/stack2nix/issues/134

• nixpkgsForGhc

  • <ghcVersion>: File containing the nixpkgs rev to use for that GHC version. This way we can pin nixpkgs versions and don't need to recompile a lot when we change nixpkgs (because we never change a nixpkgs for a ghc version). This is checked in because generating this isn't deterministic.

• overrides:

  • <ghcVersion>.nix: Haskell overrides for a specific GHC version, in order to make the build succeed. (Note: So far it doesn't seem that HIE version-specific overrides are needed, because of stack2nix).

Process:

• Update/initialize hie repo
• Update/initialize nixpkgs repo
• Remove ./generated
• Get latest release (latest tag) and revision
• Write tag to ./generated/stack2nix/revision
• Checking out that hie version

• For each stack file for a ghc version

  • Find a suitable nixpkgs revision for that ghc version

    • If ./nixpkgsForGhc/<ghcVersion> present, use that version

    • else

      • fetch nixpkgs-unstable history (or use cached version if <15 minutes ago)

      • for each channel nixpkgs revision (from newest to oldest)

        • Get ghc versions

          • If $CACHE/per-nixpkgs/ghcVersions/<revision> exists use that

          • Otherwise

            • Check out that nixpkgs version
            • Run nix evaluation to determine available compilers
            • Save result in $CACHE/per-nixpkgs/ghcVersions/<revision>

        • If it has the ghc version we need

          • Save that revision in ./nixpkgsForGhc/<ghcVersion>
          • revision found, exit channel history loop
        • Otherwise continue
      • If no fitting ghc version was found, log error

  • Calculate the sha256 for that nixpkgs version and cache it in $CACHE/per-nixpkgs/sha256/<revision>

    • Copy to ./generated/nixpkgsHashes

  • Generate the base libraries for that version into ./generated/ghcBaseLibraries

    • Check out the revision and run a nix built ghc binary to determine the list
    • Cache it into $CACHE/per-nixpkgs/per-ghcMinor/$major$minor

  • If ./generated/stack2nix/<ghcVersion> doesn't exist yet

    • Checking out that nixpkgs revision
    • Invoke stack2nix to generate $CACHE/per-hie/<rev>/<ghcVersion>.nix if it doesn't exist yet
    • Copy $CACHE/per-hie/<rev>/<ghcVersion>.nix to ./generated/stack2nix/<ghcVersion>.nix