all-hies/Design.org

Assumptions:
- The stack build does not depend on which nixpkgs version it uses, it only needs a matching ghc version

~/.cache/all-hies/ layout:

- haskell-ide-engine: HIE git repository
- nixpkgs: nixpkgs git repository
- history-nixpkgs-unstable: nixpkgs-unstable channel history from https://channels.nix.gsc.io/
- per-ghcMinor: Files for specific minor GHC versions
  - <major><minor>: File with the list of base libraries for that ghc version
- per-nixpkgs: Files for specific nixpkgs versions
  - ghcVersions: Which GHC versions a nixpkgs version has
    - <nixpkgsrev>: List of GHC versions nixpkgs revision <nixpkgsrev> has
  - sha256: The sha256 of the nixpkgs version tarball
    - <nixpkgsrev>: sha256
- per-hie: Files for specific hie versions
  - <hie-rev>: Files for hie version <hie-rev>
    - <ghcVersion-stack2nix>.nix: File generated by stack2nix for <hie-rev> and its stack file for <ghcVersion>

./ (repository) layout:
- generated: Files that can be deleted without worry, they'll be regenerated, if possible from the caches in ~/.cache/all-hies
  - stack2nix: Files generated by stack2nix
    - revision: HIE revision (tag if possible) for the generated files
    - <ghcVersion>.nix: stack2nix generated expression for HIE's <ghcVersion> stack file
  - nixpkgsHashes
    - <nixpkgsrev>: File containing the sha256 hash for nixpkgs version <nixpkgsrev>. This is to allow the Nix builds to download these versions.
  - ghcBaseLibraries:
    - <ghcVersion>: List of base libraries for that ghc version, used by Nix to set all of those to ~null~. Hack around https://github.com/input-output-hk/stack2nix/issues/134
- nixpkgsForGhc
  - <ghcVersion>: File containing the nixpkgs rev to use for that GHC version. This way we can pin nixpkgs versions and don't need to recompile a lot when we change nixpkgs (because we never change a nixpkgs for a ghc version). This is checked in because generating this isn't deterministic.
- overrides:
  - <ghcVersion>.nix: Haskell overrides for a specific GHC version, in order to make the build succeed. (Note: So far it doesn't seem that HIE version-specific overrides are needed, because of stack2nix).


Process:
- Update/initialize hie repo
- Update/initialize nixpkgs repo
- Remove ./generated
- Get latest release (latest tag) and revision
- Write tag to ./generated/stack2nix/revision
- Checking out that hie version
- For each stack file for a ghc version
  - Find a suitable nixpkgs revision for that ghc version
    - If ./nixpkgsForGhc/<ghcVersion> present, use that version
    - else
      - fetch nixpkgs-unstable history (or use cached version if <15 minutes ago)
      - for each channel nixpkgs revision (from newest to oldest)
        - Get ghc versions
          - If $CACHE/per-nixpkgs/ghcVersions/<revision> exists use that
          - Otherwise
            - Check out that nixpkgs version
            - Run nix evaluation to determine available compilers
            - Save result in $CACHE/per-nixpkgs/ghcVersions/<revision>
        - If it has the ghc version we need
          - Save that revision in ./nixpkgsForGhc/<ghcVersion>
          - revision found, exit channel history loop
        - Otherwise continue
      - If no fitting ghc version was found, log error
  - Calculate the sha256 for that nixpkgs version and cache it in $CACHE/per-nixpkgs/sha256/<revision>
    - Copy to ./generated/nixpkgsHashes
  - Generate the base libraries for that version into ./generated/ghcBaseLibraries
    - Check out the revision and run a nix built ghc binary to determine the list
    - Cache it into $CACHE/per-nixpkgs/per-ghcMinor/$major$minor
  - If ./generated/stack2nix/<ghcVersion> doesn't exist yet
    - Checking out that nixpkgs revision
    - Invoke stack2nix to generate $CACHE/per-hie/<rev>/<ghcVersion>.nix if it doesn't exist yet
    - Copy $CACHE/per-hie/<rev>/<ghcVersion>.nix to ./generated/stack2nix/<ghcVersion>.nix