foliage

A hash-friendly Haskell Package Repository.

Foliage is a tool to create custom or private Haskell package repositories, in a fully reproducible way.

Main idea

Like GitHub Pages but for Haskell Packages

A "Hackage repository" is collection of source distributions and cabal files. In addition, Hackage has implemented The Update Framework (TUF) and the repository also includes cryptographic metadata (public keys and signatures).

These files are commonly served by Hackage proper, that is the central deployment of hackage-server.

Foliage explores the idea of creating and serving this content as a static website, generated programmatically a deterministically from textual input files.

Use cases

Company internal hackage

Company XYZ has developed many Haskell packages, some of which are forks of public Haskell libraries. For various reasons XYZ might not be inclined in publishing their packages to Hackage. If XYZ is using multiple repositories for version control, keeping all the company's packages in sync will become a problem.

Currently XYZ's best option is to use cabal.project files. Each cabal package can declare its non-Hackage dependencies using the source-repository-package stanza.

Note: packages can be grouped together into a cabal project, in which case source-repository-package stanzas become the project dependencies; this distintion is inconsequential to our example.

E.g. if packageA needs packageB, hosted on GitHub; packageA's cabal.project will include:

source-repository-package
    type: git
    location: https://github.com/Company-XYZ/packageB
    tag: e70cf0c171c9a586b62b3f75d72f1591e4e6aaa1

While the presence of a git tag makes this quite reproducible; a problem arises in that these dependencies are not transitive. Without any versioning to help, one has to manually pick a working set of dependencies.

E.g. if packageC depends on packageA, packageC cabal.project will have to include something like:

-- Direct dependency
source-repository-package
    type: git
    location: https://github.com/Company-XYZ/packageA
    tag: e76fdc753e660dfa615af6c8b6a2ad9ddf6afe70

-- From packageA
source-repository-package
    type: git
    location: https://github.com/Company-XYZ/packageB
    tag: e70cf0c171c9a586b62b3f75d72f1591e4e6aaa1

Having an internal company Hackage, solves the above problem by reintroducing versioning and a familiar workflow for handling Hackage dependencies; while maintaining absolute control and flexibility over versioning policies and dependency management.

When the team behind packageA wants to push out a new version, say version 1.2.3.4, all they have to do is to update the foliage repository with a file packageA/1.2.3.4/meta.toml with content:

timestamp = 2022-03-29T06:19:50+00:00
url = 'https://github.com/Company-XYZ/packageA/tarball/e76fdc753e660dfa615af6c8b6a2ad9ddf6afe70'

Note: Any other url would work here. E.g. one could use GitHub releases: https://github.com/Company-XYZ/packageA/archive/refs/tags/v1.2.3.4.tar.gz.

ℹ️ Although the timestamp field in the package source metadata is optional, it is highly reccomended if you intend your repository users to be able to use cabal's index-state functionality. Adding a timestamp every time you add a package version ensures the newly created index is "compatible" with what the users have already fetched.

Quickstart

Adding one package

It's reccomended to create a working directory first.

Let's add a package (say typed-protocols-0.1.0.0 from https://github.com/input-output-hk/ouroboros-network/tree/master/typed-protocols, at commit hash fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa).

$ mkdir -p _sources/typed-protocols/0.1.0.0
$ cat _sources/typed-protocols/0.1.0.0/meta.toml
url = 'https://github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa'
subdir = 'typed-protocols'

Building the repository

Run foliage build.

$ foliage build
🌿 Foliage
🗝️ You don't seem to have created a set of TUF keys. I will create one in _keys
🕐 Current time set to 2022-03-30T04:02:56Z.
You can set a fixed time using the --current-time option

🕐 Expiry time set to 2023-03-30T04:02:56Z (a year from now).
🐢 Downloading https://github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa
# curl (for _cache/aHR0cHM6Ly9naXRodWIuY29tL2lucHV0LW91dHB1dC1oay9vdXJvYm9yb3MtbmV0d29yay90YXJiYWxsL2ZhMTBjYjRlZWYxZTdkM2UwOTVjZWMzYzJiYjEyMTA3NzRiN2U1ZmE/.downloaded)
# mv (for _cache/aHR0cHM6Ly9naXRodWIuY29tL2lucHV0LW91dHB1dC1oay9vdXJvYm9yb3MtbmV0d29yay90YXJiYWxsL2ZhMTBjYjRlZWYxZTdkM2UwOTVjZWMzYzJiYjEyMTA3NzRiN2U1ZmE/.downloaded)
👀 https://github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa
⚠️ Deleting cabal project files from _cache/aHR0cHM6Ly9naXRodWIuY29tL2lucHV0LW91dHB1dC1oay9vdXJvYm9yb3MtbmV0d29yay90YXJiYWxsL2ZhMTBjYjRlZWYxZTdkM2UwOTVjZWMzYzJiYjEyMTA3NzRiN2U1ZmE
✅ Written _repo/index/typed-protocols/0.1.0.0/typed-protocols.cabal
✅ Written _repo/root.json
✅ Written _repo/mirrors.json
 Creating source distribution for typed-protocols-0.1.0.0
# cabal (for _repo/package/typed-protocols-0.1.0.0.tar.gz)
# mv (for _repo/package/typed-protocols-0.1.0.0.tar.gz)
✅ Written _repo/package/typed-protocols-0.1.0.0.tar.gz
✅ Written _repo/index/typed-protocols/0.1.0.0/package.json
✅ Written _repo/01-index.tar
✅ Written _repo/01-index.tar.gz
✅ Written _repo/snapshot.json
✅ Written _repo/timestamp.json
💥 All done. The repository is now available in _repo.

If you want to rely on the cabal index-state feature you need to specify a timestamp in the meta.toml file.

E.g.

$ cat _sources/typed-protocols/0.1.0.0/meta.toml
url = 'https://github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa'
subdir = 'typed-protocols'
timestamp = 2022-03-29T06:19:50+00:00

ℹ️ Foliage uses the metadata timestamps to determine the order of the entries in 01-index. This allows you to create an index that

1. can be updated incrementally and 2) can be used with cabal's index-state feature.

With the input above foliage will produce the following:

_repo
├── 01-index.tar
├── 01-index.tar.gz
├── index
│   └── typed-protocols
│       └── 0.1.0.0
│           ├── package.json
│           └── typed-protocols.cabal
├── mirrors.json
├── package
│   └── typed-protocols-0.1.0.0.tar.gz
├── root.json
├── snapshot.json
└── timestamp.json

The content of _repo can be served over HTTP(s) to cabal, e.g. from a cabal.project file.

repository packages.example.org
  url: http://packages.example.org
  secure: True
  root-keys:
    144d97d34d0a86adb1ca7d6bdc1b2d9f0c9123e3c29e3765f5a9eece345ce4f9
    a15f6ae88a26638934d90eff28da29990a4b12c8bb0b2c12f07e9a510e839a97
    fde23c79a14bcbef6ccf198b4ad94ded4092784fcaed17c3d184008e9bf6f722
  key-threshold: 3

Where the root key ids can be obtained from _repo/root.json.

$ jq -r .signatures[].keyid _repo/root.json
144d97d34d0a86adb1ca7d6bdc1b2d9f0c9123e3c29e3765f5a9eece345ce4f9
a15f6ae88a26638934d90eff28da29990a4b12c8bb0b2c12f07e9a510e839a97
fde23c79a14bcbef6ccf198b4ad94ded4092784fcaed17c3d184008e9bf6f722

TUF keys

Foliage creates a set of private keys to sign the TUF metadata at first use. By default, the keys are created in _keys:

_keys/
├── mirrors
│   ├── 105369fb9cb1555cf4517be3e885215a7bc730bd59cf3084ea7140f9692ae847.json
│   ├── 2178cff7b2a3a6edd396371c03bc8fddb77b77b104a9fd97f6291f2c49285946.json
│   └── 4689f8a6d905a59536213a27ea577b34ff3e6d79d5a7b375458d3bb6026a5e13.json
├── root
│   ├── 144d97d34d0a86adb1ca7d6bdc1b2d9f0c9123e3c29e3765f5a9eece345ce4f9.json
│   ├── a15f6ae88a26638934d90eff28da29990a4b12c8bb0b2c12f07e9a510e839a97.json
│   └── fde23c79a14bcbef6ccf198b4ad94ded4092784fcaed17c3d184008e9bf6f722.json
├── snapshot
│   └── 07a918ccdb3ac0600a8b65f3bc96da18dfc5be65c023c64ccd0fb6a04692b64d.json
├── target
│   ├── 793b1b2730db6ec5247934ad0cc7b28ed3b05eae4ffec7e28e7b1739f1cb65b4.json
│   ├── 908041deaae700390dfd7b4c8d3eca7049c8172a205bea19de8314c4fd9ca56f.json
│   └── f102c7035da2a3b5fa82b90d8afe3e8892d13a8c6b7a4281403c340317a35014.json
└── timestamp
    └── 141da8eb2ccba61c2f6bb656b2292970d086770f5bf7d53802d2bc0ec1defa26.json

These keys are small enough you can store them in an environment variable. E.g.

Save:

$ KEYS=$(tar cz -C _keys . | base64)

Restore:

$ mkdir _keys
$ echo "$KEYS" | base64 -d | tar xz -C _keys

⚠️ These are private keys. Don't publish them along with your repository!

Revisions

Foliage supports cabal file revisions. Adding the following snippet to a package's meta.toml, will make foliage look for a cabal file in <package>/<version>/revisions/1.cabal.

[[revisions]]
  number = 1
  timestamp = 2022-03-22T14:15:00+00:00

The revised cabal file will enter the index with the timestamp provided in meta.toml.

Patches

Foliage also supports patches. Any file with .patch extension in <package>/<version>/patches will be applied as a patch to the source tarball before creating the source distribution.

⚠️ It is not possible to "apply a timestamp" to a patch. A patch changes the content of the source distribution and the repository can only provide one source distribution for a given package name and version.

If you want to add patches incrementally, you can add a new patched version. The source url for the new version can point to the non-patched url.

Author

• Andrea Bedini (@andreabedini)