mirror of
https://github.com/input-output-hk/foliage.git
synced 2024-09-11 12:25:55 +03:00
291 lines
10 KiB
Markdown
291 lines
10 KiB
Markdown
# foliage
|
|
|
|
A hash-friendly Haskell Package Repository.
|
|
|
|
Foliage is a tool to create custom or private Haskell package repositories,
|
|
in a fully reproducible way.
|
|
|
|
## Main idea
|
|
|
|
_Like GitHub Pages but for Haskell Packages_
|
|
|
|
A "Hackage repository" is collection of source distributions and cabal
|
|
files. In addition, Hackage has implemented [The Update
|
|
Framework (TUF)](https://theupdateframework.com) and the repository also
|
|
includes cryptographic metadata (public keys and signatures).
|
|
|
|
These files are commonly served by Hackage proper, that is the central
|
|
deployment of [hackage-server](https://github.com/haskell/hackage-server/).
|
|
|
|
Foliage explores the idea of creating and serving this content as a static
|
|
website, generated programmatically a deterministically from textual input
|
|
files.
|
|
|
|
# Use cases
|
|
|
|
## Company internal hackage
|
|
|
|
Company XYZ has developed many Haskell packages, some of which are forks of
|
|
public Haskell libraries. For various reasons XYZ might not be inclined in
|
|
publishing their packages to Hackage. If XYZ is using multiple repositories
|
|
for version control, keeping all the company's packages in sync will become
|
|
a problem.
|
|
|
|
Currently XYZ's best option is to use `cabal.project` files. Each cabal
|
|
package can declare its non-Hackage dependencies using the
|
|
[`source-repository-package` stanza](https://cabal.readthedocs.io/en/3.6/cabal-project.html#specifying-packages-from-remote-version-control-locations).
|
|
|
|
*Note*: packages can be grouped together into a cabal project, in which case
|
|
`source-repository-package` stanzas become the project dependencies; this
|
|
distintion is inconsequential to our example.
|
|
|
|
E.g. if packageA needs packageB, hosted on GitHub; packageA's
|
|
`cabal.project` will include:
|
|
|
|
```
|
|
source-repository-package
|
|
type: git
|
|
location: https://github.com/Company-XYZ/packageB
|
|
tag: e70cf0c171c9a586b62b3f75d72f1591e4e6aaa1
|
|
```
|
|
|
|
While the presence of a git tag makes this quite reproducible; a problem
|
|
arises in that these dependencies are not transitive. Without any
|
|
versioning to help, one has to manually pick a working set of dependencies.
|
|
|
|
E.g. if packageC depends on packageA, packageC `cabal.project` will have to
|
|
include something like:
|
|
|
|
```
|
|
-- Direct dependency
|
|
source-repository-package
|
|
type: git
|
|
location: https://github.com/Company-XYZ/packageA
|
|
tag: e76fdc753e660dfa615af6c8b6a2ad9ddf6afe70
|
|
|
|
-- From packageA
|
|
source-repository-package
|
|
type: git
|
|
location: https://github.com/Company-XYZ/packageB
|
|
tag: e70cf0c171c9a586b62b3f75d72f1591e4e6aaa1
|
|
```
|
|
|
|
Having an internal company Hackage, solves the above problem by
|
|
reintroducing versioning and a familiar workflow for handling Hackage
|
|
dependencies; while maintaining absolute control and flexibility over
|
|
versioning policies and dependency management.
|
|
|
|
When the team behind packageA wants to push out a new version, say version
|
|
1.2.3.4, all they have to do is to update the foliage repository with a
|
|
file `packageA/1.2.3.4/meta.toml` with content:
|
|
|
|
```toml
|
|
timestamp = 2022-03-29T06:19:50+00:00
|
|
github = { repo = "Company-XYZ/packageA", rev = "e76fdc753e660dfa615af6c8b6a2ad9ddf6afe70" }
|
|
```
|
|
|
|
*Note*: Rather than `github = ` one che use the more generic `url = `, e.g.
|
|
|
|
```toml
|
|
url = "https://hackage.haskell.org/package/shake-0.19.6/shake-0.19.6.tar.gz"
|
|
```
|
|
|
|
:information_source: Although the `timestamp` field in the package source metadata is
|
|
optional, it is highly reccomended if you intend your repository users to
|
|
be able to use cabal's `index-state` functionality. Adding a timestamp
|
|
every time you add a package version ensures the newly created index is
|
|
"compatible" with what the users have already fetched.
|
|
|
|
# Quickstart
|
|
|
|
### Adding one package
|
|
|
|
It's reccomended to create a working directory first.
|
|
|
|
Let's add a package (say `typed-protocols-0.1.0.0` from https://github.com/input-output-hk/ouroboros-network/tree/master/typed-protocols, at commit hash fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa).
|
|
|
|
```bash
|
|
$ mkdir -p _sources/typed-protocols/0.1.0.0
|
|
$ cat _sources/typed-protocols/0.1.0.0/meta.toml
|
|
github = { repo = "input-output-hk/ouroboros-network", rev = "fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa" }
|
|
subdir = "typed-protocols"
|
|
```
|
|
|
|
### Building the repository
|
|
|
|
Run foliage build.
|
|
|
|
```
|
|
$ foliage build
|
|
🌿 Foliage
|
|
You don't seem to have created a set of TUF keys. I will create one in _keys
|
|
Current time set to 2022-05-16T09:33:26Z. You can set a fixed time using the --current-time option.
|
|
Expiry time set to 2023-05-16T09:33:26Z (a year from now).
|
|
# curl (for RemoteAsset "https://github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa")
|
|
# tar (for OracleQ (PreparePackageSource (PackageId {pkgName = "typed-protocols", pkgVersion = "0.1.0.0"})))
|
|
# cp (for OracleQ (PreparePackageSource (PackageId {pkgName = "typed-protocols", pkgVersion = "0.1.0.0"})))
|
|
# writing (for _repo/root.json)
|
|
# writing (for _repo/mirrors.json)
|
|
_cache/packages/typed-protocols/0.1.0.0
|
|
Creating source distribution for typed-protocols-0.1.0.0
|
|
# cabal (for _repo/package/typed-protocols-0.1.0.0.tar.gz)
|
|
# mv (for _repo/package/typed-protocols-0.1.0.0.tar.gz)
|
|
# writing (for _repo/01-index.tar)
|
|
# writing (for _repo/01-index.tar.gz)
|
|
# writing (for _repo/snapshot.json)
|
|
# writing (for _repo/timestamp.json)
|
|
All done. The repository is now available in _repo.
|
|
```
|
|
|
|
If you want to rely on the cabal index-state feature you need to specify a
|
|
timestamp in the `meta.toml` file.
|
|
|
|
E.g.
|
|
|
|
```
|
|
$ cat _sources/typed-protocols/0.1.0.0/meta.toml
|
|
github = { repo = "input-output-hk/ouroboros-network", rev = "fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa" }
|
|
subdir = "typed-protocols"
|
|
timestamp = 2022-03-29T06:19:50+00:00
|
|
```
|
|
|
|
:information_source: Foliage uses the metadata timestamps to determine the
|
|
order of the entries in `01-index`. This allows you to create an index that
|
|
can be updated incrementally and can be used with cabal's
|
|
[index-state](https://cabal.readthedocs.io/en/3.6/cabal-project.html?highlight=index-state#cfg-field-index-state)
|
|
feature.
|
|
|
|
With the input above foliage will produce the following:
|
|
|
|
```
|
|
_repo
|
|
├── 01-index.tar
|
|
├── 01-index.tar.gz
|
|
├── index
|
|
│ └── typed-protocols
|
|
│ └── 0.1.0.0
|
|
│ ├── package.json
|
|
│ └── typed-protocols.cabal
|
|
├── mirrors.json
|
|
├── package
|
|
│ └── typed-protocols-0.1.0.0.tar.gz
|
|
├── root.json
|
|
├── snapshot.json
|
|
└── timestamp.json
|
|
```
|
|
|
|
The content of `_repo` can be served over HTTP(s) to cabal, e.g. from a
|
|
`cabal.project` file.
|
|
|
|
```
|
|
repository packages.example.org
|
|
url: http://packages.example.org/
|
|
secure: True
|
|
root-keys:
|
|
-- root-keys ids, see below
|
|
144d97d34d0a86adb1ca7d6bdc1b2d9f0c9123e3c29e3765f5a9eece345ce4f9
|
|
a15f6ae88a26638934d90eff28da29990a4b12c8bb0b2c12f07e9a510e839a97
|
|
fde23c79a14bcbef6ccf198b4ad94ded4092784fcaed17c3d184008e9bf6f722
|
|
key-threshold: 3
|
|
```
|
|
|
|
### TUF keys
|
|
|
|
Foliage creates a set of private keys to sign the TUF metadata at first
|
|
use. By default, the keys are created in `_keys`:
|
|
|
|
```
|
|
_keys/
|
|
├── mirrors
|
|
│ ├── 105369fb9cb1555cf4517be3e885215a7bc730bd59cf3084ea7140f9692ae847.json
|
|
│ ├── 2178cff7b2a3a6edd396371c03bc8fddb77b77b104a9fd97f6291f2c49285946.json
|
|
│ └── 4689f8a6d905a59536213a27ea577b34ff3e6d79d5a7b375458d3bb6026a5e13.json
|
|
├── root
|
|
│ ├── 144d97d34d0a86adb1ca7d6bdc1b2d9f0c9123e3c29e3765f5a9eece345ce4f9.json
|
|
│ ├── a15f6ae88a26638934d90eff28da29990a4b12c8bb0b2c12f07e9a510e839a97.json
|
|
│ └── fde23c79a14bcbef6ccf198b4ad94ded4092784fcaed17c3d184008e9bf6f722.json
|
|
├── snapshot
|
|
│ └── 07a918ccdb3ac0600a8b65f3bc96da18dfc5be65c023c64ccd0fb6a04692b64d.json
|
|
├── target
|
|
│ ├── 793b1b2730db6ec5247934ad0cc7b28ed3b05eae4ffec7e28e7b1739f1cb65b4.json
|
|
│ ├── 908041deaae700390dfd7b4c8d3eca7049c8172a205bea19de8314c4fd9ca56f.json
|
|
│ └── f102c7035da2a3b5fa82b90d8afe3e8892d13a8c6b7a4281403c340317a35014.json
|
|
└── timestamp
|
|
└── 141da8eb2ccba61c2f6bb656b2292970d086770f5bf7d53802d2bc0ec1defa26.json
|
|
```
|
|
|
|
The root-keys ids are simply the names of they key files in
|
|
`_keys/roots/*.json`.
|
|
|
|
These keys are small enough you can store them in an environment variable.
|
|
E.g.
|
|
|
|
Save:
|
|
```bash
|
|
$ KEYS=$(tar cz -C _keys . | base64)
|
|
```
|
|
|
|
Restore:
|
|
```bash
|
|
$ mkdir _keys
|
|
$ echo "$KEYS" | base64 -d | tar xz -C _keys
|
|
```
|
|
|
|
:warning: These are private keys. Don't publish them along with your
|
|
repository!
|
|
|
|
# Revisions
|
|
|
|
Foliage supports cabal file revisions. Adding the following snippet to a
|
|
package's `meta.toml`, will make foliage look for a cabal file in
|
|
`<package>/<version>/revisions/1.cabal`.
|
|
|
|
```
|
|
[[revisions]]
|
|
number = 1
|
|
timestamp = 2022-03-22T14:15:00+00:00
|
|
```
|
|
|
|
The revised cabal file will enter the index with the timestamp provided in
|
|
`meta.toml`.
|
|
|
|
# Patches
|
|
|
|
Foliage also supports patches. Any file with `.patch` extension in
|
|
`<package>/<version>/patches` will be applied as a patch to the package
|
|
sources.
|
|
|
|
Patches are applied very early in the process, and they can modify any file
|
|
in the source tree.
|
|
|
|
Revisions are independent of patches, a patched cabal file will be subject
|
|
to revisions just like the original cabal file.
|
|
|
|
:warning: It is not possible to "apply a timestamp" to a patch. A patch
|
|
changes the content of the source distribution and the repository can only
|
|
provide one source distribution for a given package name and version.
|
|
|
|
# New package versions
|
|
|
|
You can introduce new versions of existing packages by patching the cabal
|
|
file. Foliage supports a short cut for this use case. If the package
|
|
metadata file `<package>/<version>/meta.toml` includes the option
|
|
`force-version = true`, then the version in the package cabal file will be
|
|
overwritten with `<version>`. Notice this is done after applying patches.
|
|
|
|
# Accessing private repositories
|
|
|
|
Foliage uses curl to fetch remote assets. If you need to access private
|
|
repositories you can use one of the following methods.
|
|
|
|
- You can instruct curl to use a netrc file.
|
|
- Add netrc-optional to ~/.curlrc (see [here](https://curl.se/docs/manpage.html#-K) for the places curl looks in).
|
|
- Add the relevant credentials to ~/.netrc
|
|
- You can also add the credentials directly to curl's configuration, e.g.
|
|
- Add `user = "USERNAME:PASSWORD"` to .curlrc in the current directly
|
|
- Set `CURL_HOME=$PWD`
|
|
|
|
## Author
|
|
|
|
- Andrea Bedini (@andreabedini)
|