mirror of
https://github.com/ilyakooo0/nixpkgs.git
synced 2024-11-10 08:39:08 +03:00
maintainers/scripts/bootstrap-files: documentation and a script to update tarballs
This script attempts to document the exact procedure used to upload bootstrap binaries used previously. I modeled it after most recent https://github.com/NixOS/nixpkgs/pull/282517 upload. There is one deviation from it to make it easier to handle mass updates for https://github.com/NixOS/nixpkgs/issues/253713: The binaries are expected to be stored in `stdenv/$target` (and not something like `stdenv-linux/i686`. The script handles both native and cross- linux targets. `darwin` will need a bit more work to fin into this scheme, but it should be easy. Example run to generate `i686-linux` update: $ maintainers/scripts/bootstrap-files/refresh-tarballs.bash --commit --targets=i686-unknown-linux-gnu
This commit is contained in:
Sergei Trofimovich
parent
e0be3bfaf0
commit
8a256ed0a7
85
maintainers/scripts/bootstrap-files/README.md
Normal file
85
maintainers/scripts/bootstrap-files/README.md
Normal file
@ -0,0 +1,85 @@
|
|||||||
|
# Bootstrap files
|
||||||
|
|
||||||
|
Currently `nixpkgs` builds most of it's packages using bootstrap seed
|
||||||
|
binaries (without the reliance on external inputs):
|
||||||
|
|
||||||
|
- `bootstrap-tools`: an archive with the compiler toolchain and other
|
||||||
|
helper tools enough to build the rest of the `nixpkgs`.
|
||||||
|
- initial binaries needed to unpack `bootstrap-tools.*`. On `linux`
|
||||||
|
it's just `busybox`, on `darwin` it's `sh`, `bzip2`, `mkdir` and
|
||||||
|
`cpio`. These binaries can be executed directly from the store.
|
||||||
|
|
||||||
|
These are called "bootstrap files".
|
||||||
|
|
||||||
|
Bootstrap files should always be fetched from hydra and uploaded to
|
||||||
|
`tarballs.nixos.org` to guarantee that all the binaries were built from
|
||||||
|
the code committed into `nixpkgs` repository.
|
||||||
|
|
||||||
|
The uploads to `tarballs.nixos.org` are done by `@lovesegfault` today.
|
||||||
|
|
||||||
|
This document describes the procedure of updating bootstrap files in
|
||||||
|
`nixpkgs`.
|
||||||
|
|
||||||
|
## How to request the bootstrap seed update
|
||||||
|
|
||||||
|
To get the tarballs updated let's use an example `i686-unknown-linux-gnu`
|
||||||
|
target:
|
||||||
|
|
||||||
|
1. Create a local update:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ maintainers/scripts/bootstrap-files/refresh-tarballs.bash --commit --targets=i686-unknown-linux-gnu
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Test the update locally. I'll build local `hello` derivation with
|
||||||
|
the result:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ nix-build -A hello --argstr system i686-linux
|
||||||
|
```
|
||||||
|
|
||||||
|
To validate cross-targets `binfmt` `NixOS` helper can be useful.
|
||||||
|
For `riscv64-unknown-linux-gnu` the `/etc/nixox/configuraqtion.nix`
|
||||||
|
entry would be `boot.binfmt.emulatedSystems = [ "riscv64-linux" ]`.
|
||||||
|
|
||||||
|
3. Propose the commit as a PR to update bootstrap tarballs, tag people
|
||||||
|
who can help you test the updated architecture and once reviewed tag
|
||||||
|
`@lovesegfault` to upload the tarballs.
|
||||||
|
|
||||||
|
## Bootstrap files job definitions
|
||||||
|
|
||||||
|
There are two types of bootstrap files:
|
||||||
|
|
||||||
|
- natively built `stdenvBootstrapTools.build` hydra jobs in
|
||||||
|
[`nixpkgs:trunk`](https://hydra.nixos.org/jobset/nixpkgs/trunk#tabs-jobs)
|
||||||
|
jobset. Incomplete list of examples is:
|
||||||
|
|
||||||
|
* `aarch64-unknown-linux-musl.nix`
|
||||||
|
* `i686-unknown-linux-gnu.nix`
|
||||||
|
|
||||||
|
These are Tier 1 hydra platforms.
|
||||||
|
|
||||||
|
- cross-built by `bootstrapTools.build` hydra jobs in
|
||||||
|
[`nixpkgs:cross-trunk`](https://hydra.nixos.org/jobset/nixpkgs/cross-trunk#tabs-jobs)
|
||||||
|
jobset. Incomplete list of examples is:
|
||||||
|
|
||||||
|
* `mips64el-unknown-linux-gnuabi64.nix`
|
||||||
|
* `mips64el-unknown-linux-gnuabin32.nix`
|
||||||
|
* `mipsel-unknown-linux-gnu.nix`
|
||||||
|
* `powerpc64le-unknown-linux-gnu.nix`
|
||||||
|
* `riscv64-unknown-linux-gnu.nix`
|
||||||
|
|
||||||
|
These are usually Tier 2 and lower targets.
|
||||||
|
|
||||||
|
The `.build` job contains `/on-server/` subdirectory with binaries to
|
||||||
|
be uploaded to `tarballs.nixos.org`.
|
||||||
|
The files are uploaded to `tarballs.nixos.org` by writers to `S3` store.
|
||||||
|
|
||||||
|
## TODOs
|
||||||
|
|
||||||
|
- `pkgs/stdenv/darwin` file layout is slightly different from
|
||||||
|
`pkgs/stdenv/linux`. Once `linux` seed update becomes a routine we can
|
||||||
|
bring `darwin` in sync if it's feasible.
|
||||||
|
- `darwin` definition of `.build` `on-server/` directory layout differs
|
||||||
|
and should be updated.
|
||||||
|
|
||||||
282
maintainers/scripts/bootstrap-files/refresh-tarballs.bash
Executable file
282
maintainers/scripts/bootstrap-files/refresh-tarballs.bash
Executable file
@ -0,0 +1,282 @@
|
|||||||
|
#!/usr/bin/env nix-shell
|
||||||
|
#! nix-shell --pure
|
||||||
|
#! nix-shell -i bash
|
||||||
|
#! nix-shell -p curl cacert
|
||||||
|
#! nix-shell -p git
|
||||||
|
#! nix-shell -p nix
|
||||||
|
#! nix-shell -p jq
|
||||||
|
|
||||||
|
# How the refresher works:
|
||||||
|
#
|
||||||
|
# For a given list of <targets>:
|
||||||
|
# 1. fetch latest successful '.build` job
|
||||||
|
# 2. fetch oldest evaluation that contained that '.build', extract nixpkgs commit
|
||||||
|
# 3. fetch all the `.build` artifacts from '$out/on-server/' directory
|
||||||
|
# 4. calculate hashes and craft the commit message with the details on
|
||||||
|
# how to upload the result to 'tarballs.nixos.org'
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat >&2 <<EOF
|
||||||
|
Usage:
|
||||||
|
$0 [ --commit ] --targets=<target>[,<target>,...]
|
||||||
|
|
||||||
|
The tool must be ran from the root directory of 'nixpkgs' repository.
|
||||||
|
|
||||||
|
Synopsis:
|
||||||
|
'refresh-tarballs.bash' script fetches latest bootstrapFiles built
|
||||||
|
by hydra, registers them in 'nixpkgs' and provides commands to
|
||||||
|
upload seed files to 'tarballs.nixos.org'.
|
||||||
|
|
||||||
|
This is usually done in the following cases:
|
||||||
|
|
||||||
|
1. Single target fix: current bootstrap files for a single target
|
||||||
|
are problematic for some reason (target-specific bug). In this
|
||||||
|
case we can refresh just that target as:
|
||||||
|
|
||||||
|
\$ $0 --commit --targets=i686-unknown-linux-gnu
|
||||||
|
|
||||||
|
2. Routine refresh: all bootstrap files should be refreshed to avoid
|
||||||
|
debugging problems that only occur on very old binaries.
|
||||||
|
|
||||||
|
\$ $0 --commit --all-targets
|
||||||
|
|
||||||
|
To get help on uploading refreshed binaries to 'tarballs.nixos.org'
|
||||||
|
please have a look at <maintainers/scripts/bootstrap-files/README.md>.
|
||||||
|
EOF
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# log helpers
|
||||||
|
|
||||||
|
die() {
|
||||||
|
echo "ERROR: $*" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
info() {
|
||||||
|
echo "INFO: $*" >&2
|
||||||
|
}
|
||||||
|
|
||||||
|
[[ ${#@} -eq 0 ]] && usage
|
||||||
|
|
||||||
|
# known targets
|
||||||
|
|
||||||
|
NATIVE_TARGETS=(
|
||||||
|
aarch64-unknown-linux-gnu
|
||||||
|
aarch64-unknown-linux-musl
|
||||||
|
i686-unknown-linux-gnu
|
||||||
|
x86_64-unknown-linux-gnu
|
||||||
|
x86_64-unknown-linux-musl
|
||||||
|
|
||||||
|
# TODO: add darwin here once a few prerequisites are satisfied:
|
||||||
|
# - bootstrap-files are factored out into a separate file
|
||||||
|
# - the build artifacts are factored out into an `on-server`
|
||||||
|
# directory. Right onw if does not match `linux` layout.
|
||||||
|
#
|
||||||
|
#aarch64-apple-darwin
|
||||||
|
#x86_64-apple-darwin
|
||||||
|
)
|
||||||
|
|
||||||
|
is_native() {
|
||||||
|
local t target=$1
|
||||||
|
for t in "${NATIVE_TARGETS[@]}"; do
|
||||||
|
[[ $t == $target ]] && return 0
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
CROSS_TARGETS=(
|
||||||
|
armv5tel-unknown-linux-gnueabi
|
||||||
|
armv6l-unknown-linux-gnueabihf
|
||||||
|
armv6l-unknown-linux-musleabihf
|
||||||
|
armv7l-unknown-linux-gnueabihf
|
||||||
|
mips64el-unknown-linux-gnuabi64
|
||||||
|
mips64el-unknown-linux-gnuabin32
|
||||||
|
mipsel-unknown-linux-gnu
|
||||||
|
powerpc64le-unknown-linux-gnu
|
||||||
|
riscv64-unknown-linux-gnu
|
||||||
|
)
|
||||||
|
|
||||||
|
is_cross() {
|
||||||
|
local t target=$1
|
||||||
|
for t in "${CROSS_TARGETS[@]}"; do
|
||||||
|
[[ $t == $target ]] && return 0
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# collect passed options
|
||||||
|
|
||||||
|
targets=()
|
||||||
|
commit=no
|
||||||
|
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in
|
||||||
|
--all-targets)
|
||||||
|
targets+=(
|
||||||
|
${CROSS_TARGETS[@]}
|
||||||
|
${NATIVE_TARGETS[@]}
|
||||||
|
)
|
||||||
|
;;
|
||||||
|
--targets=*)
|
||||||
|
# Convert "--targets=a,b,c" to targets=(a b c) bash array.
|
||||||
|
comma_targets=${arg#--targets=}
|
||||||
|
targets+=(${comma_targets//,/ })
|
||||||
|
;;
|
||||||
|
--commit)
|
||||||
|
commit=yes
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
usage
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
for target in "${targets[@]}"; do
|
||||||
|
# Native and cross jobsets differ a bit. We'll have to pick the
|
||||||
|
# one based on target name:
|
||||||
|
if is_native $target; then
|
||||||
|
jobset=nixpkgs/trunk
|
||||||
|
job="stdenvBootstrapTools.${target}.build"
|
||||||
|
elif is_cross $target; then
|
||||||
|
jobset=nixpkgs/cross-trunk
|
||||||
|
job="bootstrapTools.${target}.build"
|
||||||
|
else
|
||||||
|
die "'$target' is not present in either of 'NATIVE_TARGETS' or 'CROSS_TARGETS'. Please add one."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 'nixpkgs' prefix where we will write new tarball hashes
|
||||||
|
case "$target" in
|
||||||
|
*linux*) nixpkgs_prefix="pkgs/stdenv/linux" ;;
|
||||||
|
*darwin*) nixpkgs_prefix="pkgs/stdenv/darwin" ;;
|
||||||
|
*) die "don't know where to put '$target'" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# We enforce s3 prefix for all targets here. This slightly differs
|
||||||
|
# from manual uploads targets where names were chosen inconsistently.
|
||||||
|
s3_prefix="stdenv/$target"
|
||||||
|
|
||||||
|
# resolve 'latest' build to the build 'id', construct the link.
|
||||||
|
latest_build_uri="https://hydra.nixos.org/job/$jobset/$job/latest"
|
||||||
|
latest_build="$target.latest-build"
|
||||||
|
info "Fetching latest successful build from '${latest_build_uri}'"
|
||||||
|
curl -s -H "Content-Type: application/json" -L "$latest_build_uri" > "$latest_build"
|
||||||
|
[[ $? -ne 0 ]] && die "Failed to fetch latest successful build"
|
||||||
|
latest_build_id=$(jq '.id' < "$latest_build")
|
||||||
|
[[ $? -ne 0 ]] && die "Did not find 'id' in latest build"
|
||||||
|
build_uri="https://hydra.nixos.org/build/${latest_build_id}"
|
||||||
|
|
||||||
|
# We pick oldest jobset evaluation and extract the 'nicpkgs' commit.
|
||||||
|
#
|
||||||
|
# We use oldest instead of latest to make the result more stable
|
||||||
|
# across unrelated 'nixpkgs' updates. Ideally two subsequent runs of
|
||||||
|
# this refresher should produce the same output (provided there are
|
||||||
|
# no bootstrapTools updates committed between the two runs).
|
||||||
|
oldest_eval_id=$(jq '.jobsetevals|min' < "$latest_build")
|
||||||
|
[[ $? -ne 0 ]] && die "Did not find 'jobsetevals' in latest build"
|
||||||
|
eval_uri="https://hydra.nixos.org/eval/${oldest_eval_id}"
|
||||||
|
eval_meta="$target.eval-meta"
|
||||||
|
info "Fetching oldest eval details from '${eval_uri}' (can take a minute)"
|
||||||
|
curl -s -H "Content-Type: application/json" -L "${eval_uri}" > "$eval_meta"
|
||||||
|
[[ $? -ne 0 ]] && die "Failed to fetch eval metadata"
|
||||||
|
nixpkgs_revision=$(jq --raw-output ".jobsetevalinputs.nixpkgs.revision" < "$eval_meta")
|
||||||
|
[[ $? -ne 0 ]] && die "Failed to fetch revision"
|
||||||
|
|
||||||
|
# Extract the build paths out of the build metadata
|
||||||
|
drvpath=$(jq --raw-output '.drvpath' < "${latest_build}")
|
||||||
|
[[ $? -ne 0 ]] && die "Did not find 'drvpath' in latest build"
|
||||||
|
outpath=$(jq --raw-output '.buildoutputs.out.path' < "${latest_build}")
|
||||||
|
[[ $? -ne 0 ]] && die "Did not find 'buildoutputs' in latest build"
|
||||||
|
build_timestamp=$(jq --raw-output '.timestamp' < "${latest_build}")
|
||||||
|
[[ $? -ne 0 ]] && die "Did not find 'timestamp' in latest build"
|
||||||
|
build_time=$(TZ=UTC LANG=C date --date="@${build_timestamp}" --rfc-email)
|
||||||
|
[[ $? -ne 0 ]] && die "Failed to format timestamp"
|
||||||
|
|
||||||
|
info "Fetching bootstrap tools to calculate hashes from '${outpath}'"
|
||||||
|
nix-store --realize "$outpath"
|
||||||
|
[[ $? -ne 0 ]] && die "Failed to fetch '${outpath}' from hydra"
|
||||||
|
|
||||||
|
fnames=()
|
||||||
|
|
||||||
|
target_file="${nixpkgs_prefix}/bootstrap-files/${target}.nix"
|
||||||
|
info "Writing '${target_file}'"
|
||||||
|
{
|
||||||
|
# header
|
||||||
|
cat <<EOF
|
||||||
|
# Autogenerated by maintainers/scripts/bootstrap-files/refresh-tarballs.bash as:
|
||||||
|
# $ ./refresh-tarballs.bash --targets=${target}
|
||||||
|
#
|
||||||
|
# Metadata:
|
||||||
|
# - nixpkgs revision: ${nixpkgs_revision}
|
||||||
|
# - hydra build: ${latest_build_uri}
|
||||||
|
# - resolved hydra build: ${build_uri}
|
||||||
|
# - instantiated derivation: ${drvpath}
|
||||||
|
# - output directory: ${outpath}
|
||||||
|
# - build time: ${build_time}
|
||||||
|
{
|
||||||
|
EOF
|
||||||
|
for p in "${outpath}/on-server"/*; do
|
||||||
|
fname=$(basename "$p")
|
||||||
|
fnames+=("$fname")
|
||||||
|
case "$fname" in
|
||||||
|
bootstrap-tools.tar.xz) attr=bootstrapTools ;;
|
||||||
|
busybox) attr=$fname ;;
|
||||||
|
*) die "Don't know how to map '$fname' to attribute name. Please update me."
|
||||||
|
esac
|
||||||
|
|
||||||
|
executable_arg=
|
||||||
|
executable_nix=
|
||||||
|
if [[ -x "$p" ]]; then
|
||||||
|
executable_arg="--executable"
|
||||||
|
executable_nix=" executable = true;"
|
||||||
|
fi
|
||||||
|
sha256=$(nix-prefetch-url $executable_arg --name "$fname" "file://$p")
|
||||||
|
[[ $? -ne 0 ]] && die "Failed to get the hash for '$p'"
|
||||||
|
sri=$(nix-hash --to-sri "sha256:$sha256")
|
||||||
|
[[ $? -ne 0 ]] && die "Failed to convert '$sha256' hash to an SRI form"
|
||||||
|
|
||||||
|
# individual file entries
|
||||||
|
cat <<EOF
|
||||||
|
$attr = import <nix/fetchurl.nix> {
|
||||||
|
url = "http://tarballs.nixos.org/${s3_prefix}/${nixpkgs_revision}/$fname";
|
||||||
|
hash = "${sri}";$(printf "\n%s" "${executable_nix}")
|
||||||
|
};
|
||||||
|
EOF
|
||||||
|
done
|
||||||
|
# footer
|
||||||
|
cat <<EOF
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
} > "${target_file}"
|
||||||
|
|
||||||
|
target_file_commit_msg=${target}.commit_message
|
||||||
|
cat > "$target_file_commit_msg" <<EOF
|
||||||
|
${nixpkgs_prefix}: update ${target} bootstrap-files
|
||||||
|
|
||||||
|
sha256sum of files to be uploaded:
|
||||||
|
|
||||||
|
$(
|
||||||
|
echo "$ sha256sum ${outpath}/on-server/*"
|
||||||
|
sha256sum ${outpath}/on-server/*
|
||||||
|
)
|
||||||
|
|
||||||
|
Suggested commands to upload files to 'tarballs.nixos.org':
|
||||||
|
|
||||||
|
$ nix-store --realize ${outpath}
|
||||||
|
$ aws s3 cp --recursive --acl public-read ${outpath}/on-server/ s3://nixpkgs-tarballs/${s3_prefix}/${nixpkgs_revision}
|
||||||
|
$ aws s3 cp --recursive s3://nixpkgs-tarballs/${s3_prefix}/${nixpkgs_revision} ./
|
||||||
|
$ sha256sum ${fnames[*]}
|
||||||
|
$ sha256sum ${outpath}/on-server/*
|
||||||
|
EOF
|
||||||
|
|
||||||
|
cat "$target_file_commit_msg"
|
||||||
|
if [[ $commit == yes ]]; then
|
||||||
|
git commit "${target_file}" -F "$target_file_commit_msg"
|
||||||
|
else
|
||||||
|
info "DRY RUN: git commit ${target_file} -F $target_file_commit_msg"
|
||||||
|
fi
|
||||||
|
rm -- "$target_file_commit_msg"
|
||||||
|
|
||||||
|
# delete temp files
|
||||||
|
rm -- "$latest_build" "$eval_meta"
|
||||||
|
done
|
||||||
Loading…
Reference in New Issue
Block a user