nixos/garage: provide multiple versions to provide an upgrade path when using NixOS service

- Add mention to release notes 23.05
- Introduce Garage v0.8
- Protect against unexpected upgrade with stateVersion
- Test matrix over 0.7 × 0.8
This commit is contained in:
Raito Bezarius 2022-10-15 22:40:34 +02:00
parent 04f4be9a29
commit 1db2175e7a
10 changed files with 409 additions and 101 deletions

View File

@ -407,6 +407,21 @@
<link xlink:href="https://github.com/google/ngx_brotli/blob/master/README.md">here</link>.
</para>
</listitem>
<listitem>
<para>
<link xlink:href="https://garagehq.deuxfleurs.fr/">Garage</link>
version is based on
<link xlink:href="options.html#opt-system.stateVersion">system.stateVersion</link>,
existing installations will keep using version 0.7. New
installations will use version 0.8. In order to upgrade a
Garage cluster, please follow
<link xlink:href="https://garagehq.deuxfleurs.fr/documentation/cookbook/upgrading/">upstream
instructions</link> and force
<link xlink:href="options.html#opt-services.garage.package">services.garage.package</link>
or upgrade accordingly
<link xlink:href="options.html#opt-system.stateVersion">system.stateVersion</link>.
</para>
</listitem>
<listitem>
<para>
Resilio sync secret keys can now be provided using a secrets

View File

@ -111,6 +111,8 @@ In addition to numerous new and upgraded packages, this release has the followin
- A new option `recommendedBrotliSettings` has been added to `services.nginx`. Learn more about compression in Brotli format [here](https://github.com/google/ngx_brotli/blob/master/README.md).
- [Garage](https://garagehq.deuxfleurs.fr/) version is based on [system.stateVersion](options.html#opt-system.stateVersion), existing installations will keep using version 0.7. New installations will use version 0.8. In order to upgrade a Garage cluster, please follow [upstream instructions](https://garagehq.deuxfleurs.fr/documentation/cookbook/upgrading/) and force [services.garage.package](options.html#opt-services.garage.package) or upgrade accordingly [system.stateVersion](options.html#opt-system.stateVersion).
- Resilio sync secret keys can now be provided using a secrets file at runtime, preventing these secrets from ending up in the Nix store.
- The `firewall` and `nat` module now has a nftables based implementation. Enable `networking.nftables` to use it.

View File

@ -0,0 +1,139 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="module-services-garage">
<title>Garage</title>
<para>
<link xlink:href="https://garagehq.deuxfleurs.fr/">Garage</link>
is an open-source, self-hostable S3 store, simpler than MinIO, for geodistributed stores.
The server setup can be automated using
<link linkend="opt-services.garage.enable">services.garage</link>. A
client configured to your local Garage instance is available in
the global environment as <literal>garage-manage</literal>.
</para>
<para>
The current default by NixOS is <package>garage_0_8</package> which is also the latest
major version available.
</para>
<section xml:id="module-services-garage-upgrade-scenarios">
<title>General considerations on upgrades</title>
<para>
Garage provides a cookbook documentation on how to upgrade:
<link xlink:href="https://garagehq.deuxfleurs.fr/documentation/cookbook/upgrading/">https://garagehq.deuxfleurs.fr/documentation/cookbook/upgrading/</link>
</para>
<warning>
<para>Garage has two types of upgrades: patch-level upgrades and minor/major version upgrades.</para>
<para>In all cases, you should read the changelog and ideally test the upgrade on a staging cluster.</para>
<para>Checking the health of your cluster can be achieved using <literal>garage-manage repair</literal>.</para>
</warning>
<warning>
<para>Until 1.0 is released, patch-level upgrades are considered as minor version upgrades.
Minor version upgrades are considered as major version upgrades.
i.e. 0.6 to 0.7 is a major version upgrade.</para>
</warning>
<itemizedlist>
<listitem>
<formalpara>
<title>Straightforward upgrades (patch-level upgrades)</title>
<para>
Upgrades must be performed one by one, i.e. for each node, stop it, upgrade it : change <link linkend="opt-system.stateVersion">stateVersion</link> or <link linkend="opt-services.garage.package">services.garage.package</link>, restart it if it was not already by switching.
</para>
</formalpara>
</listitem>
<listitem>
<formalpara>
<title>Multiple version upgrades</title>
<para>
Garage do not provide any guarantee on moving more than one major-version forward.
E.g., if you're on <literal>0.7</literal>, you cannot upgrade to <literal>0.9</literal>.
You need to upgrade to <literal>0.8</literal> first.
As long as <link linkend="opt-system.stateVersion">stateVersion</link> is declared properly,
this is enforced automatically. The module will issue a warning to remind the user to upgrade to latest
Garage <emphasis>after</emphasis> that deploy.
</para>
</formalpara>
</listitem>
</itemizedlist>
</section>
<section xml:id="module-services-garage-advanced-upgrades">
<title>Advanced upgrades (minor/major version upgrades)</title>
<para>Here are some baseline instructions to handle advanced upgrades in Garage, when in doubt, please refer to upstream instructions.</para>
<itemizedlist>
<listitem><para>Disable API and web access to Garage.</para></listitem>
<listitem><para>Perform <literal>garage-manage repair --all-nodes --yes tables</literal> and <literal>garage-manage repair --all-nodes --yes blocks</literal>.</para></listitem>
<listitem><para>Verify the resulting logs and check that data is synced properly between all nodes.
If you have time, do additional checks (<literal>scrub</literal>, <literal>block_refs</literal>, etc.).</para></listitem>
<listitem><para>Check if queues are empty by <literal>garage-manage stats</literal> or through monitoring tools.</para></listitem>
<listitem><para>Run <literal>systemctl stop garage</literal> to stop the actual Garage version.</para></listitem>
<listitem><para>Backup the metadata folder of ALL your nodes, e.g. for a metadata directory (the default one) in <literal>/var/lib/garage/meta</literal>,
you can run <literal>pushd /var/lib/garage; tar -acf meta-v0.7.tar.zst meta/; popd</literal>.</para></listitem>
<listitem><para>Run the offline migration: <literal>nix-shell -p garage_0_8 --run "garage offline-repair --yes"</literal>, this can take some time depending on how many objects are stored in your cluster.</para></listitem>
<listitem><para>Bump Garage version in your NixOS configuration, either by changing <link linkend="opt-system.stateVersion">stateVersion</link> or bumping <link linkend="opt-services.garage.package">services.garage.package</link>, this should restart Garage automatically.</para></listitem>
<listitem><para>Perform <literal>garage-manage repair --all-nodes --yes tables</literal> and <literal>garage-manage repair --all-nodes --yes blocks</literal>.</para></listitem>
<listitem><para>Wait for a full table sync to run.</para></listitem>
</itemizedlist>
<para>
Your upgraded cluster should be in a working state, re-enable API and web access.
</para>
</section>
<section xml:id="module-services-garage-maintainer-info">
<title>Maintainer information</title>
<para>
As stated in the previous paragraph, we must provide a clean upgrade-path for Garage
since it cannot move more than one major version forward on a single upgrade. This chapter
adds some notes how Garage updates should be rolled out in the future.
This is inspired from how Nextcloud does it.
</para>
<para>
While patch-level updates are no problem and can be done directly in the
package-expression (and should be backported to supported stable branches after that),
major-releases should be added in a new attribute (e.g. Garage <literal>v0.8.0</literal>
should be available in <literal>nixpkgs</literal> as <literal>pkgs.garage_0_8_0</literal>).
To provide simple upgrade paths it's generally useful to backport those as well to stable
branches. As long as the package-default isn't altered, this won't break existing setups.
After that, the versioning-warning in the <literal>garage</literal>-module should be
updated to make sure that the
<link linkend="opt-services.garage.package">package</link>-option selects the latest version
on fresh setups.
</para>
<para>
If major-releases will be abandoned by upstream, we should check first if those are needed
in NixOS for a safe upgrade-path before removing those. In that case we shold keep those
packages, but mark them as insecure in an expression like this (in
<literal>&lt;nixpkgs/pkgs/tools/filesystem/garage/default.nix&gt;</literal>):
<programlisting>/* ... */
{
garage_0_7_3 = generic {
version = "0.7.3";
sha256 = "0000000000000000000000000000000000000000000000000000";
eol = true;
};
}</programlisting>
</para>
<para>
Ideally we should make sure that it's possible to jump two NixOS versions forward:
i.e. the warnings and the logic in the module should guard a user to upgrade from a
Garage on e.g. 22.11 to a Garage on 23.11.
</para>
</section>
</chapter>

View File

@ -8,7 +8,10 @@ let
configFile = toml.generate "garage.toml" cfg.settings;
in
{
meta.maintainers = [ maintainers.raitobezarius ];
meta = {
doc = ./garage-doc.xml;
maintainers = with pkgs.lib.maintainers; [ raitobezarius ];
};
options.services.garage = {
enable = mkEnableOption (lib.mdDoc "Garage Object Storage (S3 compatible)");
@ -56,10 +59,12 @@ in
};
package = mkOption {
default = pkgs.garage;
defaultText = literalExpression "pkgs.garage";
# TODO: when 23.05 is released and if Garage 0.9 is the default, put a stateVersion check.
default = if versionAtLeast stateVersion "23.05" then pkgs.garage_0_8_0
else pkgs.garage_0_7;
defaultText = literalExpression "pkgs.garage_0_7";
type = types.package;
description = lib.mdDoc "Garage package to use.";
description = lib.mdDoc "Garage package to use, if you are upgrading from a major version, please read NixOS and Garage release notes for upgrade instructions.";
};
};

View File

@ -229,7 +229,7 @@ in {
fsck = handleTest ./fsck.nix {};
ft2-clone = handleTest ./ft2-clone.nix {};
mimir = handleTest ./mimir.nix {};
garage = handleTest ./garage.nix {};
garage = handleTest ./garage {};
gerrit = handleTest ./gerrit.nix {};
geth = handleTest ./geth.nix {};
ghostunnel = handleTest ./ghostunnel.nix {};

View File

@ -0,0 +1,98 @@
args@{ mkNode, ... }:
(import ../make-test-python.nix ({ pkgs, ...} : {
name = "garage-basic";
meta = {
maintainers = with pkgs.lib.maintainers; [ raitobezarius ];
};
nodes = {
single_node = mkNode { replicationMode = "none"; };
};
testScript = ''
from typing import List
from dataclasses import dataclass
import re
start_all()
cur_version_regex = re.compile('Current cluster layout version: (?P<ver>\d*)')
key_creation_regex = re.compile('Key name: (?P<key_name>.*)\nKey ID: (?P<key_id>.*)\nSecret key: (?P<secret_key>.*)')
@dataclass
class S3Key:
key_name: str
key_id: str
secret_key: str
@dataclass
class GarageNode:
node_id: str
host: str
def get_node_fqn(machine: Machine) -> GarageNode:
node_id, host = machine.succeed("garage node id").split('@')
return GarageNode(node_id=node_id, host=host)
def get_node_id(machine: Machine) -> str:
return get_node_fqn(machine).node_id
def get_layout_version(machine: Machine) -> int:
version_data = machine.succeed("garage layout show")
m = cur_version_regex.search(version_data)
if m and m.group('ver') is not None:
return int(m.group('ver')) + 1
else:
raise ValueError('Cannot find current layout version')
def apply_garage_layout(machine: Machine, layouts: List[str]):
for layout in layouts:
machine.succeed(f"garage layout assign {layout}")
version = get_layout_version(machine)
machine.succeed(f"garage layout apply --version {version}")
def create_api_key(machine: Machine, key_name: str) -> S3Key:
output = machine.succeed(f"garage key new --name {key_name}")
m = key_creation_regex.match(output)
if not m or not m.group('key_id') or not m.group('secret_key'):
raise ValueError('Cannot parse API key data')
return S3Key(key_name=key_name, key_id=m.group('key_id'), secret_key=m.group('secret_key'))
def get_api_key(machine: Machine, key_pattern: str) -> S3Key:
output = machine.succeed(f"garage key info {key_pattern}")
m = key_creation_regex.match(output)
if not m or not m.group('key_name') or not m.group('key_id') or not m.group('secret_key'):
raise ValueError('Cannot parse API key data')
return S3Key(key_name=m.group('key_name'), key_id=m.group('key_id'), secret_key=m.group('secret_key'))
def test_bucket_writes(node):
node.succeed("garage bucket create test-bucket")
s3_key = create_api_key(node, "test-api-key")
node.succeed("garage bucket allow --read --write test-bucket --key test-api-key")
other_s3_key = get_api_key(node, 'test-api-key')
assert other_s3_key.secret_key == other_s3_key.secret_key
node.succeed(
f"mc alias set test-garage http://[::1]:3900 {s3_key.key_id} {s3_key.secret_key} --api S3v4"
)
node.succeed("echo test | mc pipe test-garage/test-bucket/test.txt")
assert node.succeed("mc cat test-garage/test-bucket/test.txt").strip() == "test"
def test_bucket_over_http(node, bucket='test-bucket', url=None):
if url is None:
url = f"{bucket}.web.garage"
node.succeed(f'garage bucket website --allow {bucket}')
node.succeed(f'echo hello world | mc pipe test-garage/{bucket}/index.html')
assert (node.succeed(f"curl -H 'Host: {url}' http://localhost:3902")).strip() == 'hello world'
with subtest("Garage works as a single-node S3 storage"):
single_node.wait_for_unit("garage.service")
single_node.wait_for_open_port(3900)
# Now Garage is initialized.
single_node_id = get_node_id(single_node)
apply_garage_layout(single_node, [f'-z qemutest -c 1 "{single_node_id}"'])
# Now Garage is operational.
test_bucket_writes(single_node)
test_bucket_over_http(single_node)
'';
})) args

View File

@ -0,0 +1,54 @@
{ system ? builtins.currentSystem
, config ? { }
, pkgs ? import ../../.. { inherit system config; }
}:
with pkgs.lib;
let
mkNode = package: { replicationMode, publicV6Address ? "::1" }: { pkgs, ... }: {
networking.interfaces.eth1.ipv6.addresses = [{
address = publicV6Address;
prefixLength = 64;
}];
networking.firewall.allowedTCPPorts = [ 3901 3902 ];
services.garage = {
enable = true;
inherit package;
settings = {
replication_mode = replicationMode;
rpc_bind_addr = "[::]:3901";
rpc_public_addr = "[${publicV6Address}]:3901";
rpc_secret = "5c1915fa04d0b6739675c61bf5907eb0fe3d9c69850c83820f51b4d25d13868c";
s3_api = {
s3_region = "garage";
api_bind_addr = "[::]:3900";
root_domain = ".s3.garage";
};
s3_web = {
bind_addr = "[::]:3902";
root_domain = ".web.garage";
index = "index.html";
};
};
};
environment.systemPackages = [ pkgs.minio-client ];
# Garage requires at least 1GiB of free disk space to run.
virtualisation.diskSize = 2 * 1024;
};
in
foldl
(matrix: ver: matrix // {
"basic${toString ver}" = import ./basic.nix { inherit system pkgs; mkNode = mkNode pkgs."garage_${ver}"; };
"with-3node-replication${toString ver}" = import ./with-3node-replication.nix { inherit system pkgs; mkNode = mkNode pkgs."garage_${ver}"; };
})
{}
[
"0_7_3"
"0_8_0"
]

View File

@ -1,50 +1,12 @@
import ./make-test-python.nix ({ pkgs, ...} :
let
mkNode = { replicationMode, publicV6Address ? "::1" }: { pkgs, ... }: {
networking.interfaces.eth1.ipv6.addresses = [{
address = publicV6Address;
prefixLength = 64;
}];
networking.firewall.allowedTCPPorts = [ 3901 3902 ];
services.garage = {
enable = true;
settings = {
replication_mode = replicationMode;
rpc_bind_addr = "[::]:3901";
rpc_public_addr = "[${publicV6Address}]:3901";
rpc_secret = "5c1915fa04d0b6739675c61bf5907eb0fe3d9c69850c83820f51b4d25d13868c";
s3_api = {
s3_region = "garage";
api_bind_addr = "[::]:3900";
root_domain = ".s3.garage";
};
s3_web = {
bind_addr = "[::]:3902";
root_domain = ".web.garage";
index = "index.html";
};
};
};
environment.systemPackages = [ pkgs.minio-client ];
# Garage requires at least 1GiB of free disk space to run.
virtualisation.diskSize = 2 * 1024;
};
in {
name = "garage";
args@{ mkNode, ... }:
(import ../make-test-python.nix ({ pkgs, ...} :
{
name = "garage-3node-replication";
meta = {
maintainers = with pkgs.lib.maintainers; [ raitobezarius ];
};
nodes = {
single_node = mkNode { replicationMode = "none"; };
node1 = mkNode { replicationMode = 3; publicV6Address = "fc00:1::1"; };
node2 = mkNode { replicationMode = 3; publicV6Address = "fc00:1::2"; };
node3 = mkNode { replicationMode = 3; publicV6Address = "fc00:1::3"; };
@ -126,16 +88,6 @@ in {
node.succeed(f'echo hello world | mc pipe test-garage/{bucket}/index.html')
assert (node.succeed(f"curl -H 'Host: {url}' http://localhost:3902")).strip() == 'hello world'
with subtest("Garage works as a single-node S3 storage"):
single_node.wait_for_unit("garage.service")
single_node.wait_for_open_port(3900)
# Now Garage is initialized.
single_node_id = get_node_id(single_node)
apply_garage_layout(single_node, [f'-z qemutest -c 1 "{single_node_id}"'])
# Now Garage is operational.
test_bucket_writes(single_node)
test_bucket_over_http(single_node)
with subtest("Garage works as a multi-node S3 storage"):
nodes = ('node1', 'node2', 'node3', 'node4')
rev_machines = {m.name: m for m in machines}
@ -166,4 +118,4 @@ in {
for node in nodes:
test_bucket_over_http(get_machine(node))
'';
})
})) args

View File

@ -1,19 +1,19 @@
{ lib, stdenv, rustPlatform, fetchFromGitea, openssl, pkg-config, protobuf
, testers, Security, garage }:
rustPlatform.buildRustPackage rec {
, testers, Security, garage, nixosTests }:
let
generic = { version, sha256, cargoSha256, eol ? false }: rustPlatform.buildRustPackage {
pname = "garage";
version = "0.7.3";
inherit version;
src = fetchFromGitea {
domain = "git.deuxfleurs.fr";
owner = "Deuxfleurs";
repo = "garage";
rev = "v${version}";
sha256 = "sha256-WDhe2L+NalMoIy2rhfmv8KCNDMkcqBC9ezEKKocihJg=";
inherit sha256;
};
cargoSha256 = "sha256-5m4c8/upBYN8nuysDhGKEnNVJjEGC+yLrraicrAQOfI=";
inherit cargoSha256;
nativeBuildInputs = [ protobuf pkg-config ];
@ -27,7 +27,16 @@ rustPlatform.buildRustPackage rec {
# on version changes for checking if changes are required here
buildFeatures = [
"kubernetes-discovery"
];
] ++
(lib.optional (lib.versionAtLeast version "0.8") [
"bundled-libs"
"sled"
"metrics"
"k2v"
"telemetry-otlp"
"lmdb"
"sqlite"
]);
# To make integration tests pass, we include the optional k2v feature here,
# but not in buildFeatures. See:
@ -35,16 +44,47 @@ rustPlatform.buildRustPackage rec {
checkFeatures = [
"k2v"
"kubernetes-discovery"
];
] ++
(lib.optional (lib.versionAtLeast version "0.8") [
"bundled-libs"
"sled"
"metrics"
"telemetry-otlp"
"lmdb"
"sqlite"
]);
passthru = {
tests.version = testers.testVersion { package = garage; };
};
passthru = nixosTests.garage;
meta = {
description = "S3-compatible object store for small self-hosted geo-distributed deployments";
homepage = "https://garagehq.deuxfleurs.fr";
license = lib.licenses.agpl3Only;
maintainers = with lib.maintainers; [ nickcao _0x4A6F teutat3s ];
maintainers = with lib.maintainers; [ nickcao _0x4A6F teutat3s raitobezarius ];
knownVulnerabilities = (lib.optional eol "Garage version ${version} is EOL");
};
}
};
in
rec {
# Until Garage hits 1.0, 0.7.3 is equivalent to 7.3.0 for now, therefore
# we have to keep all the numbers in the version to handle major/minor/patch level.
# for <1.0.
garage_0_7_3 = generic {
version = "0.7.3";
sha256 = "sha256-WDhe2L+NalMoIy2rhfmv8KCNDMkcqBC9ezEKKocihJg=";
cargoSha256 = "sha256-5m4c8/upBYN8nuysDhGKEnNVJjEGC+yLrraicrAQOfI=";
};
garage_0_7 = garage_0_7_3;
garage_0_8_0 = generic {
version = "0.8.0";
sha256 = "sha256-c2RhHfg0+YV2E9Ckl1YSc+0nfzbHPIt0JgtT0DND9lA=";
cargoSha256 = "sha256-vITXckNOiJbMuQW6/8p7dsZThkjxg/zUy3AZBbn33no=";
};
garage_0_8 = garage_0_8_0;
garage = garage_0_8;
}

View File

@ -7355,9 +7355,12 @@ with pkgs;
gaphor = python3Packages.callPackage ../tools/misc/gaphor { };
garage = callPackage ../tools/filesystems/garage {
inherit (callPackage ../tools/filesystems/garage {
inherit (darwin.apple_sdk.frameworks) Security;
};
})
garage
garage_0_7 garage_0_8
garage_0_7_3 garage_0_8_0;
garmin-plugin = callPackage ../applications/misc/garmin-plugin {};