Commit Graph

15181 Commits

Author SHA1 Message Date
Graham Christensen
82578fc725
Merge pull request #102172 from grahamc/stage-1-datestamps
stage-1: add datestamps to logs
2020-10-30 16:13:02 -04:00
Graham Christensen
b34cf366aa
Merge pull request #102171 from grahamc/faster-ext-resize
stage-1: modprobe ext{2,3,4} before resizing (so resizing takes less than 45 minutes)
2020-10-30 16:12:50 -04:00
WORLDofPEACE
214af51225
Merge pull request #101067 from deviant/remove-caddy-agree
nixos/caddy: remove services.caddy.agree
2020-10-30 16:02:44 -04:00
Graham Christensen
ece5c0f304
stage-1: modprobe ext{2,3,4} before resizing
I noticed booting a system with an ext4 root which expanded to 5T took
quite a long time (12 minutes in some cases, 43(!) in others.)

I changed stage-1 to run `resize2fs -d 62` for extra debug output and
timing information. It revealed the adjust_superblock step taking
almost all of the time:

    [Fri Oct 30 11:10:15 UTC 2020] zero_high_bits_in_metadata: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
    [Fri Oct 30 11:21:09 UTC 2020] adjust_superblock: Memory used: 396k/4556k (295k/102k), time: 654.21/ 0.59/ 5.13

but when I ran resize2fs on a disk with the identical content growing
to the identical target size, it would only take about 30 seconds. I
looked at what happened between those two steps in the fast case with
strace and found:

```
   235	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=1795}, ru_stime={tv_sec=0, tv_usec=3590}, ...}) = 0
   236	write(1, "zero_high_bits_in_metadata: Memo"..., 84zero_high_bits_in_metadata: Memory used: 132k/0k (72k/61k), time:  0.00/ 0.00/ 0.00
   237	) = 84
   238	gettimeofday({tv_sec=1604061278, tv_usec=480147}, NULL) = 0
   239	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=1802}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   240	gettimeofday({tv_sec=1604061278, tv_usec=480192}, NULL) = 0
   241	mmap(NULL, 2564096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa3c7355000
   242	access("/sys/fs/ext4/features/lazy_itable_init", F_OK) = 0
   243	brk(0xf85000)                           = 0xf85000
   244	brk(0xfa6000)                           = 0xfa6000
   245	gettimeofday({tv_sec=1604061278, tv_usec=538828}, NULL) = 0
   246	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=58720}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   247	write(1, "adjust_superblock: Memory used: "..., 79adjust_superblock: Memory used: 396k/2504k (305k/92k), time:  0.06/ 0.06/ 0.00
   248	) = 79
   249	gettimeofday({tv_sec=1604061278, tv_usec=539119}, NULL) = 0
   250	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=58812}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   251	gettimeofday({tv_sec=1604061279, tv_usec=939}, NULL) = 0
   252	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=520411}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   253	write(1, "fix_uninit_block_bitmaps 2: Memo"..., 88fix_uninit_block_bitmaps 2: Memory used: 396k/2504k (305k/92k), time:  0.46/ 0.46/ 0.00
   254	) = 88
```

In particular the access to /sys/fs seemed interesting. Looking
at the source of resize2fs:

```
[root@ip-172-31-22-182:~/e2fsprogs-1.45.5]# rg -B2 -A1 /sys/fs/ext4/features/lazy_itable_init .
./resize/resize2fs.c
923-	if (getenv("RESIZE2FS_FORCE_LAZY_ITABLE_INIT") ||
924-	    (!getenv("RESIZE2FS_FORCE_ITABLE_INIT") &&
925:	     access("/sys/fs/ext4/features/lazy_itable_init", F_OK) == 0))
926-		lazy_itable_init = 1;
```

I confirmed /sys is mounted, and then found a bug suggesting the
ext4 module is maybe not loaded:
https://bugzilla.redhat.com/show_bug.cgi?id=1071909

My home server doesn't have ext4 loaded and had 3T to play with, so
I tried (and succeeded with) replicating the issue locally:

```
[root@kif:/scratch]# lsmod | grep -i ext

[root@kif:/scratch]# zfs create -V 3G rpool/scratch/ext4

[root@kif:/scratch]# time mkfs.ext4 /dev/zvol/rpool/scratch/ext4
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: 560a4a8f-93dc-40cc-97a5-f10049bf801f
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

real	0m2.261s
user	0m0.000s
sys	0m0.025s

[root@kif:/scratch]# zfs set volsize=3T rpool/scratch/ext4

[root@kif:/scratch]# time resize2fs -d 62 /dev/zvol/rpool/scratch/ext4
resize2fs 1.45.5 (07-Jan-2020)
fs has 11 inodes, 1 groups required.
fs requires 16390 data blocks.
With 1 group(s), we have 22234 blocks available.
Last group's overhead is 10534
Need 16390 data blocks in last group
Final size of last group is 26924
Estimated blocks needed: 26924
Extents safety margin: 49
Resizing the filesystem on /dev/zvol/rpool/scratch/ext4 to 805306368 (4k) blocks.
read_bitmaps: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
read_bitmaps: I/O read: 1MB, write: 0MB, rate: 3802.28MB/s
fix_uninit_block_bitmaps 1: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
resize_group_descriptors: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
move_bg_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
zero_high_bits_in_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
```

here it got stuck for quite some time ... straceing this 20 minutes in revealed this in a tight loop:

```
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
fallocate(3, FALLOC_FL_ZERO_RANGE, 2222649901056, 2097152) = 0
fsync(3)                                = 0
```

it finally ended 43(!) minutes later:

```
adjust_superblock: Memory used: 264k/3592k (210k/55k), time: 2554.03/ 0.16/15.07
fix_uninit_block_bitmaps 2: Memory used: 264k/3592k (210k/55k), time:  0.16/ 0.16/ 0.00
blocks_to_move: Memory used: 264k/3592k (211k/54k), time:  0.00/ 0.00/ 0.00
Number of free blocks: 755396/780023556, Needed: 0
block_mover: Memory used: 264k/3592k (216k/49k), time:  0.05/ 0.01/ 0.00
block_mover: I/O read: 1MB, write: 0MB, rate: 18.68MB/s
inode_scan_and_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
inode_ref_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
move_itables: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
calculate_summary_stats: Memory used: 264k/3592k (216k/49k), time: 16.35/16.35/ 0.00
fix_resize_inode: Memory used: 264k/3592k (222k/43k), time:  0.04/ 0.00/ 0.00
fix_resize_inode: I/O read: 1MB, write: 0MB, rate: 22.80MB/s
fix_sb_journal_backup: Memory used: 264k/3592k (222k/43k), time:  0.00/ 0.00/ 0.00
overall resize2fs: Memory used: 264k/3592k (222k/43k), time: 2570.90/16.68/15.07
overall resize2fs: I/O read: 1MB, write: 1MB, rate: 0.00MB/s
The filesystem on /dev/zvol/rpool/scratch/ext4 is now 805306368 (4k) blocks long.

real	43m1.943s
user	0m16.761s
sys	0m15.069s
```

I then cleaned up and recreated the zvol, loaded the ext4 module, created the ext4 fs,
resized the volume, and resize2fs'd and it went quite quickly:

```
[root@kif:/scratch]# zfs destroy rpool/scratch/ext4

[root@kif:/scratch]# zfs create -V 3G rpool/scratch/ext4

[root@kif:/scratch]# modprobe ext4

[root@kif:/scratch]# time resize2fs -d 62 /dev/zvol/rpool/scratch/ext4

[root@kif:/scratch]# time mkfs.ext4 /dev/zvol/rpool/scratch/ext4
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: 5b415f2f-a8c4-4ba0-ac1d-78860de77610
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

real	0m1.013s
user	0m0.001s
sys	0m0.023s

[root@kif:/scratch]# zfs set volsize=3T rpool/scratch/ext4

[root@kif:/scratch]# time resize2fs -d 62 /dev/zvol/rpool/scratch/ext4
resize2fs 1.45.5 (07-Jan-2020)
fs has 11 inodes, 1 groups required.
fs requires 16390 data blocks.
With 1 group(s), we have 22234 blocks available.
Last group's overhead is 10534
Need 16390 data blocks in last group
Final size of last group is 26924
Estimated blocks needed: 26924
Extents safety margin: 49
Resizing the filesystem on /dev/zvol/rpool/scratch/ext4 to 805306368 (4k) blocks.
read_bitmaps: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
read_bitmaps: I/O read: 1MB, write: 0MB, rate: 3389.83MB/s
fix_uninit_block_bitmaps 1: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
resize_group_descriptors: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
move_bg_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
zero_high_bits_in_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
adjust_superblock: Memory used: 264k/1540k (210k/55k), time:  0.02/ 0.02/ 0.00
fix_uninit_block_bitmaps 2: Memory used: 264k/1540k (210k/55k), time:  0.15/ 0.15/ 0.00
blocks_to_move: Memory used: 264k/1540k (211k/54k), time:  0.00/ 0.00/ 0.00
Number of free blocks: 755396/780023556, Needed: 0
block_mover: Memory used: 264k/3592k (216k/49k), time:  0.01/ 0.01/ 0.00
block_mover: I/O read: 1MB, write: 0MB, rate: 157.11MB/s
inode_scan_and_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
inode_ref_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
move_itables: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00

calculate_summary_stats: Memory used: 264k/3592k (216k/49k), time: 16.20/16.20/ 0.00
fix_resize_inode: Memory used: 264k/3592k (222k/43k), time:  0.00/ 0.00/ 0.00
fix_resize_inode: I/O read: 1MB, write: 0MB, rate: 5319.15MB/s
fix_sb_journal_backup: Memory used: 264k/3592k (222k/43k), time:  0.00/ 0.00/ 0.00
overall resize2fs: Memory used: 264k/3592k (222k/43k), time: 16.45/16.38/ 0.00
overall resize2fs: I/O read: 1MB, write: 1MB, rate: 0.06MB/s
The filesystem on /dev/zvol/rpool/scratch/ext4 is now 805306368 (4k) blocks long.

real	0m17.908s
user	0m16.386s
sys	0m0.079s
```

Success!
2020-10-30 12:18:23 -04:00
Graham Christensen
a179781696
stage-1: add datestamps to logs
When the stage-1 logs get imported in to the journal, they all get
loaded with the same timestamp. This makes it difficult to identify
what might be taking a long time in early boot.
2020-10-30 12:16:35 -04:00
Timo Kaufmann
83f48e8348
Merge pull request #95011 from Atemu/undervolt-pl
undervolt: expose power limits as Nixopts
2020-10-30 09:32:50 +01:00
Michele Guerini Rocco
1102a46ffe
Merge pull request #101724 from pickfire/patch-3
fontdir: add ttc to font regex
2020-10-30 08:41:34 +01:00
Benjamin Hipple
e00752079e
Merge pull request #102018 from 1000101/blockbook-frontend
blockbook-frontend: fix&update extraConfig example
2020-10-29 22:30:07 -04:00
Florian Klink
b8d59e93c8 nixos/networkd: allow RouteMetric= in [DHCPv6] section 2020-10-29 19:47:42 +01:00
1000101
4b8611c959 blockbook-frontend: fix&update extraConfig example 2020-10-29 11:41:41 +01:00
Philipp
fc856b89e5
nixos/murmur: add murmur group, don't run as nogroup
fixes #101980
2020-10-29 10:32:04 +01:00
Martin Weinelt
55746e0a4b
Merge pull request #98187 from mweinelt/nixos/babeld
nixos/babeld: lock down service
2020-10-29 01:24:11 +01:00
Minijackson
3fce272478 nixos/shiori: harden service with systemd 2020-10-28 20:46:30 +01:00
Linus Heckemann
2b06415ca1
Merge pull request #101370 from m1cr0man/ssl-test-certs
nixos/acme: Permissions and tests fixes
2020-10-28 17:21:57 +01:00
davidak
4166a767de doc: improve 20.09 release notes 2020-10-27 21:11:22 +01:00
AmineChikhaoui
8cae6703ef
ec2-amis: add stable NixOS 20.09 AMIs
Fixes #101694
2020-10-27 08:52:15 -04:00
Ryan Mulligan
178d373a8a
Merge pull request #83687 from primeos/wshowkeys
wshowkeys: init at 2020-03-29
2020-10-26 18:55:16 -07:00
WORLDofPEACE
ace69f768b Revert "nixos/pantheon: install nixos wallpaper"
This reverts commit 5100e4f250.

Fixes https://github.com/NixOS/nixpkgs/issues/100293
Though it's only a workaround for now.
See https://github.com/elementary/switchboard-plug-pantheon-shell/issues/246#issuecomment-716713218
We trigger the broken scenario where we have two subdirectories. Reverting
that commit undoes this.
2020-10-26 13:45:19 -04:00
Nick Hu
921287e7f0
Merge pull request #97726 from NickHu/pam_gnupg
pam: add support for pam_gnupg
2020-10-26 15:27:13 +00:00
Andreas Rammhold
1088f05940
Merge pull request #101598 from andir/nixos-build-vms-qemu
nixos/tests: follow-up to the closure reduction PR
2020-10-26 14:19:45 +01:00
rnhmjoj
9e04bba0af nixos/dnscrypt-wrapper: fix key rotation script
Fix an error in the validation code when the public key is in a
nonstandard location. The check command fails and the key is
incorrectly assumed to be expiring.
2020-10-26 13:07:49 +01:00
Jörg Thalheim
dfaa313d43
Merge pull request #101737 from aneeshusa/nginx-allow-unsetting-ssl_ciphers
nixos/nginx: Allow unsetting ssl_ciphers
2020-10-26 06:41:19 +01:00
Aneesh Agrawal
924035bb97 nixos/nginx: Allow unsetting ssl_ciphers
When using the Modern config from the Mozilla SSL config generator,
the `ssl_ciphers` parameter does not need to be set
as only TLSv1.3 is permitted and all of its ciphers are reasonable.
2020-10-26 00:35:29 -04:00
Klemens Nanni
0b8a6e787c
nixos/avahi: Enable IPv6 by default
Treat it the same as IPv4 (I'm tempted to disable IPv4 by default);
this is the only option I still need to set manually to enjoy IPv6-only
networks including printer discovery!
2020-10-26 04:06:26 +01:00
Ivan Tham
f6136d06ff
fontdir: add ttc to font regex
.ttc fonts are used by noto-fonts-cjk
2020-10-26 10:45:22 +08:00
Klemens Nanni
3216b85713 nixos/system-path: Add mkpasswd(1)
Generating password hashes, e.g. when adding new users to the system
configuration, should work out-of-the-box and offline.
2020-10-26 03:40:11 +01:00
WORLDofPEACE
4d71306596
Merge pull request #101516 from worldofpeace/gnome-polishing
GNOME polishing from Q.A findings
2020-10-25 18:41:34 -04:00
Andreas Rammhold
d4fb7daafd
nixos-build-vms: use the driverInteractive attribute instead
This reverts commit aab534b894 & uses the
driverInteractive attribute for the test driver instead.

This has the same effect but removes the extra module in the
nixos-build-vms code.
2020-10-25 20:14:53 +01:00
Maximilian Bosch
a3041ab124
Merge pull request #101645 from andir/qemu-tests-fixup
nixos/tests: only apply qemu parameters if the options are defined
2020-10-25 19:25:50 +01:00
Benjamin Hipple
f98312fcb5
Merge pull request #79759 from lopsided98/syncoid-no-root
nixos/syncoid: automatically setup privilege delegation
2020-10-25 10:40:33 -04:00
Andreas Rammhold
f4d7493162
nixos/tests: only apply qemu parameters if the options are defined
This fixes an eval error that occurred on hydra with the small channel
and the `nixos.tests.boot.biosCdrom.x86_64-linux` attribute:

> $ nix-instantiate nixos/release-small.nix -A nixos.tests.boot.biosCdrom.x86_64-linux
> warning: unknown setting 'experimental-features'
> error: The option `virtualisation.qemu' does not exist. Definition values:
> - In `/home/andi/dev/nixos/nixpkgs/nixos/modules/testing/test-instrumentation.nix':
>     {
>       consoles = [ ];
>       package = {
> 	_type = "override";
> 	content = <derivation /nix/store/q72h2cdcb9zjgiay5gdgzwddjkbjr7xq-qemu-host-cpu-only-for-vm-tests-5.1.0.drv>;
>     ...
> (use '--show-trace' to show detailed location information)

In bc2188b we changed test test-instrumentation to also set the QEMU
package that is being used. That change unfortunately caused us to
always assing values to the virtualisation.qemu.package option even when
the option is not defined. The original code was explicitly testing for
the consoles case but the then newly extended version did not adjust the
check as the intention was probably not clear.

With this commit we are always ensuring the entire virtualisation.qemu
section exists and can thus drop the individual tests for each of the
sections since the QEMU module always defines both the package and the
consoles option when it's root is defined..
2020-10-25 13:42:01 +01:00
Felix Tenley
542f75079b nixos/mosquitto: add passwordFile and hashedPasswordFile options 2020-10-25 10:53:38 +01:00
Gabriel Ebner
a8a018ddc0
Merge pull request #101409 from rycee/dbus-warning 2020-10-25 09:16:28 +01:00
Aaron Andersen
a160fa008d
Merge pull request #100063 from aanderse/nixos/powerdns
nixos/powerdns: use upstream systemd unit
2020-10-24 18:47:43 -04:00
Andreas Rammhold
250fb4611f
Merge pull request #100456 from maralorn/boolToString
treewide: De-inline uses of lib.boolToString
2020-10-25 00:45:11 +02:00
Robert Helgesson
94819fdb5f
nixos/dbus: re-add a dummy socketActivated option
If set, then issue a warning instead of an error as previously.
2020-10-24 23:01:00 +02:00
Aaron Andersen
4f5d3794d3 nixos/powerdns: use upstream systemd unit 2020-10-24 16:40:20 -04:00
WORLDofPEACE
6bc94d149b
Merge pull request #101563 from worldofpeace/fix-pantheon-greeter-brightness
nixos/lightdm: make lightdm user shell bash
2020-10-24 11:56:34 -04:00
WORLDofPEACE
ef803ab1bb
Merge pull request #100199 from worldofpeace/seeded-config
nixos/tools: add desktopConfiguration option (to seed configuration into configuration.nix)
2020-10-24 11:35:33 -04:00
WORLDofPEACE
c134f6443a nixos/lightdm: make lightdm user shell bash
In https://github.com/NixOS/nixpkgs/issues/100119 pantheon's greeter
has g-s-d running which allows brightness controls via pkexec.
This is changed in newer versions of g-s-d (pantheon uses a fork currently),
but whenever brightness is changed with a shell of `shadow` we get
```
Oct 10 23:51:44 kirXps pkexec[18722]: lightdm: Executing command [USER=root] [TTY=unknown] [CWD=/var/lib/lightdm] [COMMAND=/run/current-system/sw/bin/elementary-settings-daemon/gsd-backlight-helper /sys/devices/pci0000:00/0000:00:02.0/drm/card0/card0-eDP-1/intel_backlight 65587]
```

I'm not sure this should be strictly needed, so we should try to
revert later on when pantheon's g-s-d is updated.
2020-10-24 11:28:18 -04:00
WORLDofPEACE
39d1599767 installation-cd-graphical-gnome: add firefox to favorite-apps 2020-10-24 11:14:44 -04:00
WORLDofPEACE
7df6af303e nixos/gnome3: add gnome-calendar to favorites 2020-10-24 11:14:41 -04:00
WORLDofPEACE
9cee7772e6 nixos/gnome3: add favoriteAppsOverride option
Rather messy and only needed for the installation cd, so it's
an internal option.
2020-10-24 11:14:22 -04:00
WORLDofPEACE
d89deddd5d nixos/flatpak: introduce guiPackages
This adds basically an indirection to systemPackages
to automatically install an interface for flatpak for their respective
environments. e.g if I enable pantheon and flatpak you'll get appcenter,
and on gnome you'll see gnome-software.

https://github.com/NixOS/nixpkgs/issues/99648#issuecomment-706691174
2020-10-24 11:14:02 -04:00
WORLDofPEACE
b1587f9e19 nixos/gnome3: don't ship gnome-software
This serves no purpose without flatpak https://github.com/NixOS/nixpkgs/issues/99648#issuecomment-706691174
2020-10-24 11:14:01 -04:00
Maximilian Bosch
48612c79b1
Merge pull request #101473 from Ma27/nixos-build-vms-qemu
nixos/nixos-build-vms: use `pkgs.qemu` for virtualisation
2020-10-24 14:43:52 +02:00
WORLDofPEACE
70dc25abd9 nixos/gnome3: don't put epiphany in favorite apps 2020-10-23 20:20:07 -04:00
Doron Behar
c90450014f
Merge pull request #101480 from Flakebi/salt 2020-10-24 01:31:41 +03:00
Jan Tojnar
3a73543401
Merge pull request #93725 from nglen/pipewire 2020-10-24 00:05:33 +02:00
WilliButz
993437d0d6
Merge pull request #96511 from Zopieux/rtl_433_prom
Add rtl_433 Prometheus exporter
2020-10-23 23:24:38 +02:00