KDE things may fail in almost undebuggable ways.
With "firehose log level" I saw this message for files that were
present:
9eb46f9b20/src/sycoca/kbuildsycoca.cpp (L636)
This, in turn, is because "hash" is computed as a sum of all modified
dates
9eb46f9b20/src/sycoca/kbuildsycoca.cpp (L610)
(Weird hashing!)
Which, in turn, meant that it was `0` here.
Finally, `0` ends up magically breaking everything in a plasma mobile
system. Since apparently desktop files will not even be attempted to be
read or used if they are not cached by ksycoca.
The task panel broke in inscrutable ways, and there were no applications
found to run.
fixes https://github.com/NixOS/mobile-nixos/issues/372
Per https://github.com/NixOS/mobile-nixos/issues/372, I observed
building an image takes ~40 minutes, and it spends the vast majority
of that performing `du` on the files that will be included in the image.
Most of that time is on process invocation overhead, and thus we can
save a lot by batching calls to `du`.
Per https://danielmiessler.com/blog/linux-xargs-vs-exec/, replacing
`find -exec du` with `find -print0 | xargs -0 du` is an optimisation to do
this. xargs will invoke du as "du fn1 fn2 fn3 fn4 ...", etc.
Before: 44mins (full output: https://paste.debian.net/1203075/ )
After: 1m45s (full output: https://paste.debian.net/1203076/ )
To reproduce this issue on a smaller dataset, try the before and after
on a directory with many files (e.g. nixpkgs checkout, which has ~27k files):
```
$ cd nixpkgs
$ time find . ! -type d -exec du {} \; | cut -f1 | sum
53288 55
real 0m20.802s
user 0m9.660s
sys 0m11.626s
$ time find . ! -type d -print0 | xargs -0 du | cut -f1 | sum
53288 55
real 0m0.097s
user 0m0.042s
sys 0m0.171s
```
That's a 200x speedup.
To discount the possibility of this issue being `find` or file I/O related:
```
$ time bash -c 'seq 27000 | xargs /run/current-system/sw/bin/echo | wc'
2 27000 150894
real 0m0.008s
user 0m0.006s
sys 0m0.006s
$ time bash -c 'for i in $(seq 27000); do /run/current-system/sw/bin/echo $i; done | wc'
27000 27000 150894
real 0m19.815s
user 0m9.876s
sys 0m10.755s
```
The ~27k process invocation still takes ~20s.
This shows the problem is process invocation.
Is this safe?
* Both before and after produce the same sector result: "926478
sectors for files" and "21491 sectors for directories"
* `-print0` and `-0` are needed to ensure this works with filenames
that contain spaces.
This is required for more hermetic evals.
A simple thing to try is, before this change, replace `default.nix` with
`throw "No thanks..."`. It would throw. It was also possible to observe
`local.nix` was being included by the warnings.
Wioth this change, `release.nix` does not include `local.nix` through
`default.nix`.
I think this was the last piece of Mobile NixOS that actually relied on
`default.nix` being a thing. We have finally completely inverted the
control, where `default.nix` uses the helpers, rather than the helpers
evaluating a specialized `default.nix`.
From now on, it should be entirely safe to experiment with
`default.nix`. We should be able to **fail noisily** when a user builds
the default empty configuration!
Recent changes in Nixpkgs makes it more strict.
This is a change that wasn't caught as needed from the migration to
structured options for device infos.
Instead of the following error message:
```
error: The option `mobile.device.info' defined in `<unknown-file>' does not exist.
(use '--show-trace' to show detailed location information)
```
We now get:
```
error: The option `mobile.device.info' defined in `.../mobile-nixos/devices/asus-z00t' does not exist.
(use '--show-trace' to show detailed location information)
```
We are now splitting the concerns more clearly, by making `baseModules`
the reason we are evaluating NixOS, rather than wholesale always
including it into the modules.
I *think* there is a new check that wasn't previously (in 19.03) part of
e2fsck. This is now causing the check to fail.
FIXME : figure out a way to disable the check, or make the image comply.
This creates a disk image that acts *just about* like how the NixOS
sd_image acts. Though it is much less useful as there is no system, not
even an initrd.
It turns out the warnings it spits out when forcing FAT32 with too few
clusters is not for nothing. At the very least the Raspberry Pi 3B does
not like the forced FAT 32. Though, it was validated that a large enough
FAT32 partition is fine with the Raspberry Pi 3B.
The algorithms inside `make_ext4fs` can be followed, but it ends up
being a bit complex. I did not figure out all variables, but the amount
of them made me reluctant to implement it as a complte formula.
Instead, I looked at the actual usable space using `df` and mapped it in
a spreadsheet. With the knowledge from actively looking at the source
code, and other data, it is known that the lookup table will work, while
not be ideal.
The fudge factor starting at 256MiB is about stable, but there is a
slight downward deviation at 512MiB, which is why 512MiB was used. The
downward deviation was not observed in other values.
Here's the table as computed.
```
MIB Fudge
5 0.84609375
8 0.5419921875
16 0.288818359375
32 0.1622314453125
64 0.09893798828125
128 0.067291259765625
256 0.0518646240234375
512 0.05208587646484375
1024 0.048187255859375
2048 0.04060554504394531
4096 0.03718090057373047
```
The difference from .52% to .37% is negligible for 0.5 vs. 4 GiB is
annoying me. The lookup table could be changed to include all known
values instead, I guess.
The factor was derived from first making a few test images, finding
there was a common thread, and then looking at what parts of FAT32 were
generated, and how they were generated. That was fun. (Yes, really.)
In addition, present the sector size as an option.
The sector size is required to set a block size in bytes, so giving the
option to the end-user is probably for the best.
The values here were chosen to keep the builder compatible with the
existing tests.
In addition, the values are quite optimized for smaller filesystem
images, e.g. building a FIRMWARE partition for the Raspberry Pi.
First of all, `du -s` will always combine the sizes into the multiple of
its given --block-size, so 4×1 byte files will show up as 1 block size
(assuming a block size > 4). The expected size would have been at least
one block size per file, so for 512, 512×4 thus 2048.
With this change, we sum the sizes ourselves.
This adds a new requirement to the filesystems, that they pass the block
size to the generic builder, though this is good, as it will increases
reproducibility. In addition, filesystems can present the option to
users, allowing the user to set their block sizes as expected.
