Commit Graph

168 Commits

Author SHA1 Message Date
Tuomas Tynkkynen
71631a922b runInLinuxVM: Use QEMU command line that works on other architectures
... by moving the existing definition to qemu-flags.nix and reusing
that.
2018-01-30 16:57:27 +02:00
Tuomas Tynkkynen
8e83158f12 nixos/qemu: Deduplicate QEMU serialDevice into qemu-flags.nix 2018-01-30 16:57:27 +02:00
Tuomas Tynkkynen
1ce1380497 nixos/qemu-flags: Take a 'pkgs' parameter
I'm gonna use it in the next commit.
2018-01-30 16:57:27 +02:00
Tuomas Tynkkynen
962e79ef32 nixos/make-disk-image.nix: Support EFI images
- Add a new parameter `imageType` that can specify either "efi" or
  "legacy" (the default which should see no change in behaviour by
  this patch).

- EFI images get a GPT partition table (instead of msdos) with a
  mandatory ESP partition (so we add an assert that `partitioned`
  is true).

- Use the partx tool from util-linux to determine exact start + size
  of the root partition. This is required because GPT stores a secondary
  partition table at the end of the disk, so we can't just have
  mkfs.ext4 create the filesystem until the end of the disk.

- (Unrelated to any EFI changes) Since we're depending on the
  `-E offset=X` option to mkfs which is only supported by e2fsprogs,
  disallow any attempts of creating partitioned disk images where
  the root filesystem is not ext4.
2018-01-22 11:18:23 +02:00
John Ericson
eec050f395
Merge pull request #33577 from dtzWill/fix/cross-2
Minor cross fixes, 2
2018-01-09 12:36:53 -05:00
Ben Gamari
a2215da9a1 make-ext4-fs: Dependencies are nativeBuildInputs 2018-01-09 11:25:19 -05:00
Ben Gamari
d680678d0a makeSquashfs: Inputs are nativeBuildInputs 2018-01-09 11:25:19 -05:00
Jörg Thalheim
814b38541f
Merge pull request #32845 from Ma27/test-driver/allow-user-units
test-driver: support testing user units
2018-01-04 09:40:41 +01:00
Maximilian Bosch
e538e00404
test-driver: support testing user units
It is quite complicated to test services using the test-driver when
declaring user services with `systemd.user.services` such as many
X11-based services like `xautolock.service`.

This change adds an optional `$user` parameter to each systemd-related
function in the test-driver and runs `systemctl --user` commands using
`su -l $user -c ...` and sets the `XDG_RUNTIME_DIR` variable
accordingly and a new function named `systemctl` which is able to run a
systemd command with or without a specified user.

The change can be confirmed with a simple VM declaration like this:

```
import ./nixos/tests/make-test.nix ({ pkgs, lib }:

with lib;

{
  name = "systemd-user-test";

  nodes.machine = {
    imports = [ ./nixos/tests/common/user-account.nix ];

    services.xserver.enable = true;
    services.xserver.displayManager.auto.enable = true;
    services.xserver.displayManager.auto.user = "bob";
    services.xserver.xautolock.enable = true;
  };

  testScript = ''
    $machine->start;
    $machine->waitForX;

    $machine->waitForUnit("xautolock.service", "bob");
    $machine->stopJob("xautolock.service", "bob");
    $machine->startJob("xautolock.service", "bob");
    $machine->systemctl("list-jobs --no-pager", "bob");
    $machine->systemctl("show 'xautolock.service' --no-pager", "bob");
  '';
})
```
2018-01-02 20:14:27 +01:00
Tuomas Tynkkynen
758b4c1ea4 make-disk-image.nix: Really fix write permissions in the store
I think the current one applies the -exec only to those that match
'-type d'. Let's switch it to something that humans can understand...
2017-12-20 17:28:46 +02:00
Tuomas Tynkkynen
c9f71974f8 make-disk-image.nix: Remove write permissions from /nix/store
Fakeroot seems to always give the owner write bit to any files touched
inside it (presumably to easily simulate the fact that root can still
modify such files). So do an explicit chmod to remove them.

This should finally solve #32242 after the EC2 images are regenerated
with this change.

https://hydra.nixos.org/build/66143116
2017-12-17 01:19:05 +02:00
Tuomas Tynkkynen
6ed6d323fe make-squashfs.nix: Improve invalid path check messages
Now the mtime problems are gone, but EC2 Hydra builders are still having
some problems:

https://hydra.nixos.org/build/66043835
2017-12-15 19:57:40 +02:00
Tuomas Tynkkynen
0d27df280f build-support/vm: Use devtmpfs, not static device nodes
In 2017, there is no reason to create a static /dev.
2017-12-12 14:31:50 +02:00
Tuomas Tynkkynen
574526d510 make-squashfs: Hack to avoid building images where nixos-install won't run
https://github.com/NixOS/nixpkgs/issues/32242
2017-12-03 03:37:45 +02:00
Tuomas Tynkkynen
8762490337 nixos/qemu: Add virtio-rng-pci to test-driver
Otherwise AArch64 KVM guests don't have enough entropy at boot.
2017-11-26 11:13:21 +02:00
Orivej Desh
50d8720d2a nixos/testing: fix after --set ceased to evaluate the value 2017-11-19 14:15:19 +00:00
Orivej Desh
470a7752d8 make-disk-image: fix eval 2017-11-13 14:56:30 +00:00
Orivej Desh
27f5cc959b make-disk-image: embed compression switch into the format name 2017-11-12 18:39:36 +00:00
Orivej Desh
608de48fe4 make-disk-image: add compressed option to compress qcow2 images 2017-11-12 18:28:06 +00:00
Orivej Desh
a2d3ee77b8 make-disk-image: clean up 2017-11-12 18:24:26 +00:00
Orivej Desh
a84ce72fdc make-disk-image: do not use faketime
Since https://github.com/lkl/linux/pull/394 cptofs preserves the source time,
which is 1970-01-01T00:00:01Z for /nix/store and recent for other files.

This reverts commit f5b3f2c5a7.
2017-11-11 20:57:14 +00:00
Orivej Desh
f218ef647c make-disk-image: clean up 2017-11-11 20:24:26 +00:00
Dan Peebles
f5b3f2c5a7 make-disk-iamge: use faketime to get a consistent timestamp
This fixes #28768 because during an image build, Nix sees bad store
timestamps and attempts to fix them, but can't fix them on a running
system (due to being inside a builder). Since timestamps on the store
are supposed to be 1 anyway, if we fix this, that fixes image building
inside booted images made this way.

Note that this adds quite a bit of noise to the output, because running
`cptofs` under `faketime` causes a bunch of seemingly spurious error
messages and my attempts to suppress them all failed. We'll fix it when
`cptofs` gets a native timestamp preservation feature.
2017-11-10 05:15:37 +00:00
Joerg Thalheim
00344162a7 Revert "nixos/test-instrumentation: softcode DISPLAY"
This reverts commit 4531ec5955.

This broke waitForX: b11ac5d29d (commitcomment-25486347)

cc @Lassulus
2017-11-08 17:04:39 +00:00
lassulus
4531ec5955 nixos/test-instrumentation: softcode DISPLAY 2017-11-07 23:01:59 +01:00
Peter Simons
757a759005 Merge pull request #30166 from LumiGuide/graphite-1.0.2
Fix graphite crash by upgrading from 0.9.15 -> 1.0.2
2017-10-18 12:58:55 +01:00
Bjørn Forsman
0ff4bb5f87 nixos: run parted with --script option
-s, --script: never prompts for user intervention

Sometimes the NixOS installer tests fail when they invoke parted, e.g.
https://hydra.nixos.org/build/62513826/nixlog/1. But instead of exiting
right there, the tests hang until the Nix builder times out (and kills
the build). With this change the tests would instead fail immediately,
which is preferred.

While at it, use "parted --script" treewide, so nobody gets build
timeout due to parted error (or misuse). (Only nixos/ use it, and only
non-interactive.)

A few instances already use the short option "-s", convert them to long
option "--short".
2017-10-14 15:29:02 +02:00
aszlig
20487112ed
nixos: Fix output path generation of runInMachine
Regression introduced by a02bb00156.

The fix is done by disabling writableStore, because the latter will set
up an overlayfs on the Nix store within the VM, which in turn will
discard all the outputs of the resulting output path.

However in runInMachine we actually *want* the contents of the generated
path and also don't want a writable store within the VM (except of
course for $out, which is writable anyway).

I've added a small regression test to verifify the output in
nixos/tests/run-in-machine.nix to make sure this won't break again in
the future.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2017-10-11 20:08:11 +02:00
Bas van Dijk
5b8ff5ed49 graphite: 0.9.15 -> 1.0.2
Fixes: #29961

Also added the option:

  services.graphite.web.extraConfig

for configuring graphite_web.
2017-10-08 03:03:22 +02:00
Dan Peebles
e2589b3ca2 Deduplicate some filterSource invocations
This version should have more conventional regexes that work across many
platforms and regex engines. This is an issue because up until Nix 1.11,
Nix called out to the libc regex matcher, which behaved differently on
Darwin and Linux. And in Nix 1.12, we're moving to std::regex which will
also behave differently here.

And yes, I do actually evaluate make-disk-image.nix on Darwin ;)
2017-08-29 20:27:04 -04:00
Dan Peebles
5f372ef67f amazon-image: support VPC/VHD as an output format for the builder 2017-08-10 22:57:26 +00:00
Tuomas Tynkkynen
1d72474df7 make-ext4-fs: Fix modification timestamps in image
Use '-f' to make timestamp calls to always return the same time. Also
Nix uses the timestamp of '1' instead of epoch directly.
2017-07-29 22:03:43 +03:00
Graham Christensen
3f40fcabbf
nixos tests: waitForWindow: output a list of windows we see prior to the final check
machine: must succeed: xwininfo -root -tree | sed 's/.*0x[0-9a-f]* \"\([^\"]*\)\".*/\1/; t; d'
    machine: exit status 0
    machine: Last chance to match /(?^:dfiirst configuration)/ on the the window list, which currently contains:
    machine: [i3 con] container around 0xf8a5f0, i3: first configuration, [i3 con] floatingcon around 0xf8c260, [i3 con] container around 0xf8a380, i3bar for output Virtual-1, [i3 con] bottom dockarea Virtual-1, [i3 con] workspace 1, [i3 con] content Virtual-1, [i3 con] top dockarea Virtual-1, [i3 con] output Virtual-1, [i3 con] workspace __i3_scratch, [i3 con] content __i3, [i3 con] pseudo-output __i3, i3
2017-06-20 21:16:35 -04:00
Graham Christensen
1b833015b7
nixos tests: waitForText: output the detected screen content prior to the last attempt
machine: Last chance to match /(?^:BALICE)/ on the screen, which currently contains:
    machine: performing optical character recognition
    machine: sending monitor command: screendump /tmp/nix-build-vm-test-run-sddm.drv-0/ocrin.ppm
    machine: Session Layout

    O O

    0 1 : 0 9

    Wednesday, June 21, 2017

    |_ I

    Select your user and enter password
2017-06-20 21:10:34 -04:00
Graham Christensen
348785eec0
nixos tests: waitUntilTTYMatches: Log TTY contents on last try
If the test has not passed yet, on the last attempt it now outputs:

    machine: Last chance to match /logine: / on TTY2, which currently contains:
    machine: running command: fold -w$(stty -F /dev/tty2 size | awk '{print $2}') /dev/vcs2
    machine: exit status 0
    machine:

    <<< Welcome to NixOS 17.09.git.a804ef4 (x86_64) - tty2 >>>

    machine login:

to help debug the problem. Notice the "logine" typo in my check.
2017-06-20 20:57:39 -04:00
Graham Christensen
56435c1404
nixos tests: retry: Count down to 0, and pass remaining attempts to the sub
Allows test functions to output diagnostic information on failure.
2017-06-20 20:54:33 -04:00
Eelco Dolstra
0996ea8a68
NixOS VM tests: Don't create a setgid group in vde_switch
Nix no longer allows this for security reasons.

http://hydra.nixos.org/build/53993125
2017-06-07 11:57:36 +02:00
Volth
e5fbeb2f11 test-driver: sendKeys() support for capital letters 2017-05-19 13:56:24 +00:00
Dan Peebles
f1708a9d7d make-disk-image: change to be less VM-centric
This changes much of the make-disk-image.nix logic (and thus most NixOS
image building) to use LKL to set up the target directory structure rather
than a Linux VM. The only work we still do in a VM is less IO-heavy stuff
that while still time-consuming, is less of the overall load. The goal is
to kill more of that stuff, but that will require deeper changes to NixOS
activation scripts and switch-to-configuration.pl, and I don't want to
bite off too much at once.
2017-04-24 02:30:00 +00:00
aszlig
a443bdc0a6
nixos/testing: Improve quality of OCR
First of all, we're now using ImageMagick to improve the screenshot so
that Tesseract has an esier time to recognize the text. The resulting
image of this post-processing is a scaled up black-and-white version
with the backgrounds almost entirely removed and the text edges a bit
blurred, so the screen shots now more or less resemble an image from a
scanner rather. This is what Tesseract is trained for by default.

As mentioned in the previous commit we now also use Tesseract 4, which
further improves the quality of text recognition.

I've spent countless hours just to test different postprocessing
variants and testing what works best for our tests and this is the one
that worked best so far. It's certainly not perfect and I'd like to
avoid the scaling step but we're way better off than before.

In addition to this, the OCR process is now done without an intermediate
file, solely using pipes.

I've tested this using the following VM tests which have OCR enabled:

 * nixos/tests/chromium.nix -A stable
 * nixos/tests/emacs-daemon.nix
 * nixos/tests/installer.nix -A luksroot
 * nixos/tests/lightdm.nix
 * nixos/tests/plasma5.nix
 * nixos/tests/sddm.nix

All of the tests still succeed and comparing some of the recognition
results to the earlier results it now also detects a lot more text than
before this commit.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2017-04-11 03:21:53 +02:00
aszlig
288a79187c
tesseract: Reintroduce enableLanguages
I've removed that attribute in 68bc260ca2,
because the language files no longer were distributed as seperate files,
but if we for example only want to use the English training data, the
closure size of Tesseract gets quite large (around 1.2 GB), which is a
bit much just to be able to run NixOS VM tests.

For this reason I've also switched the VM tests back to using only the
English language.

Tested using the following VM tests (the ones that have OCR enabled) on
x86_64-linux:

 * nixos/tests/chromium.nix -A stable
 * nixos/tests/emacs-daemon.nix
 * nixos/tests/installer.nix -A luksroot
 * nixos/tests/lightdm.nix
 * nixos/tests/plasma5.nix
 * nixos/tests/sddm.nix

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2017-04-11 03:21:26 +02:00
Nikolay Amiantov
1f0ce0eaf8 makeTest: don't rely on makeWrapper not quoting newlines 2017-03-20 16:15:38 +03:00
Dan Peebles
e798f573f0 make-disk-image.nix: set last fsck time on ext4 images to enable resize-on-startup 2017-02-26 02:02:22 +00:00
Dan Peebles
49641e0de0 make-disk-image.nix: support additional filesystem contents
This makes make-disk-image.nix slightly more consistent with other image
builders we have. Unfortunately I duplicated some code in doing so, but
this is temporary duplication on the path to consolidating everything.
See https://github.com/NixOS/nixpkgs/issues/23052 for more details on that.

I'm also exposing the option in the amazon-image.nix maintainer module.
2017-02-22 23:49:49 +00:00
aszlig
a5e9668c5c
nixos/test: Fix escaping for copyFileFromHost
A long-time issue and one of the reasons I've never used that function
before. So let's remove that todo-comment and escape the contents
properly.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cc: @edolstra
2017-02-17 19:45:49 +01:00
Graham Christensen
59e77daf5b
nixos tests: make getTTYText smart about tty size 2017-02-12 18:27:06 -05:00
volth
50ec3fe1ac
test-driver: support punctuation in sendChars 2017-01-08 09:17:05 -05:00
Chris Martin
3956435d0a eval-config.nix: configuration->configuration.nix
`configuration` seems to be a reference to an argument that was
removed seven years ago in commit 2892aed7.

`configuration.nix` makes it a big more clear what we're referring to.
2017-01-04 19:15:47 +01:00
Joachim Fasting
48f270db52
make-system-tarball.sh: use numeric uids
Ownership should now always be 0/0.

Pointed out by @dezgeg at
https://github.com/NixOS/nixpkgs/pull/21305#discussion_r93258149
2017-01-01 06:03:05 +01:00
Eelco Dolstra
996e0ef777
test-driver: Show test duration 2016-12-29 16:23:27 +01:00