Commit Graph

833 Commits

Author SHA1 Message Date
Nick Bathum
519958c94b
nixos/lib/make-multi-disk-zfs-image: expose memSize parameter
closes #178095
2023-02-26 20:18:04 -05:00
Nick Bathum
59462da917
nixos/lib/make-disk-image: expose memSize parameter
Generating large disk images can fail from out-of-memory in vm-run.
2023-02-26 12:08:58 -05:00
pennae
417dd2ad16 nixos-render-docs: add options asciidoc converter
same reasoning as for the earlier commonmark converter.
2023-02-21 18:19:00 +01:00
pennae
4d3aef762f nixos-render-docs: add options commonmark converter
the old method of pasting parts of options.json into a markdown document
and hoping for the best no longer works now that options.json contains
more than just docbook. given the infrastructure we have now we can
actually render options.md properly, so we may as well do that.
2023-02-21 18:19:00 +01:00
pennae
81cf173256 nixos-render-docs: use multiprocessing for options
options processing is pretty slow right now, mostly because the
markdown-it-py parser is pure python (and with performance
pessimizations at that). options parsing *is* embarassingly parallel
though, so we can just fork out all the work to worker processes and
collect the results.

multiprocessing probably has a greater benefit on linux than on darwin
since the worker spawning method darwin uses is less efficient than
fork() on linux. this hasn't been tested on darwin, only on linux, but
if anything darwin will be faster with its preferred method.
2023-02-18 13:17:57 +01:00
K900
254426ec4b nixos/lib/testing: set default timeout for VM tests 2023-02-16 19:17:53 +03:00
pennae
8fe19590c3 nixos/make-options-doc: fix related packages link label
the pkgs path literal surely shouldn't include the title, right?
2023-02-08 15:23:34 +01:00
Jacek Galowicz
7f88d9c34c
Merge pull request #214910 from rnhmjoj/pr-gnupg-test
nixos/tests/gnupg: init
2023-02-07 09:17:06 +01:00
rnhmjoj
067d688b16
nixos/test-driver: handle decoding errors in Machine.execute
The output of a command is not guaranteed to be valid UTF-8, so the
decoding can fail raising UnicodeDecodeError. If this happens during a
`succeeds` the check will be erroneously marked failed.

This changes the error handling to the "replace" mode, where invalid
codepoints are replaced with � (REPLACEMENT CHARACTER U+FFFD) and the
decoding can go on.
2023-02-07 08:47:14 +01:00
rnhmjoj
f2929eb949
nixos/test-driver: drop logging from Machine.send_monitor_command
Several machine operations, like `send_chars` and `send_key`, are
implemented by calling `send_monitor_command`, possibly multiple times.
This generates a huge amount of unnecessary noise in the log, because
`send_monitor_command` is a low-level operation and an implementation
detail.

Here's an excerpt from a highlighted log before and afte the change.

Before:

    subtest: Can generate a PGP key
    machine: sending keys 'gpg --gen-key\n'
    machine: sending monitor command: sendkey g
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey g, in 0.00 seconds)
    machine: sending monitor command: sendkey p
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey p, in 0.00 seconds)
    machine: sending monitor command: sendkey g
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey g, in 0.00 seconds)
    machine: sending monitor command: sendkey spc
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey spc, in 0.00 seconds)
    machine: sending monitor command: sendkey 0x0C
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey 0x0C, in 0.00 seconds)
    machine: sending monitor command: sendkey 0x0C
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey 0x0C, in 0.00 seconds)
    machine: sending monitor command: sendkey g
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey g, in 0.00 seconds)
    machine: sending monitor command: sendkey e
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey e, in 0.00 seconds)
    machine: sending monitor command: sendkey n
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey n, in 0.00 seconds)
    machine: sending monitor command: sendkey 0x0C
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey 0x0C, in 0.00 seconds)
    machine: sending monitor command: sendkey k
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey k, in 0.00 seconds)
    machine: sending monitor command: sendkey e
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey e, in 0.00 seconds)
    machine: sending monitor command: sendkey y
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey y, in 0.00 seconds)
    machine: sending monitor command: sendkey ret
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey ret, in 0.00 seconds)
    (finished: sending keys 'gpg --gen-key\n', in 0.15 seconds)

After:

    subtest: Can generate a PGP key
    machine: sending keys 'gpg --gen-key\n'
    (finished: sending keys 'gpg --gen-key\n', in 0.15 seconds)
2023-02-07 08:40:54 +01:00
Alyssa Ross
32d066dce3
channel: add --show-trace to nix-env command
Otherwise, when this fails on Hydra, we have no way of seeing what
went wrong.
2023-02-03 23:02:47 +00:00
pennae
5b6dcece88
Merge pull request #212684 from pennae/nixos-render-docs
nixos-render-docs: init, use for some manual rendering to docbook
2023-01-30 19:26:07 +01:00
Vladimír Čunát
23ce77d76e Revert #178290: nixos/virtualisation: add option
...for explicitly named network interfaces

This reverts commit 6ae3e7695e.
(and evaluation fixups 08d26bbb72 7aed90a969)
Some of the tests fail or time out after the merge.
2023-01-30 07:55:50 -08:00
pennae
e8c5618b67 nixos-render-docs: add some better CLI infrastructure
using environment variables isn't great once multiple input or output
formats get involved (which will happen soon). now is a good time to set
a pattern for future converters.
2023-01-27 20:07:34 +01:00
pennae
be6a25368f nixos-render-docs: init from optionsToDocbook.py
this new package shall eventually contain the rendering code necessary
to produce the entirety of the nixos (not nixpkgs) manual, in all of its
various output formats.
2023-01-27 20:07:33 +01:00
Jörg Thalheim
e80e3878b3
Merge pull request #178533 from Mic92/nixos-tests-shell
nixos/tests: extend shell_interact to accept alternative socat addresses
2023-01-27 12:24:08 +00:00
Jörg Thalheim
29db54c373 nixos/tests: extend shell_interact to accept alternative socat addresses
`shell_interact()` is currently not nice to use.  If you try to cancel
the socat process, it will also break the nixos test. Furthermore
ptpython creates it's own terminal that subprocesses are running in,
which breaks some of the terminal features of socat.
Hence this commit extends `shell_interact` to allow also to connect to
arbitrary servers i.e. tcp servers started by socat.
2023-01-27 13:09:25 +01:00
pennae
9ff987764c nixos/make-options-doc: enable smartquotes and replacements
the rest of the nixos manual has them enabled, so we should enable them
here too for consistency.

this changes rendered output pervasively. changes also include quotes in
types (eg in `strings concatenated with "\n"`), but since those are not
code this is probably fine. if not we can probably add a myst role to
inhibit replacements.
2023-01-26 00:38:06 +01:00
pennae
de22a26b4c nixos/make-options-doc: render option types through md
no changes to rendered output, but options.xml loses a few spaces.
2023-01-26 00:38:06 +01:00
pennae
6b677d9148 nixos/make-options-doc: disable inline html
we target not only html, but also manpages. inline html is not helpful
for the latter.
2023-01-26 00:38:06 +01:00
pennae
ba1f533134 nixos/make-option-docs: wrap md parser+renderer into Converter
the new rendering tool will consist of a number of different Converters,
each with a possibly specialized Renderer.
2023-01-26 00:38:06 +01:00
pennae
3e45994a3b nixos/make-options-doc: don't use inspect for optionsToDocbook
the rules are fixed, and we want to support all of them (or throw a
useful error message). this will also become the base for a generic
renderer system, so let's just list all the rules statically.
2023-01-26 00:38:06 +01:00
Robert Hensing
d255493e8f nixos/optionsToDocbook.py: Add docbook ns for when the appendix tag is omitted 2023-01-26 00:32:56 +01:00
pennae
0175a91aa3 nixos/make-options-doc: split docbook conversion from mergeJSON
this restores mergeJSON to its former glory if…merging json, and
extracts the MD rendering into a new script that will run instead of the
py+nix+xslt pipeline we previously ran to convert options.json to docbook.

this change alone gives a noticable performance boost when building
docs (18s instead of 27s to build optionsDocBook).

no changes to rendered output, except for a single example in the
rsnapshot module that uses hard tabs for indentation instead of spaces.
this probably isn't important.

docbook warnings remain with mergeJSON since the other processing steps
output single files instead of directories. since we'll only keep the
check until 23.11 this is probably also not important to fix.

also contains a few improvements to error reporting in the MD renderers.
2023-01-26 00:32:56 +01:00
pennae
381dcd7f9d nixos/make-options-doc: remove program argument to xslt
this doesn't seem to be set anywhere, not from make-options-doc nor
anywhere else.
2023-01-26 00:32:56 +01:00
Andrew Hoff
6ae3e7695e nixos/virtualisation: add option for explicitly named network interfaces
Adds a new option to the virtualisation modules that enables specifying
explicitly named network interfaces in QEMU VMs. The existing
`virtualisation.vlans` is still supported for cases where the name of
the network interface is irrelevant.
2023-01-22 12:20:40 -05:00
pennae
e2019c49d4 nixos/make-options-doc: use markdown-it-py for rendering
only whitespace changes (mostly empty descriptions rendered as literal
line breaks and trailing space toPretty generates, but that were dropped
by mistune).
2023-01-22 17:50:09 +01:00
pennae
248e1b957d nixos/make-options-doc: render related packages with md
don't generate docbook for related packages, generate markdown instead.
this could be extended further to not even generate markdown but have
mergeJSON handle all of the rendering. markdown will work fine for now
though.
2023-01-22 17:50:09 +01:00
pennae
fa8a594c56 nixos/make-options-doc: deprecate \n\n parbreak
only whitespace changes to rendered outputs, all in the vicinity or body
of admonitions. previously admonitions would not receive paragraph
breaks even when they should have because the description postprocessing
did not match on their contents.
2023-01-22 17:50:09 +01:00
pennae
c93e5dde67 nixos/make-options-doc: rearrange paras in related packages/defaults/examples
this mirrors what we will be able to create with markdown. no change to
rendered outputs.
2023-01-22 17:50:09 +01:00
pennae
3a5f1ae029 nixos/make-options-doc: render default/example contents through MD
removes some trailing whitespaces from the html output, no other changes.
2023-01-22 17:50:09 +01:00
pennae
2bd8129a47 nixos/make-options-doc: make whitespace more md-compatible
markdown-it-py creates different whitespace leaders/trailers than are
currently emitted, and when we convert examples and defaults to render
via markdown the spacing will change too. this has no effect on rendered
output.
2023-01-22 17:50:09 +01:00
pennae
d1aa187c0e nixos/make-options-doc: don't escape link urls twice
mistune already does escaping. it does escaping for html, but the
difference is small enough that can just ignore that we're actually
targeting docbook here.
2023-01-22 17:50:09 +01:00
pennae
2cb43da587 nixos/make-options-doc: remove trailing whitespace from strings
this was done only to make the conversion to MD easier to verify. we no
longer need it, and not keeping whitespace does not affect rendered outputs.

stripping will have to stay for now because description postprocessing
would add empty paragraphs otherwise.
2023-01-22 17:50:09 +01:00
pennae
df09c21fb2 nixos/documentation: deprecate docbook option docs
following the plan in https://github.com/NixOS/nixpkgs/pull/189318#discussion_r961764451

also adds an activation script to print the warning during activation
instead of during build, otherwise folks using the new CLI that hides
build logs by default might never see the warning.
2023-01-22 17:08:40 +01:00
Guillaume Girol
33afbf39f6 treewide: switch to nativeCheckInputs
checkInputs used to be added to nativeBuildInputs. Now we have
nativeCheckInputs to do that instead. Doing this treewide change allows
to keep hashes identical to before the introduction of
nativeCheckInputs.
2023-01-21 12:00:00 +00:00
Naïm Favier
bff4bb5540
Merge pull request #208762 from ncfavier/link-manpages-options-doc 2023-01-09 11:57:01 +01:00
Naïm Favier
fb1bc8de93
lib/modules: make mkAliasOptionModule emit DocBook
Follow-up to https://github.com/NixOS/nixpkgs/pull/208407

Removing `mdDoc` isn't enough, we need to emit actual DocBook.
2023-01-08 22:44:18 +01:00
Domen Kožar
dce3b02d44
Merge pull request #209501 from sandydoo/improve/fix-commonmark-doc-isues
nixos/make-options-doc: skip re-escaping literal expressions
2023-01-08 11:14:57 +00:00
sandydoo
18df33f5bc
nixos/make-options-doc: skip re-escaping strings 2023-01-08 08:52:17 +00:00
pennae
9da5f12ecf modules: add mkPackageOptionMD
another transitional option factory, like mkAliasOptionModuleMD.
2023-01-05 02:33:13 +01:00
pennae
4c1cfbdb84 modules: add mkAliasOptionModuleMD
mkAliasOptionModule should not default to mdDoc descriptions because
that can break out-of-tree users of documentation infrastructure. add an
explicitly-MD variant for now, to be removed some time after the MD
transition is complete.
2023-01-05 02:33:13 +01:00
Naïm Favier
4fb500d629
nixos/doc: fix some manpage references 2023-01-03 14:03:35 +01:00
Naïm Favier
f61e538c43
nixos/make-options-doc: link manpages
Add links to manpages without a link using the mapping defined in
`doc/manpage-urls.json`, as is already done for the nixpkgs and NixOS
manuals.
2023-01-03 14:03:35 +01:00
Jacek Galowicz
869545857f
Merge pull request #208354 from bjornfor/nixos-test-driver-quote-shell-args
nixos/test-driver: quote some shell command lines
2022-12-30 22:41:04 +01:00
Bjørn Forsman
c876f0c146 nixos/test-driver: quote some shell command lines
This makes them robust against paths with whitespace.
2022-12-30 13:19:59 +01:00
Bjørn Forsman
4c45c3f8f2 nixos/test-driver: use ASCII single quotes everywhere
A few places used Unicode U+2018/U+2019 left/right single quotes (but
not always correctly balanced). Let's just use plain ASCII single quotes
everywhere.
2022-12-30 13:00:19 +01:00
Bjørn Forsman
ce5644f658 nixos/test-driver: use f-strings instead of .format()
For readability.

Suggested-by: @tfc
2022-12-29 23:51:35 +01:00
Bjørn Forsman
dba49a43a0 nixos/test-driver: add optional address arg to wait_for_{open,closed}_port
This is useful for testing servers configured to listen on specific
addresses.
2022-12-29 13:50:17 +01:00
Naïm Favier
3fc528ff7f
Merge pull request #207095 from ncfavier/linux-custom-kernel 2022-12-27 17:10:43 +01:00