git.iko.soy/typeable/nixpkgs
on master help sign in
Merge pull request #146905 from Synthetica9/failure_mode

nixos/test-driver: add polling_condition

commit048fd95f106ef4dca58ff01159207a027ab612d9 authorJacek Galowicz <jacek@galowicz.de> dateSat 8 Jan 2022, 18:28 (5 years ago) parent41216b0793a2f5
8 files changed+667−413
nixos/doc/manual/development/writing-nixos-tests.section.md+51 nixos/doc/manual/from_md/development/writing-nixos-tests.section.xml+457−389 nixos/lib/test-driver/default.nix+2−2 nixos/lib/test-driver/setup.py+1−1 nixos/lib/test-driver/test_driver/driver.py+40−1 nixos/lib/test-driver/test_driver/machine.py+9 nixos/lib/test-driver/test_driver/polling_condition.py+77 nixos/tests/vscodium.nix+30−20
nixos/doc/manual/development/writing-nixos-tests.section.md+51
@@ -88,6 +88,8 @@ starting them in parallel:
8888start_all()
8989```
9090
91+## Machine objects {#ssec-machine-objects}
92+
9193The following methods are available on machine objects:
9294
9395`start`
@@ -313,3 +315,52 @@ repository):
313315 # fmt: on
314316 '';
315317```
318+
319+## Failing tests early {#ssec-failing-tests-early}
320+
321+To fail tests early when certain invariables are no longer met (instead of waiting for the build to time out), the decorator `polling_condition` is provided. For example, if we are testing a program `foo` that should not quit after being started, we might write the following:
322+
323+```py
324+@polling_condition
325+def foo_running():
326+ machine.succeed("pgrep -x foo")
327+
328+
329+machine.succeed("foo --start")
330+machine.wait_until_succeeds("pgrep -x foo")
331+
332+with foo_running:
333+ ... # Put `foo` through its paces
334+```
335+
336+
337+`polling_condition` takes the following (optional) arguments:
338+
339+`seconds_interval`
340+
341+:
342+ specifies how often the condition should be polled:
343+
344+ ```py
345+ @polling_condition(seconds_interval=10)
346+ def foo_running():
347+ machine.succeed("pgrep -x foo")
348+ ```
349+
350+`description`
351+
352+:
353+ is used in the log when the condition is checked. If this is not provided, the description is pulled from the docstring of the function. These two are therefore equivalent:
354+
355+ ```py
356+ @polling_condition
357+ def foo_running():
358+ "check that foo is running"
359+ machine.succeed("pgrep -x foo")
360+ ```
361+
362+ ```py
363+ @polling_condition(description="check that foo is running")
364+ def foo_running():
365+ machine.succeed("pgrep -x foo")
366+ ```
nixos/doc/manual/from_md/development/writing-nixos-tests.section.xml+457−389
@@ -117,407 +117,413 @@ if not &quot;Linux&quot; in machine.succeed(&quot;uname&quot;):
117117 <programlisting language="python">
118118start_all()
119119</programlisting>
120− <para>
121− The following methods are available on machine objects:
122− </para>
123− <variablelist>
124− <varlistentry>
125− <term>
126− <literal>start</literal>
127− </term>
128− <listitem>
129− <para>
130− Start the virtual machine. This method is asynchronous — it
131− does not wait for the machine to finish booting.
132− </para>
133− </listitem>
134− </varlistentry>
135− <varlistentry>
136− <term>
137− <literal>shutdown</literal>
138− </term>
139− <listitem>
140− <para>
141− Shut down the machine, waiting for the VM to exit.
142− </para>
143− </listitem>
144− </varlistentry>
145− <varlistentry>
146− <term>
147− <literal>crash</literal>
148− </term>
149− <listitem>
150− <para>
151− Simulate a sudden power failure, by telling the VM to exit
152− immediately.
153− </para>
154− </listitem>
155− </varlistentry>
156− <varlistentry>
157− <term>
158− <literal>block</literal>
159− </term>
160− <listitem>
161− <para>
162− Simulate unplugging the Ethernet cable that connects the
163− machine to the other machines.
164− </para>
165− </listitem>
166− </varlistentry>
167− <varlistentry>
168− <term>
169− <literal>unblock</literal>
170− </term>
171− <listitem>
172− <para>
173− Undo the effect of <literal>block</literal>.
174− </para>
175− </listitem>
176− </varlistentry>
177− <varlistentry>
178− <term>
179− <literal>screenshot</literal>
180− </term>
181− <listitem>
182− <para>
183− Take a picture of the display of the virtual machine, in PNG
184− format. The screenshot is linked from the HTML log.
185− </para>
186− </listitem>
187− </varlistentry>
188− <varlistentry>
189− <term>
190− <literal>get_screen_text_variants</literal>
191− </term>
192− <listitem>
193− <para>
194− Return a list of different interpretations of what is
195− currently visible on the machine's screen using optical
196− character recognition. The number and order of the
197− interpretations is not specified and is subject to change, but
198− if no exception is raised at least one will be returned.
199− </para>
200− <note>
120+ <section xml:id="ssec-machine-objects">
121+ <title>Machine objects</title>
122+ <para>
123+ The following methods are available on machine objects:
124+ </para>
125+ <variablelist>
126+ <varlistentry>
127+ <term>
128+ <literal>start</literal>
129+ </term>
130+ <listitem>
201131 <para>
202− This requires passing <literal>enableOCR</literal> to the
203− test attribute set.
132+ Start the virtual machine. This method is asynchronous — it
133+ does not wait for the machine to finish booting.
204134 </para>
205− </note>
206− </listitem>
207− </varlistentry>
208− <varlistentry>
209− <term>
210− <literal>get_screen_text</literal>
211− </term>
212− <listitem>
213− <para>
214− Return a textual representation of what is currently visible
215− on the machine's screen using optical character recognition.
216− </para>
217− <note>
135+ </listitem>
136+ </varlistentry>
137+ <varlistentry>
138+ <term>
139+ <literal>shutdown</literal>
140+ </term>
141+ <listitem>
218142 <para>
219− This requires passing <literal>enableOCR</literal> to the
220− test attribute set.
143+ Shut down the machine, waiting for the VM to exit.
221144 </para>
222− </note>
223− </listitem>
224− </varlistentry>
225− <varlistentry>
226− <term>
227− <literal>send_monitor_command</literal>
228− </term>
229− <listitem>
230− <para>
231− Send a command to the QEMU monitor. This is rarely used, but
232− allows doing stuff such as attaching virtual USB disks to a
233− running machine.
234− </para>
235− </listitem>
236− </varlistentry>
237− <varlistentry>
238− <term>
239− <literal>send_key</literal>
240− </term>
241− <listitem>
242− <para>
243− Simulate pressing keys on the virtual keyboard, e.g.,
244− <literal>send_key(&quot;ctrl-alt-delete&quot;)</literal>.
245− </para>
246− </listitem>
247− </varlistentry>
248− <varlistentry>
249− <term>
250− <literal>send_chars</literal>
251− </term>
252− <listitem>
253− <para>
254− Simulate typing a sequence of characters on the virtual
255− keyboard, e.g.,
256− <literal>send_chars(&quot;foobar\n&quot;)</literal> will type
257− the string <literal>foobar</literal> followed by the Enter
258− key.
259− </para>
260− </listitem>
261− </varlistentry>
262− <varlistentry>
263− <term>
264− <literal>execute</literal>
265− </term>
266− <listitem>
267− <para>
268− Execute a shell command, returning a list
269− <literal>(status, stdout)</literal>. If the command detaches,
270− it must close stdout, as <literal>execute</literal> will wait
271− for this to consume all output reliably. This can be achieved
272− by redirecting stdout to stderr <literal>&gt;&amp;2</literal>,
273− to <literal>/dev/console</literal>,
274− <literal>/dev/null</literal> or a file. Examples of detaching
275− commands are <literal>sleep 365d &amp;</literal>, where the
276− shell forks a new process that can write to stdout and
277− <literal>xclip -i</literal>, where the
278− <literal>xclip</literal> command itself forks without closing
279− stdout. Takes an optional parameter
280− <literal>check_return</literal> that defaults to
281− <literal>True</literal>. Setting this parameter to
282− <literal>False</literal> will not check for the return code
283− and return -1 instead. This can be used for commands that shut
284− down the VM and would therefore break the pipe that would be
285− used for retrieving the return code.
286− </para>
287− </listitem>
288− </varlistentry>
289− <varlistentry>
290− <term>
291− <literal>succeed</literal>
292− </term>
293− <listitem>
294− <para>
295− Execute a shell command, raising an exception if the exit
296− status is not zero, otherwise returning the standard output.
297− Commands are run with <literal>set -euo pipefail</literal>
298− set:
299− </para>
300− <itemizedlist>
301− <listitem>
302− <para>
303− If several commands are separated by <literal>;</literal>
304− and one fails, the command as a whole will fail.
305− </para>
306− </listitem>
307− <listitem>
145+ </listitem>
146+ </varlistentry>
147+ <varlistentry>
148+ <term>
149+ <literal>crash</literal>
150+ </term>
151+ <listitem>
152+ <para>
153+ Simulate a sudden power failure, by telling the VM to exit
154+ immediately.
155+ </para>
156+ </listitem>
157+ </varlistentry>
158+ <varlistentry>
159+ <term>
160+ <literal>block</literal>
161+ </term>
162+ <listitem>
163+ <para>
164+ Simulate unplugging the Ethernet cable that connects the
165+ machine to the other machines.
166+ </para>
167+ </listitem>
168+ </varlistentry>
169+ <varlistentry>
170+ <term>
171+ <literal>unblock</literal>
172+ </term>
173+ <listitem>
174+ <para>
175+ Undo the effect of <literal>block</literal>.
176+ </para>
177+ </listitem>
178+ </varlistentry>
179+ <varlistentry>
180+ <term>
181+ <literal>screenshot</literal>
182+ </term>
183+ <listitem>
184+ <para>
185+ Take a picture of the display of the virtual machine, in PNG
186+ format. The screenshot is linked from the HTML log.
187+ </para>
188+ </listitem>
189+ </varlistentry>
190+ <varlistentry>
191+ <term>
192+ <literal>get_screen_text_variants</literal>
193+ </term>
194+ <listitem>
195+ <para>
196+ Return a list of different interpretations of what is
197+ currently visible on the machine's screen using optical
198+ character recognition. The number and order of the
199+ interpretations is not specified and is subject to change,
200+ but if no exception is raised at least one will be returned.
201+ </para>
202+ <note>
308203 <para>
309− For pipelines, the last non-zero exit status will be
310− returned (if there is one, zero will be returned
311− otherwise).
204+ This requires passing <literal>enableOCR</literal> to the
205+ test attribute set.
312206 </para>
313− </listitem>
314− <listitem>
207+ </note>
208+ </listitem>
209+ </varlistentry>
210+ <varlistentry>
211+ <term>
212+ <literal>get_screen_text</literal>
213+ </term>
214+ <listitem>
215+ <para>
216+ Return a textual representation of what is currently visible
217+ on the machine's screen using optical character recognition.
218+ </para>
219+ <note>
315220 <para>
316− Dereferencing unset variables fail the command.
221+ This requires passing <literal>enableOCR</literal> to the
222+ test attribute set.
317223 </para>
318− </listitem>
319− <listitem>
224+ </note>
225+ </listitem>
226+ </varlistentry>
227+ <varlistentry>
228+ <term>
229+ <literal>send_monitor_command</literal>
230+ </term>
231+ <listitem>
232+ <para>
233+ Send a command to the QEMU monitor. This is rarely used, but
234+ allows doing stuff such as attaching virtual USB disks to a
235+ running machine.
236+ </para>
237+ </listitem>
238+ </varlistentry>
239+ <varlistentry>
240+ <term>
241+ <literal>send_key</literal>
242+ </term>
243+ <listitem>
244+ <para>
245+ Simulate pressing keys on the virtual keyboard, e.g.,
246+ <literal>send_key(&quot;ctrl-alt-delete&quot;)</literal>.
247+ </para>
248+ </listitem>
249+ </varlistentry>
250+ <varlistentry>
251+ <term>
252+ <literal>send_chars</literal>
253+ </term>
254+ <listitem>
255+ <para>
256+ Simulate typing a sequence of characters on the virtual
257+ keyboard, e.g.,
258+ <literal>send_chars(&quot;foobar\n&quot;)</literal> will
259+ type the string <literal>foobar</literal> followed by the
260+ Enter key.
261+ </para>
262+ </listitem>
263+ </varlistentry>
264+ <varlistentry>
265+ <term>
266+ <literal>execute</literal>
267+ </term>
268+ <listitem>
269+ <para>
270+ Execute a shell command, returning a list
271+ <literal>(status, stdout)</literal>. If the command
272+ detaches, it must close stdout, as
273+ <literal>execute</literal> will wait for this to consume all
274+ output reliably. This can be achieved by redirecting stdout
275+ to stderr <literal>&gt;&amp;2</literal>, to
276+ <literal>/dev/console</literal>,
277+ <literal>/dev/null</literal> or a file. Examples of
278+ detaching commands are <literal>sleep 365d &amp;</literal>,
279+ where the shell forks a new process that can write to stdout
280+ and <literal>xclip -i</literal>, where the
281+ <literal>xclip</literal> command itself forks without
282+ closing stdout. Takes an optional parameter
283+ <literal>check_return</literal> that defaults to
284+ <literal>True</literal>. Setting this parameter to
285+ <literal>False</literal> will not check for the return code
286+ and return -1 instead. This can be used for commands that
287+ shut down the VM and would therefore break the pipe that
288+ would be used for retrieving the return code.
289+ </para>
290+ </listitem>
291+ </varlistentry>
292+ <varlistentry>
293+ <term>
294+ <literal>succeed</literal>
295+ </term>
296+ <listitem>
297+ <para>
298+ Execute a shell command, raising an exception if the exit
299+ status is not zero, otherwise returning the standard output.
300+ Commands are run with <literal>set -euo pipefail</literal>
301+ set:
302+ </para>
303+ <itemizedlist>
304+ <listitem>
305+ <para>
306+ If several commands are separated by
307+ <literal>;</literal> and one fails, the command as a
308+ whole will fail.
309+ </para>
310+ </listitem>
311+ <listitem>
312+ <para>
313+ For pipelines, the last non-zero exit status will be
314+ returned (if there is one, zero will be returned
315+ otherwise).
316+ </para>
317+ </listitem>
318+ <listitem>
319+ <para>
320+ Dereferencing unset variables fail the command.
321+ </para>
322+ </listitem>
323+ <listitem>
324+ <para>
325+ It will wait for stdout to be closed. See
326+ <literal>execute</literal> for the implications.
327+ </para>
328+ </listitem>
329+ </itemizedlist>
330+ </listitem>
331+ </varlistentry>
332+ <varlistentry>
333+ <term>
334+ <literal>fail</literal>
335+ </term>
336+ <listitem>
337+ <para>
338+ Like <literal>succeed</literal>, but raising an exception if
339+ the command returns a zero status.
340+ </para>
341+ </listitem>
342+ </varlistentry>
343+ <varlistentry>
344+ <term>
345+ <literal>wait_until_succeeds</literal>
346+ </term>
347+ <listitem>
348+ <para>
349+ Repeat a shell command with 1-second intervals until it
350+ succeeds.
351+ </para>
352+ </listitem>
353+ </varlistentry>
354+ <varlistentry>
355+ <term>
356+ <literal>wait_until_fails</literal>
357+ </term>
358+ <listitem>
359+ <para>
360+ Repeat a shell command with 1-second intervals until it
361+ fails.
362+ </para>
363+ </listitem>
364+ </varlistentry>
365+ <varlistentry>
366+ <term>
367+ <literal>wait_for_unit</literal>
368+ </term>
369+ <listitem>
370+ <para>
371+ Wait until the specified systemd unit has reached the
372+ <quote>active</quote> state.
373+ </para>
374+ </listitem>
375+ </varlistentry>
376+ <varlistentry>
377+ <term>
378+ <literal>wait_for_file</literal>
379+ </term>
380+ <listitem>
381+ <para>
382+ Wait until the specified file exists.
383+ </para>
384+ </listitem>
385+ </varlistentry>
386+ <varlistentry>
387+ <term>
388+ <literal>wait_for_open_port</literal>
389+ </term>
390+ <listitem>
391+ <para>
392+ Wait until a process is listening on the given TCP port (on
393+ <literal>localhost</literal>, at least).
394+ </para>
395+ </listitem>
396+ </varlistentry>
397+ <varlistentry>
398+ <term>
399+ <literal>wait_for_closed_port</literal>
400+ </term>
401+ <listitem>
402+ <para>
403+ Wait until nobody is listening on the given TCP port.
404+ </para>
405+ </listitem>
406+ </varlistentry>
407+ <varlistentry>
408+ <term>
409+ <literal>wait_for_x</literal>
410+ </term>
411+ <listitem>
412+ <para>
413+ Wait until the X11 server is accepting connections.
414+ </para>
415+ </listitem>
416+ </varlistentry>
417+ <varlistentry>
418+ <term>
419+ <literal>wait_for_text</literal>
420+ </term>
421+ <listitem>
422+ <para>
423+ Wait until the supplied regular expressions matches the
424+ textual contents of the screen by using optical character
425+ recognition (see <literal>get_screen_text</literal> and
426+ <literal>get_screen_text_variants</literal>).
427+ </para>
428+ <note>
320429 <para>
321− It will wait for stdout to be closed. See
322− <literal>execute</literal> for the implications.
430+ This requires passing <literal>enableOCR</literal> to the
431+ test attribute set.
323432 </para>
324− </listitem>
325− </itemizedlist>
326− </listitem>
327− </varlistentry>
328− <varlistentry>
329− <term>
330− <literal>fail</literal>
331− </term>
332− <listitem>
333− <para>
334− Like <literal>succeed</literal>, but raising an exception if
335− the command returns a zero status.
336− </para>
337− </listitem>
338− </varlistentry>
339− <varlistentry>
340− <term>
341− <literal>wait_until_succeeds</literal>
342− </term>
343− <listitem>
344− <para>
345− Repeat a shell command with 1-second intervals until it
346− succeeds.
347− </para>
348− </listitem>
349− </varlistentry>
350− <varlistentry>
351− <term>
352− <literal>wait_until_fails</literal>
353− </term>
354− <listitem>
355− <para>
356− Repeat a shell command with 1-second intervals until it fails.
357− </para>
358− </listitem>
359− </varlistentry>
360− <varlistentry>
361− <term>
362− <literal>wait_for_unit</literal>
363− </term>
364− <listitem>
365− <para>
366− Wait until the specified systemd unit has reached the
367− <quote>active</quote> state.
368− </para>
369− </listitem>
370− </varlistentry>
371− <varlistentry>
372− <term>
373− <literal>wait_for_file</literal>
374− </term>
375− <listitem>
376− <para>
377− Wait until the specified file exists.
378− </para>
379− </listitem>
380− </varlistentry>
381− <varlistentry>
382− <term>
383− <literal>wait_for_open_port</literal>
384− </term>
385− <listitem>
386− <para>
387− Wait until a process is listening on the given TCP port (on
388− <literal>localhost</literal>, at least).
389− </para>
390− </listitem>
391− </varlistentry>
392− <varlistentry>
393− <term>
394− <literal>wait_for_closed_port</literal>
395− </term>
396− <listitem>
397− <para>
398− Wait until nobody is listening on the given TCP port.
399− </para>
400− </listitem>
401− </varlistentry>
402− <varlistentry>
403− <term>
404− <literal>wait_for_x</literal>
405− </term>
406− <listitem>
407− <para>
408− Wait until the X11 server is accepting connections.
409− </para>
410− </listitem>
411− </varlistentry>
412− <varlistentry>
413− <term>
414− <literal>wait_for_text</literal>
415− </term>
416− <listitem>
417− <para>
418− Wait until the supplied regular expressions matches the
419− textual contents of the screen by using optical character
420− recognition (see <literal>get_screen_text</literal> and
421− <literal>get_screen_text_variants</literal>).
422− </para>
423− <note>
433+ </note>
434+ </listitem>
435+ </varlistentry>
436+ <varlistentry>
437+ <term>
438+ <literal>wait_for_console_text</literal>
439+ </term>
440+ <listitem>
424441 <para>
425− This requires passing <literal>enableOCR</literal> to the
426− test attribute set.
442+ Wait until the supplied regular expressions match a line of
443+ the serial console output. This method is useful when OCR is
444+ not possibile or accurate enough.
427445 </para>
428− </note>
429− </listitem>
430− </varlistentry>
431− <varlistentry>
432− <term>
433− <literal>wait_for_console_text</literal>
434− </term>
435− <listitem>
436− <para>
437− Wait until the supplied regular expressions match a line of
438− the serial console output. This method is useful when OCR is
439− not possibile or accurate enough.
440− </para>
441− </listitem>
442− </varlistentry>
443− <varlistentry>
444− <term>
445− <literal>wait_for_window</literal>
446− </term>
447− <listitem>
448− <para>
449− Wait until an X11 window has appeared whose name matches the
450− given regular expression, e.g.,
451− <literal>wait_for_window(&quot;Terminal&quot;)</literal>.
452− </para>
453− </listitem>
454− </varlistentry>
455− <varlistentry>
456− <term>
457− <literal>copy_from_host</literal>
458− </term>
459− <listitem>
460− <para>
461− Copies a file from host to machine, e.g.,
462− <literal>copy_from_host(&quot;myfile&quot;, &quot;/etc/my/important/file&quot;)</literal>.
463− </para>
464− <para>
465− The first argument is the file on the host. The file needs to
466− be accessible while building the nix derivation. The second
467− argument is the location of the file on the machine.
468− </para>
469− </listitem>
470− </varlistentry>
471− <varlistentry>
472− <term>
473− <literal>systemctl</literal>
474− </term>
475− <listitem>
476− <para>
477− Runs <literal>systemctl</literal> commands with optional
478− support for <literal>systemctl --user</literal>
479− </para>
480− <programlisting language="python">
446+ </listitem>
447+ </varlistentry>
448+ <varlistentry>
449+ <term>
450+ <literal>wait_for_window</literal>
451+ </term>
452+ <listitem>
453+ <para>
454+ Wait until an X11 window has appeared whose name matches the
455+ given regular expression, e.g.,
456+ <literal>wait_for_window(&quot;Terminal&quot;)</literal>.
457+ </para>
458+ </listitem>
459+ </varlistentry>
460+ <varlistentry>
461+ <term>
462+ <literal>copy_from_host</literal>
463+ </term>
464+ <listitem>
465+ <para>
466+ Copies a file from host to machine, e.g.,
467+ <literal>copy_from_host(&quot;myfile&quot;, &quot;/etc/my/important/file&quot;)</literal>.
468+ </para>
469+ <para>
470+ The first argument is the file on the host. The file needs
471+ to be accessible while building the nix derivation. The
472+ second argument is the location of the file on the machine.
473+ </para>
474+ </listitem>
475+ </varlistentry>
476+ <varlistentry>
477+ <term>
478+ <literal>systemctl</literal>
479+ </term>
480+ <listitem>
481+ <para>
482+ Runs <literal>systemctl</literal> commands with optional
483+ support for <literal>systemctl --user</literal>
484+ </para>
485+ <programlisting language="python">
481486machine.systemctl(&quot;list-jobs --no-pager&quot;) # runs `systemctl list-jobs --no-pager`
482487machine.systemctl(&quot;list-jobs --no-pager&quot;, &quot;any-user&quot;) # spawns a shell for `any-user` and runs `systemctl --user list-jobs --no-pager`
483488</programlisting>
484− </listitem>
485− </varlistentry>
486− <varlistentry>
487− <term>
488− <literal>shell_interact</literal>
489− </term>
490− <listitem>
491− <para>
492− Allows you to directly interact with the guest shell. This
493− should only be used during test development, not in production
494− tests. Killing the interactive session with
495− <literal>Ctrl-d</literal> or <literal>Ctrl-c</literal> also
496− ends the guest session.
497− </para>
498− </listitem>
499− </varlistentry>
500− </variablelist>
501− <para>
502− To test user units declared by
503− <literal>systemd.user.services</literal> the optional
504− <literal>user</literal> argument can be used:
505− </para>
506− <programlisting language="python">
489+ </listitem>
490+ </varlistentry>
491+ <varlistentry>
492+ <term>
493+ <literal>shell_interact</literal>
494+ </term>
495+ <listitem>
496+ <para>
497+ Allows you to directly interact with the guest shell. This
498+ should only be used during test development, not in
499+ production tests. Killing the interactive session with
500+ <literal>Ctrl-d</literal> or <literal>Ctrl-c</literal> also
501+ ends the guest session.
502+ </para>
503+ </listitem>
504+ </varlistentry>
505+ </variablelist>
506+ <para>
507+ To test user units declared by
508+ <literal>systemd.user.services</literal> the optional
509+ <literal>user</literal> argument can be used:
510+ </para>
511+ <programlisting language="python">
507512machine.start()
508513machine.wait_for_x()
509514machine.wait_for_unit(&quot;xautolock.service&quot;, &quot;x-session-user&quot;)
510515</programlisting>
511− <para>
512− This applies to <literal>systemctl</literal>,
513− <literal>get_unit_info</literal>, <literal>wait_for_unit</literal>,
514− <literal>start_job</literal> and <literal>stop_job</literal>.
515− </para>
516− <para>
517− For faster dev cycles it's also possible to disable the code-linters
518− (this shouldn't be commited though):
519− </para>
520− <programlisting language="bash">
516+ <para>
517+ This applies to <literal>systemctl</literal>,
518+ <literal>get_unit_info</literal>,
519+ <literal>wait_for_unit</literal>, <literal>start_job</literal> and
520+ <literal>stop_job</literal>.
521+ </para>
522+ <para>
523+ For faster dev cycles it's also possible to disable the
524+ code-linters (this shouldn't be commited though):
525+ </para>
526+ <programlisting language="bash">
521527import ./make-test-python.nix {
522528 skipLint = true;
523529 machine =
@@ -531,13 +537,13 @@ import ./make-test-python.nix {
531537 '';
532538}
533539</programlisting>
534− <para>
535− This will produce a Nix warning at evaluation time. To fully disable
536− the linter, wrap the test script in comment directives to disable
537− the Black linter directly (again, don't commit this within the
538− Nixpkgs repository):
539− </para>
540− <programlisting language="bash">
540+ <para>
541+ This will produce a Nix warning at evaluation time. To fully
542+ disable the linter, wrap the test script in comment directives to
543+ disable the Black linter directly (again, don't commit this within
544+ the Nixpkgs repository):
545+ </para>
546+ <programlisting language="bash">
541547 testScript =
542548 ''
543549 # fmt: off
@@ -545,4 +551,66 @@ import ./make-test-python.nix {
545551 # fmt: on
546552 '';
547553</programlisting>
554+ </section>
555+ <section xml:id="ssec-failing-tests-early">
556+ <title>Failing tests early</title>
557+ <para>
558+ To fail tests early when certain invariables are no longer met
559+ (instead of waiting for the build to time out), the decorator
560+ <literal>polling_condition</literal> is provided. For example, if
561+ we are testing a program <literal>foo</literal> that should not
562+ quit after being started, we might write the following:
563+ </para>
564+ <programlisting language="python">
565+@polling_condition
566+def foo_running():
567+ machine.succeed(&quot;pgrep -x foo&quot;)
568+
569+
570+machine.succeed(&quot;foo --start&quot;)
571+machine.wait_until_succeeds(&quot;pgrep -x foo&quot;)
572+
573+with foo_running:
574+ ... # Put `foo` through its paces
575+</programlisting>
576+ <para>
577+ <literal>polling_condition</literal> takes the following
578+ (optional) arguments:
579+ </para>
580+ <para>
581+ <literal>seconds_interval</literal>
582+ </para>
583+ <para>
584+ : specifies how often the condition should be polled:
585+ </para>
586+ <programlisting>
587+```py
588+@polling_condition(seconds_interval=10)
589+def foo_running():
590+ machine.succeed(&quot;pgrep -x foo&quot;)
591+```
592+</programlisting>
593+ <para>
594+ <literal>description</literal>
595+ </para>
596+ <para>
597+ : is used in the log when the condition is checked. If this is not
598+ provided, the description is pulled from the docstring of the
599+ function. These two are therefore equivalent:
600+ </para>
601+ <programlisting>
602+```py
603+@polling_condition
604+def foo_running():
605+ &quot;check that foo is running&quot;
606+ machine.succeed(&quot;pgrep -x foo&quot;)
607+```
608+
609+```py
610+@polling_condition(description=&quot;check that foo is running&quot;)
611+def foo_running():
612+ machine.succeed(&quot;pgrep -x foo&quot;)
613+```
614+</programlisting>
615+ </section>
548616</section>
nixos/lib/test-driver/default.nix+2−2
@@ -14,7 +14,7 @@
1414
1515python3Packages.buildPythonApplication rec {
1616 pname = "nixos-test-driver";
17− version = "1.0";
17+ version = "1.1";
1818 src = ./.;
1919
2020 propagatedBuildInputs = [ coreutils netpbm python3Packages.colorama python3Packages.ptpython qemu_pkg socat vde2 ]
@@ -26,7 +26,7 @@ python3Packages.buildPythonApplication rec {
2626 mypy --disallow-untyped-defs \
2727 --no-implicit-optional \
2828 --ignore-missing-imports ${src}/test_driver
29− pylint --errors-only ${src}/test_driver
29+ pylint --errors-only --enable=unused-import ${src}/test_driver
3030 black --check --diff ${src}/test_driver
3131 '';
3232}
nixos/lib/test-driver/setup.py+1−1
@@ -2,7 +2,7 @@ from setuptools import setup, find_packages
22
33setup(
44 name="nixos-test-driver",
5− version='1.0',
5+ version='1.1',
66 packages=find_packages(),
77 entry_points={
88 "console_scripts": [
nixos/lib/test-driver/test_driver/driver.py+40−1
@@ -1,12 +1,13 @@
11from contextlib import contextmanager
22from pathlib import Path
3−from typing import Any, Dict, Iterator, List
3+from typing import Any, Dict, Iterator, List, Union, Optional, Callable, ContextManager
44import os
55import tempfile
66
77from test_driver.logger import rootlog
88from test_driver.machine import Machine, NixStartScript, retry
99from test_driver.vlan import VLan
10+from test_driver.polling_condition import PollingCondition
1011
1112
1213class Driver:
@@ -16,6 +17,7 @@ class Driver:
1617 tests: str
1718 vlans: List[VLan]
1819 machines: List[Machine]
20+ polling_conditions: List[PollingCondition]
1921
2022 def __init__(
2123 self,
@@ -36,12 +38,15 @@ class Driver:
3638 for s in scripts:
3739 yield NixStartScript(s)
3840
41+ self.polling_conditions = []
42+
3943 self.machines = [
4044 Machine(
4145 start_command=cmd,
4246 keep_vm_state=keep_vm_state,
4347 name=cmd.machine_name,
4448 tmp_dir=tmp_dir,
49+ callbacks=[self.check_polling_conditions],
4550 )
4651 for cmd in cmd(start_scripts)
4752 ]
@@ -84,6 +89,7 @@ class Driver:
8489 retry=retry,
8590 serial_stdout_off=self.serial_stdout_off,
8691 serial_stdout_on=self.serial_stdout_on,
92+ polling_condition=self.polling_condition,
8793 Machine=Machine, # for typing
8894 )
8995 machine_symbols = {m.name: m for m in self.machines}
@@ -159,3 +165,36 @@ class Driver:
159165
160166 def serial_stdout_off(self) -> None:
161167 rootlog._print_serial_logs = False
168+
169+ def check_polling_conditions(self) -> None:
170+ for condition in self.polling_conditions:
171+ condition.maybe_raise()
172+
173+ def polling_condition(
174+ self,
175+ fun_: Optional[Callable] = None,
176+ *,
177+ seconds_interval: float = 2.0,
178+ description: Optional[str] = None,
179+ ) -> Union[Callable[[Callable], ContextManager], ContextManager]:
180+ driver = self
181+
182+ class Poll:
183+ def __init__(self, fun: Callable):
184+ self.condition = PollingCondition(
185+ fun,
186+ seconds_interval,
187+ description,
188+ )
189+
190+ def __enter__(self) -> None:
191+ driver.polling_conditions.append(self.condition)
192+
193+ def __exit__(self, a, b, c) -> None: # type: ignore
194+ res = driver.polling_conditions.pop()
195+ assert res is self.condition
196+
197+ if fun_ is None:
198+ return Poll
199+ else:
200+ return Poll(fun_)
nixos/lib/test-driver/test_driver/machine.py+9
@@ -318,6 +318,7 @@ class Machine:
318318 # Store last serial console lines for use
319319 # of wait_for_console_text
320320 last_lines: Queue = Queue()
321+ callbacks: List[Callable]
321322
322323 def __repr__(self) -> str:
323324 return f"<Machine '{self.name}'>"
@@ -329,12 +330,14 @@ class Machine:
329330 name: str = "machine",
330331 keep_vm_state: bool = False,
331332 allow_reboot: bool = False,
333+ callbacks: Optional[List[Callable]] = None,
332334 ) -> None:
333335 self.tmp_dir = tmp_dir
334336 self.keep_vm_state = keep_vm_state
335337 self.allow_reboot = allow_reboot
336338 self.name = name
337339 self.start_command = start_command
340+ self.callbacks = callbacks if callbacks is not None else []
338341
339342 # set up directories
340343 self.shared_dir = self.tmp_dir / "shared-xchg"
@@ -406,6 +409,7 @@ class Machine:
406409 return answer
407410
408411 def send_monitor_command(self, command: str) -> str:
412+ self.run_callbacks()
409413 with self.nested("sending monitor command: {}".format(command)):
410414 message = ("{}\n".format(command)).encode()
411415 assert self.monitor is not None
@@ -509,6 +513,7 @@ class Machine:
509513 def execute(
510514 self, command: str, check_return: bool = True, timeout: Optional[int] = 900
511515 ) -> Tuple[int, str]:
516+ self.run_callbacks()
512517 self.connect()
513518
514519 if timeout is not None:
@@ -969,3 +974,7 @@ class Machine:
969974 self.shell.close()
970975 self.monitor.close()
971976 self.serial_thread.join()
977+
978+ def run_callbacks(self) -> None:
979+ for callback in self.callbacks:
980+ callback()
nixos/lib/test-driver/test_driver/polling_condition.pynew file+77
@@ -0,0 +1,77 @@
1+from typing import Callable, Optional
2+import time
3+
4+from .logger import rootlog
5+
6+
7+class PollingConditionFailed(Exception):
8+ pass
9+
10+
11+class PollingCondition:
12+ condition: Callable[[], bool]
13+ seconds_interval: float
14+ description: Optional[str]
15+
16+ last_called: float
17+ entered: bool
18+
19+ def __init__(
20+ self,
21+ condition: Callable[[], Optional[bool]],
22+ seconds_interval: float = 2.0,
23+ description: Optional[str] = None,
24+ ):
25+ self.condition = condition # type: ignore
26+ self.seconds_interval = seconds_interval
27+
28+ if description is None:
29+ if condition.__doc__:
30+ self.description = condition.__doc__
31+ else:
32+ self.description = condition.__name__
33+ else:
34+ self.description = str(description)
35+
36+ self.last_called = float("-inf")
37+ self.entered = False
38+
39+ def check(self) -> bool:
40+ if self.entered or not self.overdue:
41+ return True
42+
43+ with self, rootlog.nested(self.nested_message):
44+ rootlog.info(f"Time since last: {time.monotonic() - self.last_called:.2f}s")
45+ try:
46+ res = self.condition() # type: ignore
47+ except Exception:
48+ res = False
49+ res = res is None or res
50+ rootlog.info(self.status_message(res))
51+ return res
52+
53+ def maybe_raise(self) -> None:
54+ if not self.check():
55+ raise PollingConditionFailed(self.status_message(False))
56+
57+ def status_message(self, status: bool) -> str:
58+ return f"Polling condition {'succeeded' if status else 'failed'}: {self.description}"
59+
60+ @property
61+ def nested_message(self) -> str:
62+ nested_message = ["Checking polling condition"]
63+ if self.description is not None:
64+ nested_message.append(repr(self.description))
65+
66+ return " ".join(nested_message)
67+
68+ @property
69+ def overdue(self) -> bool:
70+ return self.last_called + self.seconds_interval < time.monotonic()
71+
72+ def __enter__(self) -> None:
73+ self.entered = True
74+
75+ def __exit__(self, exc_type, exc_value, traceback) -> None: # type: ignore
76+ self.entered = False
77+ self.last_called = time.monotonic()
nixos/tests/vscodium.nix+30−20
@@ -34,36 +34,46 @@ let
3434 };
3535 enableOCR = true;
3636 testScript = ''
37+ @polling_condition
38+ def codium_running():
39+ machine.succeed('pgrep -x codium')
40+
41+
3742 start_all()
3843
3944 machine.wait_for_unit('graphical.target')
4045 machine.wait_until_succeeds('pgrep -x codium')
4146
42− # Wait until vscodium is visible. "File" is in the menu bar.
43− machine.wait_for_text('File')
44− machine.screenshot('start_screen')
47+ with codium_running:
48+ # Wait until vscodium is visible. "File" is in the menu bar.
49+ machine.wait_for_text('Get Started')
50+ machine.screenshot('start_screen')
51+
52+ test_string = 'testfile'
4553
46− test_string = 'testfile'
54+ # Create a new file
55+ machine.send_key('ctrl-n')
56+ machine.wait_for_text('Untitled')
57+ machine.screenshot('empty_editor')
4758
48− # Create a new file
49− machine.send_key('ctrl-n')
50− machine.wait_for_text('Untitled')
51− machine.screenshot('empty_editor')
59+ # Type a string
60+ machine.send_chars(test_string)
61+ machine.wait_for_text(test_string)
62+ machine.screenshot('editor')
5263
53− # Type a string
54− machine.send_chars(test_string)
55− machine.wait_for_text(test_string)
56− machine.screenshot('editor')
64+ # Save the file
65+ machine.send_key('ctrl-s')
66+ machine.wait_for_text('Save')
67+ machine.screenshot('save_window')
68+ machine.send_key('ret')
5769
58− # Save the file
59− machine.send_key('ctrl-s')
60− machine.wait_for_text('Save')
61− machine.screenshot('save_window')
62− machine.send_key('ret')
70+ # (the default filename is the first line of the file)
71+ machine.wait_for_file(f'/home/alice/{test_string}')
6372
64− # (the default filename is the first line of the file)
65− machine.wait_for_file(f'/home/alice/{test_string}')
73+ machine.send_key('ctrl-q')
74+ machine.wait_until_fails('pgrep -x codium')
6675 '';
6776 });
6877
69−in builtins.mapAttrs (k: v: mkTest k v { }) tests
78+in
79+builtins.mapAttrs (k: v: mkTest k v { }) tests