Commit Graph

21 Commits

Author SHA1 Message Date
David Peter
d13454a0d4 Add additional integration test for failing command 2022-02-22 22:51:09 +01:00
David Peter
e666b01e09 Add simple integation test for --shell=none 2022-02-22 22:51:09 +01:00
David Peter
365ff2584d Switch to --shell=none/-N, do not change the default 2022-02-22 22:51:09 +01:00
David Peter
d9418e0731 Use raw executor by default, skip intermediate shell
This adds a new "raw executor" (next to the "shell executor") that
allows hyperfine to execute commands directly without any intermediate
shell.

The command line is split into tokens (using the `shell-words` crate),
and according to POSIX rules. The first token is taken as the executable,
and the rest as arguments.

The new executor is enabled by default. In order to select the shell
executor, users will have to pass `--shell=default`.

This allows us to reduce measurement noise and to benchmark very quick
commands. It also decreases the time to run benchmarks, as we don't need
the calibration phase.

Also, it allows one to make sure that the executed command is not
implemented as a shell builtin. For example `hyperfine true`
and `hyperfine --shell=default true` return different times due
to the fact that `bash` executes `true` as a NOP.

Co-authored: Ciprian Dorin Craciun <ciprian@volution.ro>
2022-02-22 22:51:09 +01:00
David Peter
d27cbceed2 Add integration test for --show-output 2022-02-22 13:24:05 +01:00
David Peter
6887835f34 Proper shell overhead computation 2022-02-22 09:07:40 +01:00
David Peter
6f51232a8a Add debug/mock mode 2022-02-22 09:07:40 +01:00
David Peter
3348b37b8e Faster tests 2022-02-22 09:07:40 +01:00
David Peter
c0ce6b9537 Add integration test for '--shell' 2022-02-22 09:07:40 +01:00
David Peter
8c5ae87ca2 Better error handling in run_intermediate_command 2022-02-22 09:07:40 +01:00
David Peter
72d27729fa New Scheduler struct 2022-02-19 16:53:02 +01:00
David Peter
5df981ae7b Expand execution order tests 2022-02-06 22:50:11 +01:00
David Peter
34dabcbed7 Integration test for duplicate parameter names 2022-02-06 20:19:50 +01:00
David Peter
4d3314f26f Add integration test for prepare option 2022-02-06 20:19:50 +01:00
David Peter
b3c0022386 Add more integration tests 2022-02-06 20:19:50 +01:00
David Peter
4fa0aba266 Add new integration test 2022-02-06 20:19:50 +01:00
David Peter
c751a7735c Move execution order tests to separate file 2022-02-06 20:19:50 +01:00
Ævar Arnfjörð Bjarmason
017d55a4a1 Add --setup (-s) option, like --prepare but runs once per batch
Add a --setup option for the use-case of wanting to do one-off setup
before a set of benchmarks, not the once-per-test setup that --prepare
does. This is useful for the cases noted in the updated documentation.

Per the feedback on https://github.com/sharkdp/hyperfine/pull/448 this
new "--setup" option will steal the "-s" short option from
"--style" (initially this was called "--build" and used the
non-conflicting "-b").

Potential future work:

I'd prefer if this and --cleanup took N number of commands, so you
could do e.g.:

    hyperfine -L rev master,next \
        -b 'git worktree add /tmp/test-{r}'  \
        -b 'make -C /tmp/test-{r} all'  \
	'make -C /tmp/test-{r} test' \
	-c 'git worktree remove /tmp/test-{r}'

I.e. a shortcut around not providing these with &&, which makes things
a bit more readable.

But the --cleanup option doesn't do that, so let's just go with what
it's doing for consistency, so for this you'll now need to do:

    hyperfine -L rev master,next \
        -b 'git worktree add /tmp/test-{r} &&
	    make -C /tmp/test-{r} all'  \
	'make -C /tmp/test-{r} test' \
	-c 'git worktree remove /tmp/test-{r}'
2021-11-16 21:36:35 +01:00
Ævar Arnfjörð Bjarmason
bffd476128 Add support for one run with --runs=1
Extend the --runs=N option added in d78c33b (Added options to specify
the max/exact numbers of runs., 2018-09-09) to support --runs=1
instead of dying with a usage error.

This is useful to do ad-hoc testing of your commands while they might
still have syntax errors, or just for doing one run where you don't
care about the stddev.

Before:

    $ /usr/bin/hyperfine -s basic -r 1 -L n 5,10 'sleep 0.{n}'
    Error: Number of runs below two

After (-s basic) also works:

    $ hyperfine -r 1 -L n 5,10 'sleep 0.{n}'
    Benchmark 1: sleep 0.5
      Time (abs ≡):        500.6 ms               [User: 0.6 ms, System: 0.0 ms]

    Benchmark 2: sleep 0.10
      Time (abs ≡):        100.8 ms               [User: 0.7 ms, System: 0.0 ms]

    Summary
      'sleep 0.10' ran
        4.97 ± 0.00 times faster than 'sleep 0.5'

This likewise combines correctly with -m and -M, probably not very
useful, but if you're tweaking an existing command-line:

    $ hyperfine -m 1 -M 1 -L n 5,10 'sleep 0.{n}'
    Benchmark 1: sleep 0.5
      Time (abs ≡):        500.6 ms               [User: 0.5 ms, System: 0.0 ms]

    Benchmark 2: sleep 0.10
      Time (abs ≡):        100.6 ms               [User: 0.6 ms, System: 0.0 ms]

    Summary
      'sleep 0.10' ran
        4.98 ± 0.00 times faster than 'sleep 0.5'

The "± 0.00" output in "faster than" should probably be adjusted too,
or we could keep it for consistency. I didn't implement that because
this is the first time I do anything in Rust, and I ran out of
template to copy when wanting to quickly implement this in
write_benchmark_comparison() in main.rs.
2021-11-14 10:28:24 +01:00
David Peter
d31936ec28 Fix tests on Windows 2021-08-23 22:37:02 +02:00
David Peter
e71ff559f3 Add simple integration tests 2021-08-23 22:37:02 +02:00