git.iko.soy/typeable/nixpkgs
on master help sign in
Merge pull request #95700 from jonringer/doc-pytestcheckhook

doc/python: add pytestCheckHook section

commit6770ab0771703b573a0a1c1f6c238352bebc9c31 authorFrederik Rietdijk <freddyrietdijk@fridh.nl> dateTue 18 Aug 2020, 08:21 (6 years ago) parentc581528233dc9c
1 file changed+117−2
doc/languages-frameworks/python.section.md+117−2
doc/languages-frameworks/python.section.md+117−2
@@ -538,8 +538,123 @@ buildPythonPackage rec {
538538```
539539Note also the line `doCheck = false;`, we explicitly disabled running the test-suite.
540540
541+#### Testing Python Packages
542+
543+It is highly encouraged to have testing as part of the package build. This
544+helps to avoid situations where the package was able to build and install,
545+but is not usable at runtime. Currently, all packages will use the `test`
546+command provided by the setup.py (i.e. `python setup.py test`). However,
547+this is currently deprecated https://github.com/pypa/setuptools/pull/1878
548+and your package should provide its own checkPhase.
549+
550+*NOTE:* The `checkPhase` for python maps to the `installCheckPhase` on a
551+normal derivation. This is due to many python packages not behaving well
552+to the pre-installed version of the package. Version info, and natively
553+compiled extensions generally only exist in the install directory, and
554+thus can cause issues when a test suite asserts on that behavior.
555+
556+*NOTE:* Tests should only be disabled if they don't agree with nix
557+(e.g. external dependencies, network access, flakey tests), however,
558+as many tests should be enabled as possible. Failing tests can still be
559+a good indication that the package is not in a valid state.
560+
561+#### Using pytest
562+
563+Pytest is the most common test runner for python repositories. A trivial
564+test run would be:
565+```
566+ checkInputs = [ pytest ];
567+ checkPhase = "pytest";
568+```
569+
570+However, many repositories' test suites do not translate well to nix's build
571+sandbox, and will generally need many tests to be disabled.
572+
573+To filter tests using pytest, one can do the following:
574+```
575+ checkInputs = [ pytest ];
576+ # avoid tests which need additional data or touch network
577+ checkPhase = ''
578+ pytest tests/ --ignore=tests/integration -k 'not download and not update'
579+ '';
580+```
581+
582+`--ignore` will tell pytest to ignore that file or directory from being
583+collected as part of a test run. This is useful is a file uses a package
584+which is not available in nixpkgs, thus skipping that test file is much
585+easier than having to create a new package.
586+
587+`-k` is used to define a predicate for test names. In this example, we are
588+filtering out tests which contain `download` or `update` in their test case name.
589+Only one `-k` argument is allows, and thus a long predicate should be concatenated
590+with "\" and wrapped to the next line.
591+
592+*NOTE:* In pytest==6.0.1, the use of "\" to continue a line (e.g. `-k 'not download \'`) has
593+been removed, in this case, it's recommended to use `pytestCheckHook`.
594+
595+#### Using pytestCheckHook
596+
597+`pytestCheckHook` is a convenient hook which will substitute the setuptools
598+`test` command for a checkPhase which runs `pytest`. This is also beneficial
599+when a package may need many items disabled to run the test suite.
600+
601+Using the example above, the analagous pytestCheckHook usage would be:
602+```
603+ checkInputs = [ pytestCheckHook ];
604+
605+ # requires additional data
606+ pytestFlagsArray = [ "tests/" "--ignore=tests/integration" ];
607+
608+ disabledTests = [
609+ # touches network
610+ "download"
611+ "update"
612+ ];
613+```
614+
615+This is expecially useful when tests need to be conditionallydisabled,
616+for example:
617+
618+```
619+ disabledTests = [
620+ # touches network
621+ "download"
622+ "update"
623+ ] ++ lib.optionals (pythonAtLeast "3.8") [
624+ # broken due to python3.8 async changes
625+ "async"
626+ ] ++ lib.optionals stdenv.isDarwin [
627+ # can fail when building with other packages
628+ "socket"
629+ ];
630+```
631+Trying to concatenate the related strings to disable tests in a regular checkPhase
632+would be much harder to read. This also enables us to comment on why specific tests
633+are disabled.
634+
635+#### Using pythonImportsCheck
636+
637+Although unit tests are highly prefered to valid correctness of a package. Not
638+all packages have test suites that can be ran easily, and some have none at all.
639+To help ensure the package still works, `pythonImportsCheck` can attempt to import
640+the listed modules.
641+
642+```
643+ pythonImportsCheck = [ "requests" "urllib" ];
644+```
645+roughly translates to:
646+```
647+ postCheck = ''
648+ PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH
649+ python -c "import requests; import urllib"
650+ '';
651+```
652+However, this is done in it's own phase, and not dependent on whether `doCheck = true;`
653+
654+This can also be useful in verifying that the package doesn't assume commonly
655+present packages (e.g. `setuptools`)
541656
542−#### Develop local package
657+### Develop local package
543658
544659As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode)
545660(`python setup.py develop`); instead of installing the package this command
@@ -1017,7 +1132,7 @@ are used in `buildPythonPackage`.
10171132- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system
10181133 (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`.
10191134- `pipInstallHook` to install wheels.
1020−- `pytestCheckHook` to run tests with `pytest`.
1135+- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
10211136- `pythonCatchConflictsHook` to check whether a Python package is not already existing.
10221137- `pythonImportsCheckHook` to check whether importing the listed modules works.
10231138- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.