@@ -538,6 +538,99 @@ buildPythonPackage rec {
538538```
539539Note also the line `doCheck = false;`, we explicitly disabled running the test-suite.
540540
541+#### Testing Python Packages
542+
543+It is highly encouraged to have testing as part of the package build. This
544+helps to avoid situations where the package was able to build and install,
545+but is not usable at runtime. Currently, all packages will use the `test`
546+command provided by the setup.py. However, this is currently deprecated
547+https://github.com/pypa/setuptools/pull/1878 and your package should provide
548+it's own checkPhase.
549+
550+*NOTE:* The `checkPhase` for python maps to the `installCheckPhase` on a
551+normal derivation. This is due to many python packages not behaving well
552+to the pre-installed version of the package. Version info, and natively
553+compiled extensions generally only exist in the install directory, and
554+thus can cause issues when a test suite asserts on that behavior.
555+
556+*NOTE:* Tests should only be disabled if they don't agree with nix
557+(e.g. external dependencies, network access, flakey tests), however,
558+as many tests should be enabled as possible. Failing tests can still be
559+a good indication that the package is not in a valid state.
560+
561+#### Using pytest
562+
563+Pytest is the most common test runner for python repositories. A trivial
564+test run would be:
565+```
566+ checkInputs = [ pytest ];
567+ checkPhase = "pytest";
568+```
569+
570+However, many repositories' test suites do not translate well to nix's build
571+sandbox, and will generally need many tests to be disabled.
572+
573+To filter tests using pytest, one can do the following:
574+```
575+ checkInputs = [ pytest ];
576+ # avoid tests which need additional data or touch network
577+ checkPhase = ''
578+ pytest tests/ --ignore=tests/integration -k 'not download and not update'
579+ '';
580+```
581+
582+`--ignore` will tell pytest to ignore that file or directory from being
583+collected as part of a test run. This is useful is a file uses a package
584+which is not available in nixpkgs, thus skipping that test file is much
585+easier than having to create a new package.
586+
587+`-k` is used to define a predicate for test names. In this example, we are
588+filtering out tests which contain `download` or `update` in their test case name.
589+Only one `-k` argument is allows, and thus a long predicate should be concatenated
590+with "\" and wrapped to the next line.
591+
592+*NOTE:* In pytest==6.0.1, the use of "\" to continue a line (e.g. `-k 'not download \'`) has
593+been removed, in this case, it's recommended to use `pytestCheckHook`.
594+
595+#### Using pytestCheckHook
596+
597+`pytestCheckHook` is a convenient hook which will substitute the setuptools
598+`test` command for a checkPhase which runs `pytest`. This is also beneficial
599+when a package may need many items disabled to run the test suite.
600+
601+Using the example above, the analagous pytestCheckHook usage would be:
602+```
603+ checkInputs = [ pytestCheckHook ];
604+
605+ # requires additional data
606+ pytestFlagsArray = [ "tests/" "--ignore=tests/integration" ];
607+
608+ disabledTests = [
609+ # touches network
610+ "download"
611+ "update"
612+ ];
613+```
614+
615+This is expecially useful when tests need to be conditionallydisabled,
616+for example:
617+
618+```
619+ disabledTests = [
620+ # touches network
621+ "download"
622+ "update"
623+ ] ++ lib.optionals (pythonAtLeast "3.8") [
624+ # broken due to python3.8 async changes
625+ "async"
626+ ] ++ lib.optionals stdenv.isDarwin [
627+ # can fail when building with other packages
628+ "socket"
629+ ];
630+```
631+Trying to concatenate the related strings to disable tests in a regular checkPhase
632+would be much harder to read. This also enables us to comment on why specific tests
633+are disabled.
541634
542635#### Develop local package
543636
@@ -1017,7 +1110,7 @@ are used in `buildPythonPackage`.
10171110- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system
10181111 (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`.
10191112- `pipInstallHook` to install wheels.
1020−- `pytestCheckHook` to run tests with `pytest`.
1113+- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
10211114- `pythonCatchConflictsHook` to check whether a Python package is not already existing.
10221115- `pythonImportsCheckHook` to check whether importing the listed modules works.
10231116- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.