chia-blockchain/tests
Kyle Altendorf 2eb4fdeaee
Disable the pytest-monitor plugin in CI if not checking results (#10837)
* disable the pytest-monitor plugin if not reporting results

pytest-monitor uses multiprocessing and has caused multiple confusing
issues.  Perhaps it can be adjusted to not use multiprocessing, but
for now lets just isolate the oddities to where we actually use it.

* use a template for resource usage check, similar to timelord install

* hint testconfig.custom_vars
2022-03-24 09:30:42 -07:00
..
blockchain run tests in parallel in CI (#10499) 2022-03-22 15:04:07 -07:00
clvm Improve the CI runs w.r.t. timelord installation (#10673) 2022-03-17 09:05:33 -07:00
core Disable the pytest-monitor plugin in CI if not checking results (#10837) 2022-03-24 09:30:42 -07:00
farmer_harvester Change name to order of returned values. Enforce mandatory naming and inclusion of start_services parameter (#10769) 2022-03-17 08:39:30 -07:00
generator run tests in parallel in CI (#10499) 2022-03-22 15:04:07 -07:00
plotting run tests in parallel in CI (#10499) 2022-03-22 15:04:07 -07:00
pools minor followup to config locking (#10696) 2022-03-17 09:03:58 -07:00
runner_templates Disable the pytest-monitor plugin in CI if not checking results (#10837) 2022-03-24 09:30:42 -07:00
simulation Improve the CI runs w.r.t. timelord installation (#10673) 2022-03-17 09:05:33 -07:00
tools run tests in parallel in CI (#10499) 2022-03-22 15:04:07 -07:00
util use DEFAULT_ROOT_PATH in tests (#10801) 2022-03-24 09:29:05 -07:00
wallet run tests in parallel in CI (#10499) 2022-03-22 15:04:07 -07:00
weight_proof run tests in parallel in CI (#10499) 2022-03-22 15:04:07 -07:00
__init__.py Added tests, improved serialization 2019-08-05 14:38:16 +09:00
block_tools.py use DEFAULT_ROOT_PATH in tests (#10801) 2022-03-24 09:29:05 -07:00
build-init-files.py do not generate __init__.py in .pytest_cache (#10270) 2022-02-17 07:27:22 -08:00
build-workflows.py Disable the pytest-monitor plugin in CI if not checking results (#10837) 2022-03-24 09:30:42 -07:00
check_pytest_monitor_output.py add stress tests with malicious generators (#8570) 2021-09-29 11:41:49 -07:00
chia-start-sim Rename src to chia. 2021-04-04 21:41:39 -07:00
conftest.py Ak.setup nodes (#10619) 2022-03-16 11:47:54 -07:00
connection_utils.py Remove block tools and keychain globals (#10358) 2022-03-10 11:06:49 -08:00
pytest.ini run tests in parallel in CI (#10499) 2022-03-22 15:04:07 -07:00
README.md Improve style consistency in root directory docs files (#4493) 2021-05-12 13:12:30 -07:00
setup_nodes.py Remove accidental parameters from calls to setup_simulators_and_wallets and prevent future mistakes (#10770) 2022-03-21 16:26:11 -07:00
setup_services.py when running multiple services in the same process (in tests), don't initialize logging for all of them, and don't set the proctitle of the test (#10686) 2022-03-17 14:58:30 -07:00
testconfig.py Disable the pytest-monitor plugin in CI if not checking results (#10837) 2022-03-24 09:30:42 -07:00
time_out_assert.py Timed assertion output simplification (#9290) 2021-11-19 11:14:18 -08:00
wallet_tools.py New RPC get_coin_records_by_hint - Get coins for a given hint (#10715) 2022-03-17 08:48:09 -07:00

Test CI Job generation

The CI jobs for these tests are managed by build-workflows.py.

If you add a test file, or move one to another directory, please run build-workflows.py. Tests are recognized by the file glob test_*.py. Changing the contents of a file does not require running build-workflows.py.

We currently use github actions. Default runners have two vcpus. The workflows are located in ../.github/workflows/.

The inputs to build-workflows.py are the templates in runner-templates, the file testconfig.py in this directory, and the optional config.py files in some test subdirectories. Files in the template directory ending in include.yml are included in jobs based on the per-directory settings.

The generated workflows are output to ../.github/workflows/.

Each subdirectory below the directories root_test_dirs in testconfig.py becomes a job in the github workflow matrix. If your jobs run too long, simply move some tests into new subdirectories and run build-workflows.py. A workflow built from a parent directory does not include the tests in its subdirectories. The subdirectory jobs do not include the tests from their parents.

testconfig.py

In the top tests directory, testconfig.py contains the application settings and the per-directory default settings.

config.py

Each directory has an optional config.py file, which can override the per-directory default settings.

Per directory settings defaults:

parallel = False
checkout_blocks_and_plots = True
install_timelord = True
job_timeout = 30

Parallel test execution

If you are certain that all the tests in a directory can run in parallel, set parallel = True in config.py inside that directory.

Optional job stages

Set checkout_blocks_and_plots to False to omit checking out the test-cache repo.

Set install_timelord to False to omit the step of installing a Time Lord for your directory's job.

Job Timeout

Set job_timeout to the number of minutes you want the CI system to wait before it kills your job. Add two or three minutes to allow for job setup.