Commit Graph

97 Commits

Author SHA1 Message Date
Amine Khaldi
173f919fb7
Merge commit '10d09e6cd554a7f6a2b61b5373cdf5ac9cf82a21' into checkpoint/main_from_release_2.0.0_10d09e6cd554a7f6a2b61b5373cdf5ac9cf82a21 2023-08-18 20:17:31 +01:00
Amine Khaldi
936d1f9c89
Merge commit 'b59315836dd086e6506f10fd2bef3e620baabff0' into checkpoint/main_from_release_2.0.0_b59315836dd086e6506f10fd2bef3e620baabff0 2023-08-18 18:35:58 +01:00
Earle Lowe
10d09e6cd5
Always use at least one worker for chia plots check (#16099)
* Always use at least one worker

* Fix type

* Black
2023-08-17 21:40:47 -05:00
Arvid Norberg
7b0bea4043
various typing fixes (#15899)
* add blspy-stubs to improve mypy

* various typing fixes

* avoid cast

* use ValueError exception instead of assert when aborting picking a wallet key
2023-08-17 19:10:40 -05:00
Earle Lowe
b59315836d
Adjust harvester warning to 8 seconds from 5 (#16100)
* Adjust harvester warning to 8 seconds from 5

* Update time in check plots to match harvester warning

* Fix up warning message
2023-08-17 18:37:18 -05:00
Florin Chirica
0deb8e6dc6
Timeout for retrieving a decompressor. (#16047)
* Timeout for retrieving a decompressor.

* Pre-commit fix

* Use new chiapos

* Use chiapos 2.0.1

---------

Co-authored-by: wallentx <william.allentx@gmail.com>
2023-08-16 18:40:29 -05:00
Florin Chirica
cbf971e7f5
Don't farm compressed plots if context count is 0 (#16004)
* Don't farm compressed plots if context count is 0

* Move checks outside cache.
2023-08-11 13:40:49 -05:00
William Allen
3a3b7e5ac2
Fix default parallel decompressor count (#15845) 2023-07-24 21:49:45 -05:00
Izumi Hoshino
ec5ef087fe
Added defaults to get_harvester_config API (#15820) 2023-07-20 13:25:16 -05:00
Izumi Hoshino
85d14f561a
Added compression level and harvesting mode to harvester protocol/mes… (#15776)
* Added compression level and harvesting mode to harvester protocol/messages

* Added test

* Fixed lint error
2023-07-18 16:24:56 -05:00
wjblanke
69674ccf46
Fixed add lock around plot info (#15767)
* add lock around plot info

* set threads back to 5
2023-07-14 10:15:22 -05:00
William Allen
f96690bdc5
Renaming instances of 'decompresser' to 'decompressor' (#15751)
* Renaming instances of 'decompresser' to 'decompressor'

* Making 'parallel_decompressors_count' singular

* Using chiapos version that has matching spelling

* Putting the clsp file back where I found it

* fix ver

---------

Co-authored-by: Jack Nelson <j.nelson@chia.net>
2023-07-13 09:59:51 -05:00
Arvid Norberg
7b67fe0a4a
compressed plot support (#15702)
* Compression branch.

* disable GPU farming in tests

---------

Co-authored-by: Florin Chirica <fchirica96@gmail.com>
2023-07-12 16:49:15 -05:00
Arvid Norberg
2559fbe6c4
fix typo in chia plots check (#15727) 2023-07-10 14:34:03 -05:00
Arvid Norberg
01e6e7776f
multi threaded chia plots check (#15693)
* improve harvester warning messages

* run plot check in multiple threads
2023-07-06 15:38:26 -05:00
dustinface
ebc5f3c124
util: Improve list_to_batches (#15415)
* Make `list_to_batches` work with collections instead of lists only

* Move `to_batches` into `chia.util.misc`

* Only support `set` and `list`

* Drop `tests.generator.test_to_batches` exclusion

* Improve type restrictions and be more coverage friendly in tests
2023-06-06 12:00:07 -05:00
Kyle Altendorf
d84917f5df
hint chia.plotting.create_plots (#14692)
* hint chia.plotting.create_plots

* correct the import

* fixup

* imports

* get final from typing_extensions
2023-03-08 12:57:50 -06:00
Kyle Altendorf
3a5c1a5bd5
hint chia.harvester.harvester, harvester_api (#14690)
* hint chia.harvester.harvester

* hint chia.harvester.harvester_api
2023-02-28 11:42:58 -06:00
Kyle Altendorf
0ed70fd534
hint chia.plotting.check_plots (#14691)
* hint chia.plotting.check_plots

* use the hints
2023-02-28 11:42:39 -06:00
Kyle Altendorf
96dad39392
remove unneeded parentheses (#14654) 2023-02-23 10:42:44 -06:00
Kyle Altendorf
41ca9622a3
close the keychain proxy in more failure and cancellation cases (#14415) 2023-02-22 02:51:26 -06:00
Arvid Norberg
27cbd08229
Factor out member functions of ProofOfSpace to be free functions (#13930) 2022-11-17 13:46:22 -06:00
dustinface
978c3708aa
Catch exceptions raised when resolving plot directories (#13786)
* plotting: Catch exceptions raised when resolving plot directories

* `try/except/else` -> `try/except/continue`
2022-11-10 19:49:31 -06:00
Izumi Hoshino
42e67f049e
Prevent a crash that happens when config["harvester"]["plot_directories"] is not a list (#13551) 2022-09-30 17:49:07 -05:00
Kyle Altendorf
3b084a165b
configure isort to add the future annotations import (#13327)
* configure isort to add the future annotations import

* apply the new isort setting

* remove type ignores for new mypy (#13539)

https://pypi.org/project/mypy/0.981/

* another
2022-09-30 03:40:22 -05:00
dustinface
ec059e9a4f
harvester: Some log updates (#13508)
* Log the number of cache entries and the time it took to parse them

* Log `started` and `done` as `INFO` and `batch_processed` as `DEBUG`

* Log the plot found message with `DEBUG` level

* Some additional debug logs in `new_signage_point_harvester`
2022-09-29 14:53:06 -05:00
Kyle Altendorf
bbd032e1c9
embrace click 8 (#13559)
* click==8.1.3, black==22.8.0

* black

* click option creator hinting

* ignores for to-be-resolved issue

* remove unused import
2022-09-29 11:24:36 -05:00
Kyle Altendorf
4bcbcdbbe4
remove type ignores for new mypy (#13539)
https://pypi.org/project/mypy/0.981/
2022-09-26 16:54:19 -05:00
dustinface
0df4d8cbe4
setup|plotting: chiapos==1.0.11 + drop invalid cache entries (#13502)
* Bump `chiapos` to `1.0.11`

* Drop invalid cache entries
2022-09-23 17:30:30 -05:00
Izumi Hoshino
190f8e1be2
Suppress debug log (#13476)
* Suppress debug log

* Delete the lines
2022-09-19 20:20:16 -05:00
joshpainter
29643d1feb
Fix bug in remove_plot_directory (#13332)
Revert commit c400d816af
2022-09-06 13:04:49 -05:00
Jack Nelson
ff8ecb35a0
Fix no keys loaded error by making KeychainProxy automatically reconnect when a connection is lost (#12782)
* Add proper types to daemon client & keychain

* pre type pass on keychain proxy

* add auto restarting keychain proxy

* finalize keychain proxy v2

* move event calls

* fix close on timeout

* fix possible key leak in logging

* fix last little error

* log on disconnection & reduce duplicate code

* Update client.py
2022-08-04 20:28:58 -05:00
Kyle Altendorf
90516263aa
remove chia.util.path.mkdir() (#10842)
* remove chia.util.path.mkdir()

* remove submodule change

* less tmpdir, more tmp_path
2022-06-08 10:24:40 -05:00
dustinface
9b3bed885e
harvester: Make PlotsRefreshParameter streamable + use from_json_dict (#11759)
* Make `PlotsRefreshParameter` streamable

* Use `from_json_dict` instead of `dataclass_from_dict`

* Fix `test_plot_manager.py`
2022-06-03 12:35:50 -05:00
dustinface
15c51434f3
plotting|util: plotting.cache.DiskCache -> util.misc.VersionedBlob (#11587) 2022-05-24 16:20:38 -07:00
dustinface
c400d816af
harvester: Validate the path in add_plot_directory (#11625)
Make sure it exists, is a directory and hasn't been added already.
2022-05-24 13:21:44 -07:00
Kyle Altendorf
a9f7cce06b
Skip plot_sync_callback if delta is None, hint Harvester.server (#11620)
* hint Harvester.server as Optional[ChiaServer]

* use a Protocol instead of Callable

* oh right, Optional
2022-05-24 13:21:14 -07:00
dustinface
d73fa27ea7
harvester: Sort plot_paths before splitting it into batches (#11508)
After changing `plot_paths` to be a set in #11204 the plots are loaded 
out of order. Sorting the list before splitting it to batches recovers 
the load order. 

Fixes #11499
2022-05-24 13:20:27 -07:00
dustinface
541b99e311
harvester: Introduce recursive_plot_scan (#11468)
* harvester: Introduce `recursive_plot_scan`

* Change log and merge boths lookups

* Drop redundant assignments
2022-05-09 16:43:53 -07:00
dustinface
7d11d54f89
plotting: Enable isort (#11135) 2022-04-21 17:00:00 -07:00
dustinface
553e46ab6d
harvester: Use a set instead of a list to speed up availability checks (#11204) 2022-04-20 11:43:31 -07:00
dustinface
1e7703fe6a
plotting|tests|setup: Improve PlotManager cache (#9903)
* plotting|tests: Extend the lifetime of plot data cache entries

* plotting|tests: Add `prover` and `farmer_public_key` to the cache

* plotting: Missing type hints in `Cache` and `CacheEntry`

* setup: Bump `chiapos` to 1.0.10

* plotting: Move cache classes into `chia.plotting.cache`

* plotting: Introduce `CacheEntry.from_disk_prover`

* Make `Cache` a dataclass

* `staticmethod` -> `classmethod`

* Store cache data as `bytes` in `DiskCache`
2022-04-20 11:08:35 -07:00
Mariano Sorgente
9ff3fc993f
Ms.plot load perf2 (#10978)
* 2.7 seconds -> 0.45 seconds

* Merge

* Work on create_plots refactor

* Try to fix tests

* Try to fix tests

* Use new functions

* Fix block_tools by adding dir

* Extra argument

* Try to fix cyclic import

* isort

* Drop warning

* Some cleanups around `exclude_final_dir` and directory adding

* Cleanup `min_mainnet_k_size` checks

* Drop unrelated changes

* Fixes after rebase

* Fix cyclic import

* Update tests/block_tools.py

Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>

* Update tests/block_tools.py

Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>

Co-authored-by: xdustinface <xdustinfacex@gmail.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
2022-04-11 09:16:01 -07:00
dustinface
a48fd43100
streamable: Simplify and force correct usage (#10509)
* streamable: Merge `strictdataclass` into `Streamable` class

* tests: Test not supported streamable types

* streamable: Reorder decorators

* streamable: Simplify streamable decorator and force correct usage/syntax

* streamable: Just move some stuff around in the file

* streamable: Improve syntax error messages

* mypy: Drop `type_checking.py` and `test_type_checking.py` from exclusion

* streamable: Use cached fields instead of `__annotations__`

This is now possible after merging `__post_init__` into `Streamable`

* Introduce `DefinitionError` as `StreamableError`

* `/t` -> `    `
2022-04-08 18:29:32 -07:00
Mariano Sorgente
2f9e718073
Ms.fast test blockchain (#11051)
* more work on test blockchain

* Optimize test_blockchain.py

* Fix weight proof bug

* Rename variable

* first rc_sub_slot hash bug

* New plots

* try with a new ID

* Run without cache

* Address test blocks and plots preparation.

* Update constant in test_compact_protocol().

* Update this constant too.

* Revert accidental altering of the gui submodule in ae7e3295f280a591e76c4dffdea75fb74ea5de6f.

* Fix benchmark test

* Revert mozilla-ca change

* Rebase on main

Co-authored-by: almog <almogdepaz@gmail.com>
Co-authored-by: Amine Khaldi <amine.khaldi@reactos.org>
2022-04-08 09:57:59 -07:00
dustinface
ded9f68583
chia|tests|github: Implement, integrate and test plot sync protocol (#9695)
* protocols|server: Define new harvester plot refreshing protocol messages

* protocols: Bump `protocol_version` to `0.0.34`

* tests: Introduce `setup_farmer_multi_harvester`

Allows to run a test setup with 1 farmer and mutiple harvesters.

* plotting: Add an initial plot loading indication to `PlotManager`

* plotting|tests: Don't add removed duplicates to `total_result.removed`

`PlotRefreshResult.removed` should only contain plots that were loaded
properly before they were removed. It shouldn't contain e.g. removed
duplicates or invalid plots since those are synced in an extra sync step
and not as diff but as whole list every time.

* harvester: Reset `PlotManager` on shutdown

* plot_sync: Implement plot sync protocol

* farmer|harvester: Integrate and enable plot sync

* tests: Implement tests for the plot sync protocol

* farmer|tests: Drop obsolete harvester caching code

* setup: Add `chia.plot_sync` to packages

* plot_sync: Type hints in `DeltaType`

* plot_sync: Drop parameters in `super()` calls

* plot_sync: Introduce `send_response` helper in `Receiver._process`

* plot_sync: Add some parentheses

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* plot_sync: Additional hint for a `Receiver.process_path_list` parameter

* plot_sync: Force named parameters in `Receiver.process_path_list`

* test: Fix fixtures after rebase

* tests: Fix sorting after rebase

* tests: Return type hint for `plot_sync_setup`

* tests: Rename `WSChiaConnection` and move it in the outer scope

* tests|plot_sync: More type hints

* tests: Rework some delta tests

* tests: Drop a `range` and iterate over the list directly

* tests: Use the proper flags to overwrite

* test: More missing duplicates tests

* tests: Drop `ExpectedResult.reset`

* tests: Reduce some asserts

* tests: Add messages to some `assert False` statements

* tests: Introduce `ErrorSimulation` enum in `test_sync_simulated.py`

* tests: Use `secrects` instead of `Crypto.Random`

* Fixes after rebase

* Import from `typing_extensions` to support python 3.7

* Drop task name to support python 3.7

* Introduce `Sender.syncing`, `Sender.connected` and a log about the task

* Add `tests/plot_sync/config.py`

* Align the multi harvester fixture with what we do in other places

* Update the workflows

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-04-07 17:10:44 -07:00
Kyle Altendorf
e0a1fc4eaf
Switch to integrated lock_and_load_config() context manager (#10698)
* minor lock scope reduction

* use the lock in tests

* Use the passed root_path in configure CLI command

* switch to lock_and_load_config()

* oops

* cleanup

* make _load_config_maybe_locked() private

* black

* Remove future improvement opportunity TODO comment
2022-03-28 12:52:51 -07:00
Kyle Altendorf
48cd97cf47
minor followup to config locking (#10696)
* minor lock scope reduction

* use the lock in tests
2022-03-17 09:03:58 -07:00
Kyle Altendorf
dbeff36ae0
Add configuration locking (#10680)
* Add configuration locking

Extracted from https://github.com/Chia-Network/chia-blockchain/pull/10631

* note that fasteners will likely be replaced by filelock

* Fix test_multiple_writers on macOS

* create_all_ssl() doesn't need to be inside the config access lock

* add warnings about not using async within get_config_lock() get lock contexts

* no need to pre-touch the lock file

* .yaml.lock instead of just .lock

* test_multiple_writers() is sync

* Revert "add warnings about not using async within get_config_lock() get lock contexts"

This reverts commit 681af3835b.

* reduce lock context size in chia_init()

* use an exit stack in load_config()

* avoid config existence precheck

* only lock around the read in load_config()

* do not raise e, just raise

* tidy new imports

* fix queue empty check in test_config.py

* remove commented out code in test_config.py

* remove unused import

Co-authored-by: Jeff Cruikshank <jeff@chia.net>
2022-03-12 06:52:47 -08:00
dustinface
0bf370c642
plotting: Only lock while actually accessing PlotManager.plots (#10675) 2022-03-11 16:23:01 -08:00