* explore a simplification of the blockchain priority lock queue
* add some tests
* correct task tracking
* use time.perf_counter for better resolution on windows
* just count an integer for request order retention
* stop using time in the (new) tests as well
* add tests and a couple no covers
* less existing test refactoring
* use a sync PriorityQueue
* switch to deques
* address bugs and simplify priority to deque mapping
* remove unused attribute ._priority_type
* make LockQueu.create() not async
* explain the active element check on wait cancellation
* drop LockQueue._process()
* import final from typing_extensions
* rename LockQueue to PriorityMutex
* remove test from mypy exclusions
* clean up straggling lock references
* ignore test failure case line coverage
* add a monkeypatch test ;[
* remove queued callback feature
* remove todos
* Make `list_to_batches` work with collections instead of lists only
* Move `to_batches` into `chia.util.misc`
* Only support `set` and `list`
* Drop `tests.generator.test_to_batches` exclusion
* Improve type restrictions and be more coverage friendly in tests
* added get_wallet_balances rpc api
* whitespace
* get_wallet_balances for GUI convenience
* test for specified ids
* wait for bundles to get into mempool
* Convert blockchain_constants's consensus_mode param into a fixture and use it to run daemon_simulation one at a time.
* Simulation test seems to support only one running at a time too.
* update build_network_protocol_files.py to generate the mypy-compatible files (which were fixed already, but the generator script was not). Specifically, tests/util/test_network_protocol_files.py was changed manually. This makes it possible to rerun the generation without breaking mypy
* run build_network_protocol_files.py in CI to ensure its generated files are up-to-date
* fixup mypy exclusions
make sure tests that use BlockTools use a single source of consensus constants. This is in preparations for being able to run tests with different sets of constants, e.g. the hard-fork constants
* remove softfork-logic for 1.7 softfork (which has already activated)
* add new constants for soft-fork3, hard-fork and the other plot-filter adjustments
* Add some more daemon "rpc" tests
* fixes due to some platform differences
* Add unlock_keyring tests and some experiments with monkeypatch
* rewrote simpler tests into single paramterized test
* add parameters to patch function
* Use monkeypatch.context for better isolation
* remove monkeypatch and test by deleting the keyring file
* remove now unused monkeypatch function
* changes from PR suggestions
* more readable parameterization per suggestions
* datacases
* description and id property
* swap description and route order
* *cases
---------
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* Exploring identical spend aggregation.
* Attempt to address review.
* Revert this.
* Explore spend-based processing instead of the bundle-based one.
* Attempt to address reviews.
* Update against chia_rs and add some improvements.
* Leverage the fact that we're in a function now to perform early exits.
* Explore propagating exceptions.
* Relax the exception handling.
* Add unit tests.
* Add some comments and perform some renames for more readability.
* Refactor tests and split them by scenario.
* Take Arvid's suggestion to simplify the check.
* Improve test readability.
* Make the set explicit instead of computing it with union().
* Run in MEMPOOL_MODE for cost and additions.
* Use int_from_bytes() instead of int.from_bytes().
* Add more unit tests for run_for_cost_and_additions (to cover coin amount 0 as well as a negative amount).
* Don't use as_python() for created coins.
* Account for the cost of create coin conditions and extract additions from the NPCResult.
* Rely on NPCResult for the maximum allowed cost.
* Stop looking for the relevant spend at the first occurrence (then process its created coins).
* Add a unit test for spending a coin in different ways then finding it spent in new peak.
This makes sure all mempool items that would spend that coin get removed.
* Parameterize the test for spending a coin in different ways then finding it spent in new peak, to account for both the optimized and the reorg code paths.
* Keep track of coin spends cost and additions in mempool items in case they won't make it into the block, so we don't rerun that again next time.
* Extend replace by fee tests with eligibility scenarios.
* Refactor find_duplicate_spends into get_deduplication_info and process_mempool_items into create_bundle_from_mempool_items.
Slightly refactor check_removals and create_bundle_from_mempool.
* Cast to uint64 in the return statement.
* Adapt to the recent addition of seq.
* Add unit tests for mempool items' bundle_coin_spends.
* Leverage the recently added make_test_conds when creating this npc_result.
* Pass in bundle_coin_spends and max_cost instead of InternalMempoolItem so that it remains internal.
* Add a test for the scenario where we pick a sub-optimal aggregation by deduplicating on solution A, seen in a relatively higher FPC item, then receiving better cost saving items on solution B after that one.
* Add some end-to-end tests for identical spend aggregation.
* Add a comment documenting why eligible_coin_spends map is ephemeral.
* First crack at VC puzzles
* Add p2_puzzle_or_hidden_puzzle
* First crack at drivers
* Basic test infra
* compile clsp and test covenant layer
* lint
* Add test for match_covenant_layer
* Add tests for DID TP
* Add tests for DID backdoor
* Add tests for p2_puzzle_or_hidden_puzzle
* Change DID backdoor -> p2_puzzle_w_auth + did_puzzle_authorizer
* Add a VerifiedCredential class
* Make p2_puz_or_hidden_puz viral
* Add morpher solution to covenant layer
* Add capability to update the proofs of a VC
* update_proofs -> do_spend
* Finish lifecycle of VC
* Make backdoor puzzle, remove p2_puzzle_w_auth from stack
* Extract p2_puzzle_w_auth to its own files
* lint
* rework launch process so it works after revocation
* Add some extra tests for the final state after revocation
* Add cost logging
* slight opimizations
* python comments and ergonomic improvements
* Comments for chialisp
* Add VC wallet and store infra
* Add a negative test for updating proofs w/o DID
* First crack at CR CAT
* Turn two hashes into one
* Extract some shatrees out of puzzle hash creation
* Add a stager function
* Precalculate initial_singleton_inner_puzhash_hash
* Add another stager
* Refine stagers
* Add cost logging and assert provider is authorized
* Add sketched out CRCAT class
* black
* Test most of CRCAT drivers
* lint
* Remove an unnecessary param
* Don't commit to singleton ID in parent morpher
* Extract DID to ownership layer
* lint
* use intitial puzzle hash for covenant as proof to morpher
* Collapse hash of parent morhper
* Remove stagers
* Rename some params
* Use SELF_HASH construct
* remove dependency on shatree
* Minor optimization of curried hashes
* Balance tree
* hint -> remark
* Make a v2 ownership layer for cost-sake
* (WIP) Migrate to tp hash at ownership layer
* (WIP) Return tp_hash from tp
* use a stager again
* use constants for ints
* use more constants
* inline a function, get rid of constant
* Switch to tree metadata on ownership layer
* Impelement wallet launch vc
* Minor fixes and rename a bunch of stuff
* lint
* Fix tests
* First crack at proof checking
* Remove unused imports
* Merge tests to get better coverageg
* Fix is_vc methods
* pre-commit
* pre-commit again
* Implement VC spend
* make sql pre-commit check work with git worktrees
* CRCATWallet
* Comment reasoning in script file
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* Add a concept of VCProofs
* Show proofs in vc_get_vc_list as well
* Implememnt CRCAT wallet
* Extend lineage store
* Make tests pass
* WalletIdentifier fix
* Add revocation RPC
* forgotten bytes32
* Return coin name with vc record
* First crack at adding VC authorizations to offers
* Add CROuterPuzzle
* Fix wallet_funcs.py
* VCWallet changes only
* Fix wallet db schema test
* Add RPC client methods and tests
* Allow a path for user to clear VC
* Mint VC command
* Get VCs command
* Forgotten hex compilation
* Automatically fetch DID when we have it
* Update Proofs command
* Add Proof Reveal command
* Get Proofs For Root command
* Revoke VC command
* Chialisp pre-commit cleanup
* Deprecate as_python()
* Add a limit to some sql queries
* Fix some bad imports
* Fix changed wallet sync function
* Check for tail condition amounts in CAT spending
* Add RPC docs
* only need one mojo for a singleton
* Fix comment
* remove create_puzhash from add_new_wallet
* Remove create_tandam_xch_tx
* fix wallet_state_manager hinting
* Rename RPCs
* Use streamable classes for RPC request parsing
* add back accidentally deleted method call
* Remove unnecessary copy() call
* Add negative test for using wrong DID to update proofs/revoke vc
* Add negative test for making sure invalid proofs can't be proven
* Add negative test for trying to piggyback on top an unrelated VC spend
* Make output_amount a uint64
* some chialisp bug fixes
* chialisp readability improvements
* Force same provider on provider update of DID
* typo in EML
* Add vc_wallet folders to package list
* mypy
* Increase test coverage a bit
* Add a bit more coverage
* rework wallet creation
* Delete is_addition_relevant
* Coveralls ignores
* remove brick reference
* coveralls-ignore --> pragma: no cover
* Couple more ignores
* typo corrections
---------
Co-authored-by: ytx1991 <t.yu@chia.net>
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* Add basic test for get_farmed_amount RPC
* remove unused line
* simplified test
* add a tx with a fee for get_farmed_amount test
* minor cleanup
* Apply suggestions from code review
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* Some updates from PR comments
---------
Co-authored-by: Kyle Altendorf <sda@fstab.net>
Use the one from the connection data like we do it with the port. The
result there is slightly different since its resolved so `localhost`
becomes an IP in `peer_info` but i anyway will open a PR soon to
transition from `str` to `IPAddress` in there.
* Make conflicting_items a list instead of a set as we're only iterating over it, we're not leveraging set ownership.
This simplifies using can_replace() and also prepares ground to simplify check_removals() next.
* Optimize check_removals by checking for mempool conflicts all at once.
* Add flexibility to `log_exceptions()`
Allows configuration of the exception or exceptions to be caught, whether or not to show the traceback, the log level, and also adds the exception type to aid readability of errors like `KeyError` where the only text is the value of the key requested.
* add `log_exceptions(message=)`
* add tests for `chia.util.log_exceptions()`
* Update test_log_exceptions.py
* Update test_log_exceptions.py
* back to parametrize
* more mypy platforms in ci
* check not checks
* oh yeah, activate...
* add 3.11
* adjust job names
* actually use the right python version for install
* adjust for 3.7 hinting support
* exclude macos 3.7 and macos arm 3.8
* back to 3.8 for now
* major_dot_minor
* oops
* move ignore comments for 3.7
* and more
Use "item" instead of "spend" in mempool's items functions names.
With this we're consistent with mempool's internal items map (_items) and _row_to_item as well as the return types.
* remove unused run_generator_mempool() function
* move run_generator_unsafe() into the tests, which is the only place it's used
* remove (somewhat meaningless) setup_generator_args() and create_generator_args()
* remove unused GENERATOR_MOD in mempool_check_conditions.py
* remove redundant get_generator() function
* transition analyze-chain.py to use run_block_generator() and drop dependency on GENERATOR_MOD
* fixup type hints in test_rom.py
* fixup type hints in test_compression.py
* fixup type hints in test_generator_types.py
limit the mempool size used for transactions with short expiration times. This makes it so transactions that expire in the next 15 minutes may not take more than one block worth of cost in the mempool
* Drop unused `List[SubEpochSummary]` return
* Drop unused `SubEpochSummary` return
* Raise in `WalletWeightProofHandler.validate_weight_proof`
* Move validation logs
* Set the new weight proof inside `fetch_and_update_weight_proof`
Which was `fetch_and_validate_the_weight_proof` before.
* Drop `WalletWeightProofHandler._weight_proof_tasks`, just await them
Doesn't seem to make a lot sense to have a list of tasks where we always
just add a task, await it and remove it after. This happens in the long
sync so to parent task will cancel the awaited task anyway.
Refactor function `is_trusted_peer` to a standalone easily testable
function. Leverage the existing `is_localhost` function to extend
coverage to IPv6 local address for the trusted peer check
Added tests for the new standalone function
* optimize at-capacity check in mempool.add_to_pool() by only computing the total cost once and remove all items in a single call
* use sqlite logic
* add test for Mempool.add_to_pool() when it's full
* Apply `final` decorator to `DataLayerWallet`
* Simplify `DataLayerWallet.create_new_dl_wallet`
* Simplify `DataLayerWallet.create`
* Improve `WalletStateManager.get_dl_wallet`
* More use of `WalletStateManager.get_dl_wallet`
* Only try `get_dl_wallet`
* prepare mempool spends_by_feerate test for more test cases, move it to test_mempool.py (since it's testing the Mempool class)
* use insertion order as tie-breaker in mempool
* wallet: Improve coin state retry wait logic and improve retry store test
* Avoid sleeping after uncatched exceptions
* Sleep only at the beginning since we have reduced timeout now in tests
* Fix the mempool fee per cost calculation.
It's currently performing integer division instead of float division, resulting in incorrect sorting of mempool items by fee rate. Items with x.y FPC get all treated as items of x.0 FPC.
* Add unit tests for mempool's spends_by_feerate.
* slightly faster nft complex offer tests
* cached bundle, faster offer matching
* nft metadata test fix
* more optimizations
* use get_nft_count method in tests
* use counts in bulk minting, avoid sleep
* flake
* offer cache fix
* forward compat param, max int16
* only parse programs with pairs
* cache offer data on method call, not init
* black
from https://github.com/Chia-Network/chia-blockchain/actions/runs/4455716184/usage i see macos with nominal low 40s runtime and outliers into the low 50s, windows has a wider variance with highs in the low 50s and 8'ish hung runs that timed out at the 75 minute timeout, linux looks like 30'ish with a high of maybe 36.
* enable soft-fork2
* add blockchain (consensus) test for time-lock conditions (non-ephemeral spend)
* introduce new soft-fork rule to compare ASSERT_SECONDS_* conditions against the previous transaction block's timestamp (to be consistent with ASSERT_HEIGHT_* conditions)
* bump chia_rs. This updates the mempool rules to disallow relative height- and time conditions on ephemeral coin spends
* implement assert_before in mempool_check_time_locks. Extend ephemeral coin test in blockchain with assert_before conditions
* implement support for assert_before conditions in compute_assert_height()
* support assert-before in mempool
* add timelock rule
* address review comments
* mempool min fee increase is a constant
* make can_replace() a free function rather than a member of mempool. It doesn't need to be a member, and a free function is easier to test
* simplify can_replace by passing in MempoolItem
* slightly simplify handling of conflicting mempool items in mempool_manager, to avoid double lookups
* simplify can_replace() by just passing removal_names instead of the whole dict
* add unit test for can_replace()
* `assert spend_bundle is not None`, required after hinting in next commit
* Don't start duplicated RPC servers, use the already started ones
* Wait for wallet sync before minting
* Move `request_time` creation into `is_peer_synced`
* Pass in `height: uint32` instead of `header_block: HeaderBlock`
* Disconnect untrusted peers as soon as we have a synced trusted
* Add `test_long_sync_untrusted_break`
* Ignore already disconnected peers
* better handling of offer status and failed txs
* 5 retries before tx is marked failed
* increase test timeouts
* fixed test
* require only difference vs whole fee
* polish + fee coins kick in only if we're missing fee
* Stop sending change to a new address
* Fix bug
* Add config & unit tests
* Resolve comments
* Refine based on the requirements
* Change config to str
* Disable addr reuse for unit test since the hardcode offer doesn't compatible with it
* Cover offer trade
* Add unit test and RPC support
* Add reuse option for cli
* Enhance unit test
* Fix unit test
* Add unit tests for create_bundle_from_mempool().
* Factor out some shared code into a function.
* Add a variation for sending transactions in reverse order.
* Split some logic into functions.
* Refactor setting up the mempool with coins as well as big cost spend bundle handling.
* Remove valid # type: ignore
A user ran into an exception on this line that mypy should have caught. Let's see what it says.
```python-traceback
2021-05-06T07:31:41.595 full_node full_node_server : ERROR Exception , exception Stack: Traceback (most recent call last):
File "chia\server\server.py", line 356, in start_client
File "aiohttp\client.py", line 763, in _ws_connect
File "aiohttp\client.py", line 521, in _request
File "aiohttp\connector.py", line 535, in connect
File "aiohttp\connector.py", line 892, in _create_connection
File "aiohttp\connector.py", line 1032, in _create_direct_connection
File "aiohttp\connector.py", line 969, in _wrap_create_connection
File "asyncio\base_events.py", line 949, in create_connection
File "asyncio\selector_events.py", line 473, in sock_connect
concurrent.futures._base.CancelledError
2021-05-06T07:31:45.016 daemon asyncio : ERROR Task exception was never retrieved
future: <Task finished coro=<kill_service() done, defined at chia\daemon\server.py:833> exception=AttributeError("'list' object has no attribute 'pid'")>
Traceback (most recent call last):
File "chia\daemon\server.py", line 841, in kill_service
File "chia\daemon\server.py", line 805, in kill_process
AttributeError: 'list' object has no attribute 'pid'
2021-05-06T07:32:09.965 full_node full_node_server : ERROR Exception: <class 'concurrent.futures._base.CancelledError'>, closing connection None. Traceback (most recent call last):
File "chia\server\server.py", line 531, in api_call
File "asyncio\tasks.py", line 435, in wait_for
concurrent.futures._base.CancelledError
2021-05-06T07:33:20.573 full_node full_node_server : ERROR Exception , exception Stack: Traceback (most recent call last):
File "chia\server\server.py", line 356, in start_client
File "aiohttp\client.py", line 763, in _ws_connect
File "aiohttp\client.py", line 521, in _request
File "aiohttp\connector.py", line 535, in connect
File "aiohttp\connector.py", line 892, in _create_connection
File "aiohttp\connector.py", line 1032, in _create_direct_connection
File "aiohttp\connector.py", line 969, in _wrap_create_connection
```
* fix plotter service killing
* just make it always be a list of processes for all services
* catch up tests
* Update chia/daemon/server.py
* clarify test with unknown condition opcode
* fix bug in mempool_manager test_ephemeral_timelock
* simplify test_blockchain test_ephemeral_timelock by factoring out the with_garbage parameter
* fix bug in make_test_conds() test utility
* bump chia_rs to 0.2.4, which preserves assert_seconds_relative 0 in parsing conditions. This allows for the 1.8.0 soft-fork to make the existing time-lock conditions stricter, > instead of >=. This is to match the existing ASSERT_HEIGHT_RELATIVE, which already is >
* fixup separating ENABLE_ASSERT_BEFORE from MEMPOOL_MODE
* Use a low value for SOFT_FORK2_HEIGHT during tests and cover the case before soft-fork2
---------
Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com>
* simplify daemon connection handling
* Add tests
* return list of service names during removal
* set remove raises KeyError
* Better test type hints
* Add types and code refactor
* fix test typos
* small type adjustments
* test code improvements
* Better error handling for bad json
* handle proper but unexpected JSON values
* Update chia/daemon/server.py
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* test improvements
* check for empty thing equivalence
* Improved logging for ConnectionReset
---------
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* bump chia_rs to version 0.2.3
* add new error codes for assert_my_birth_*
* add new condition codes for ASSERT_MY_BIRTH_*
* add logic for ASSERT_MY_BIRTH_* conditions
* implement Mempool using an in-memory sqlite database
* remove_from_pool with empty list is a no-op
* the order is not important in get_coin_records_by_puzzle_hash() or get_block_spends()
* use format string in log statement
* make MempoolItem not streamable, to improve performance of creating objects
* user shorter names for constants in test_mempool.py
* fix typo in test_ephemeral_timelock in mempool test. The conditions should only be applied to the ephemeral coin, not the coin creating it
* use shorter names in test-ephemeral-coin test parameters
* use shorter names in TestConditions to make the test easier to read
* user shorter names in test_blockchain test_ephemeral_timelock, to make the test easier to read
* extend test_ephemeral_timelock to cover soft-fork 2
* extend test_conditions to cover soft-fork 2
* full_node: Rename `FullNode.resond_block` to `FullNode.add_bock`
The naming `respond_block` is confusing here imo, we have a full node
API method called `respond_block` but here in the `FullNode` class this
method is there to try adding a block to the full nodes's chain so i
think `add_block` is the better choice?
Also change the `respond_block: full_node_protocol.RespondBlock`
parameter to `block: FullBlock` which lets us get rid of many
`full_node_protocol.RespondBlock` wrappings.
* `FullNode.receive_block_batch` -> `FullNode.add_block_batch`
* `receive_unfinished_block` -> `add_unfinished_block` + change parameter
* `FullNode.respond_transaction` -> `FullNode.add_transaction`
* `FullNode.respond_end_of_sub_slot` -> `FullNode.add_end_of_sub_slot`
* `FullNode.respond_compact_vdf` -> `FullNode.add_compact_vdf`
* `respond_compact_proof_of_time` -> `add_compact_proof_of_time`
* `respond_transaction_semaphore` -> `add_transaction_semaphore`
* Introduce BlockRecordProtocol as a subset of BlockRecord that the mempool manager uses for peak.
* Create BenchBlockRecord and use it for benchmarks/mempool.py.
* PR #14611 didn't land yet (keep 3.7 support for now).
* We don't need this guidance anymore.
* Make `WalletNode.set_sync_mode` an `asynccontextmanager`
* Only fetch the height if debug logs are enabled
* `yield` the start height of the sync context in `set_sync_mode`.
* fix test asserts to not require the same object, just the same value
* make Mempool's implementation private and give it a public interface
* fixup test that used to count fee *levels* but now count transactions
* make conditions_for_solution able to parse untrusted CLVM structures
* demote as_atom_list() to be a test-only function in test_rom.py, which is the only place it's used now
* Drop redundant fixtures in `test_dl_wallet.py`
* Drop `two_wallet_nodes_five_freeze`
Maybe im just not getting it but the `five_freeze` doesn't sound right
here to me and the fixture does the same as `two_wallet_nodes` so i
guess we can just drop it?
* Drop `wallet_and_node` fixture
* Drop both `wallet_node` fixtures
The one in `conftest.py` and the one in `test_dl_wallet.py`.
* Drop both `wallet_node` fixtures
The one in `conftest.py` and the one in `test_dl_wallet.py`.
* Drop `wallet_node_sim_and_wallet` fixture
* Drop `wallet_node_simulator` fixture
* Drop `one_wallet_node` fixture
* Check Wallet DB integrity
* Update command line help
* Improve duplicate DerivationPath index error message
* Move tests to chia/tests
* lint
* py init file
* Print WalletType names
* Don't complain about wallets not having derivation entries that don't need them
* Validate addresses used in order
* Update tests with new error output
* Fix check_addresses_used_contiguous in the case when the last address of the previous wallet was unused
* Augment mempool's total_mempool_fees to int as it shouldn't be limited by type Mojos.
* Add an assert to make sure the total mempool fees are correct.
