Commit Graph

911 Commits

Author SHA1 Message Date
Mariano Sorgente
817baa3096
Bind port 0 to fix race condition when grabbing available ports (#11578)
* port 0 to fix flakiness

* Try fixing setup_full_system

* Try fixing setup_full_system, and lint

* More attempts to fix

* No more calls to get random ports in setup_nodes

* Revert accidental changes

* Timelord extra arg

* Try with port 0

* Fix daemon test, and lint

* Try without 0.0.0.0

* Back to 0.0.0.0

* Try a few timelord changes to get test running

* Increase timeout again

* Use the correct interface to get the port

* INFO logging to debug issue

* Revert "INFO logging to debug issue"

This reverts commit 7c379e5cca.

* Fix advertised port log

* Add extra log

* Logging back

* Rollback the timelord changes

* Try port 0 timelord

* Revert "Try port 0 timelord"

This reverts commit 4997faf3b2.

* Try full green, change ordering

* Remove unused var

* speed up simulation and cleanup

* Now try without the port config

* Fix a flaky call to get_event_loop

* Try getting the port dynamically

* No dynamic port

* Try changing the ordering

* Try adding a sleep

* Back to what works

* Timelord before vdf clients

* Dynamic port for 1st timelord

* Revert "Dynamic port for 1st timelord"

This reverts commit 0f322a15b7.

* Revert "Timelord before vdf clients"

This reverts commit 3286c34696.

* Revert "Back to what works"

This reverts commit 30380dffb7.

* Revert "Try adding a sleep"

This reverts commit 9212b665a6.

* Revert "Try changing the ordering"

This reverts commit a62597d70d.

* Revert "No dynamic port"

This reverts commit 5d2e15749b.

* Revert "Try getting the port dynamically"

This reverts commit ef9cd75679.

* Revert "Fix a flaky call to get_event_loop"

This reverts commit 01a000fdfb.

* Try one to 0

* Just not 0

* Don't get port dynamically

* Cleanup a bit

* Fix

* Some cleanup work

* Some cleanup work

* Fix daemon test

* Cleanup

* Remove arguments
2022-05-23 08:13:49 -07:00
Kyle Altendorf
71a8ac2e69
Get get_args() and get_origin() from typing_extensions (#11571)
* Get get_args() and get_origin() from typing_extensions

* Update streamable.py

* cleanup
2022-05-19 10:59:21 -07:00
Kyle Altendorf
6082e929c9
Remove unneeded uint64() intermediate in CAT wallet (#11575) 2022-05-19 10:58:45 -07:00
Arvid Norberg
afae2ad5fe
make recurse_jsonify() work directly on types (#11537)
* make recurse_jsonify() work directly on types, circumventing the dataclasses.asdict() step. This enables simpler integration of non dataclasses into the Streamable and JSON protocols

* add benchmark for Streamable.to_json_dict()
2022-05-18 09:15:03 -07:00
Kyle Altendorf
cb5470b806
hints and strict type checking for test_wallet (#11541)
* hints and strict type checking for test_wallet

* fixup

* flip flop
2022-05-18 09:12:39 -07:00
Mariano Sorgente
31ed32628f
Optimize code to not perform useless subgroup checks (#11546)
* Optimize code to not perform useless subgroup checks

* Revert less important optimizations
2022-05-18 09:09:56 -07:00
Jeff
99275e6e37
Support searching derived addresses on testnet. (#11449)
* Support searching derived addresses on testnet.

* Added tests
2022-05-16 13:29:33 -07:00
Jeff
34e2dde8db
Keep daemon websocket alive during keyring unlock (#11371)
* When prompting to unlock the keyring during daemon launch, it's possible
that the daemon websocket connection will timeout before the user has
entered their passphrase. We can't quickly detect that the connection
has timed-out, so instead we now reconnect after collecting the
passphrase, unlock the keyring, and then close the old connection.

* mypy fix

* Switch to using asyncio.loop.run_in_executor to collect the passphrase on its own thread.

* Use a new ThreadPoolExecutor configured with 1 worker thread.
2022-05-16 13:29:10 -07:00
Arvid Norberg
51fc5f6e88
remove the cache from CoinStore. It appears the cost of maintaining the cache outweighs the gains. (#11540) 2022-05-16 13:28:42 -07:00
Jack Nelson
ecada58d36
Coin Selection Refactor With CAT Coin Selection Refactor (#9975)
* add exact match and best exact match algorithms

* optimize algorithm further

this might be good.

* lint

* fix bad logic

* add final algorithms

* delete lint

* oops

* Update coin_selection.py

* simplify and fix knapsack algoritm

* simplify code and correct logic

* make it way better.

* clarify comments and check for edge cases.

* add comments and stuff

* improve coin selection

addressed comments
Thanks!

* add coin_selection rpc tests.

* clean up and add new unit tests

* undo test changes

* add extra test cases

* move coin_selection to its own function and switch to it for cat and main wallet.

* add cat tests

* lint

* make function align with standards

also removed test

* make test better

* add proper types

* Improve code clarity

* wallet: fix coin selection bugs

* wallet: add an assert just in case

* tests: add some sleeps to reduce flakiness

* Isort

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* fix bad merge

* lint

* fix tests

* address aforementioned changes.

* remove wallet test

* isort

* more tests and fixes

* lint

* rename to amount for coin selection rpc

* fix incase we have no smaller coins

* fix tests + lint

* re add asserts

* oops

missed me.

* lint

* fix test

* Squashed commit of the following:

commit 34a2235de5
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Apr 13 10:09:42 2022 -0400

    clarify comment

commit adbf7f4f94
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Tue Apr 12 20:27:05 2022 -0400

    linty lint

commit 5ebc1ac9fd
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Tue Apr 12 20:17:19 2022 -0400

    add failure test and final changes

commit 7e5a21b4c2
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Tue Apr 12 19:35:18 2022 -0400

    add descriptions and slim down code

commit 31c95b916d
Merge: d7b91295b d9b0ef5f3
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Mon Apr 11 10:12:05 2022 -0400

    Merge branch 'jack-cat-coinselection' into jn_coinselection_dust

commit d7b91295b5
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Sun Apr 10 20:31:09 2022 -0400

    lint

commit 30dc7c0ab4
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Sun Apr 10 20:25:52 2022 -0400

    fix tests

commit 6c8c2e4874
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Thu Mar 31 15:06:00 2022 -0400

    remove duplicate code.

commit 9f79b6f304
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Thu Mar 31 15:01:10 2022 -0400

    address more concerns

commit 67c1b3929f
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Thu Mar 31 12:59:05 2022 -0400

    fix logic error

commit 2d19a53245
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Thu Mar 31 11:47:52 2022 -0400

    simplify and de duplicate code

commit 6ab1cc79bb
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 21:34:50 2022 -0400

    add function and select individual coin

commit 582c17aa8d
Merge: ce2165942 618fbaeba
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 21:14:37 2022 -0400

    Merge branch 'jack-cat-coinselection' into jn_coinselection_dust

commit ce21659429
Merge: 16aabb3fd 6daba28db
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 20:53:21 2022 -0400

    Merge branch 'jack-cat-coinselection' into jn_coinselection_dust

commit 16aabb3fd5
Merge: 0b9fc2845 2286fe426
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 20:49:02 2022 -0400

    Merge branch 'jack-cat-coinselection' into jn_coinselection_dust

commit 0b9fc28455
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 20:38:12 2022 -0400

    lint

commit 62e74c72f4
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 20:34:22 2022 -0400

    fix logic and tests

commit e738f44320
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 18:52:05 2022 -0400

    deal with dust and add tests

* make sure that we do not use any dust

* minor change

* address concerns

* adjust comments

* adjust comment

Co-authored-by: Mariano Sorgente <sorgente711@gmail.com>
Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-05-16 10:42:46 -07:00
Mariano Sorgente
56e6e68eb3
Uses the new from_bytes_unchecked method in blspy, to improve perfo… (#11463)
* Uses the new `from_bytes_unchecked` method in blspy, to improve performance

* Update test

* Fix merge conflict

* Use from_bytes_unchecked in post_init and from_json

* More uniform code
2022-05-16 09:49:50 -07:00
Arvid Norberg
e74ff02b09
reduce the redundant computations of coin_ids in block_body_validation (#11530) 2022-05-15 15:46:44 -07:00
Kyle Altendorf
0e419822e0
Allow services to set a non-default max request body size limit (#11516) 2022-05-14 02:08:10 -07:00
Kyle Altendorf
7c91f470f4
simplify SizedBytes and StructStream (#11429)
* simplify SizedBytes and StructStream

* lint

* super()  !!!

* do not pass parameter up to super().__init__()

* Update chia/util/struct_stream.py

Co-authored-by: Arvid Norberg <arvid@libtorrent.org>

* parse fixed-width int data from class name

* add int512 and uint128 .from_bytes(), test .parse() failures

* test serialization against struct.pack()

* use typing_extensions for final

* override ignore

* stop using struct for StructStream

oh the irony

* fixup .to_bytes() to accept parameters again for where we use that

* bring back signed parameter

* format

* adjust tests for new exception

* eliminate custom coding for uint128 and int512

* tidy

* remove unused StructStream.PACK attribute

* add direct tests for parse_metadata_from_name()

* stricter hinting

* remove no-longer-needed typeshed work-around

* apply strict type checking to all touched files

* remove StructStream override of .to_bytes()

* tidy

* types touchup

* add unused parameter comments

Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
Co-authored-by: wjblanke <wjb98672@gmail.com>
2022-05-14 02:05:27 -07:00
Kyle Altendorf
d1e4a38462
less except -> false (#10219)
* less except -> false

* establish_connection() and .perform_handshake() can just return None

* remove more pointless, and no wrong, assertions
2022-05-14 02:03:56 -07:00
Adam Kelly
298ea9ea20
Resend transactions (#11167)
* Resend transactions

* Don't recheck transactions more frequently than timeout

* Add wallet resend parameter to config, move timeout code out of tx store, but close to call site

* Add a test for wallet transaction resend

* Add test for wallet retry

* isort new files for precommit and update workflows

* Use correct fixture name

* LGTM - remove unused import

Co-authored-by: Earle Lowe <e.lowe@chia.net>
2022-05-14 02:03:10 -07:00
Mariano Sorgente
d1e445fac2
Ms.mypy consensus (#11403)
* Mypy for consensus dir (Except blockchain.py)

* Finish up consensus folder mypy

* Lint

* isort

* lint

* awaitable over coroutine

* less Any

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-05-12 15:21:47 -07:00
Arvid Norberg
3795a52152
fix some LGTM issues (#11453) 2022-05-12 15:17:24 -07:00
dustinface
fc2d6d3db0
harvester: Fix deadlock on disconnect after a repeated plot sync start (#11481)
* Test plot sync start and disconnect while a sync is in progress

* Drop `Receiver._lock`, instead wait for `Receiver.sync_active` to clear

* Properly stop the sender and plot manager refreshing on disconnect

* Poll faster, drop the log

* Reset `_sync_id` at the very end of `_finalize_sync`
2022-05-12 15:16:22 -07:00
Mariano Sorgente
ce92f36055
Fix Newer block issue (#11486)
* Fix Newer block issue

* Fallback to checking memory
2022-05-12 15:15:36 -07:00
Arvid Norberg
a4ec7e7cb8
Replace MerkleSet with the rust implementation of computing the merkle root. This saves about 20% of the CPU work in the main thread when validating blocks. (#11432) 2022-05-10 04:42:45 -07:00
Kyle Altendorf
6788c0a5ef
Use context manager for locking clvm files (#11467) 2022-05-10 04:41:57 -07:00
dustinface
541b99e311
harvester: Introduce recursive_plot_scan (#11468)
* harvester: Introduce `recursive_plot_scan`

* Change log and merge boths lookups

* Drop redundant assignments
2022-05-09 16:43:53 -07:00
dustinface
d4c2f0c14a
tests: Improve test_wallet_rpc.py (#11148)
* Move wallet RPC test setup into a fixture

* Fix `mypy` issues

* Enable  `isort`

* Move `tx_in_mempool` out and reuse it instead of `tx_in_mempool_2`

* Introduce `get_confirmed_balance` and `get_unconfirmed_balance`

* Use `pytest.raises`

* Only test with an untrusted full node

* Introduce `generate_funds`

* Drop sleeps and create less blocks

* Recover trusted/untrusted sync parameterization
2022-05-05 08:22:02 -07:00
Mariano Sorgente
e0df18f85d
Ms.hint refactor (#11222)
* Some work on hints

* More cleanup and test_generator_tools

* Change relevant calls

* More refactor and cleanup

* More refactor

* Small fix

* Create wrapper object PeakPostProcessingResult

* Lint and small fix

* Fix a hint bug

* Fix hint update_wallets

* Fix test_full_sync

* pre-commit

* Start with PR review comments

* More efficient iteration

* Remove tx_removals_additions_and_hints

* Revert mozilla ca

* Fix issue with startup
2022-05-05 08:19:57 -07:00
Zachary Brown
a1db012b7a
Add libgmp-dev to Bladebit installation (#11406) 2022-05-05 08:17:14 -07:00
dustinface
629c303077
cmds: Use the new plot_count of get_pool_state in plotnft cmds (#11414) 2022-05-05 08:16:10 -07:00
Arvid Norberg
1b2d119cc3
optimize hash_coin_list() by sorting pure coin_ids, rather than Coins that need to compute its name multiple times during sorting, and also by sorting the actual binary coin names, not hexadecimal strings (#11421) 2022-05-04 07:24:55 -07:00
dustinface
0d5251c3a7
farmer|gui: Enable paginated plot loading and improved state reporting (#11367)
* farmer: Adjust notifications from the farmer to the UI

- Only send the data for the harvester which actually sent an update
- Notify for each loaded batch during initial loading

* Enable improved farmer/harvester GUI
2022-05-03 15:33:38 -07:00
wjblanke
0156d3471b
readd ping_task as keepalive for daemon (#11356)
* readd ping_task as keepalive for daemon

* raised and lowered logging levels

* flake
2022-05-03 10:36:43 -07:00
dustinface
d00d045d9c
farmer|rpc|tests: Implement paginated harvester plot endpoints (#11365)
* farmer|rpc|tests: Implement paginated harvester plot endpoints

* Simplify filtering

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Let the API handle the exceptions

* Simplify the other filtering too

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Simplify count assertions

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Refactor `is_filter_match` to `plot_matches_filter`

And just convert to `Plot` in tests.

* Move `chia.util.misc.KeyValue` to `chia.rpc.farmer_rpc_api.FilterItem`

* Rename `peer_id` to `node_id` to be match `get_harvesters_{summary}`

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-05-03 09:17:05 -07:00
Arvid Norberg
340e26e154
simplify merkle set (#11416)
* _make_middle doesn't need to be a member function

* add some unit tests
2022-05-03 07:49:29 -07:00
Zachary Brown
3767fed513
update plotters installation to include an apt update before the apt install (#11400) 2022-05-02 15:56:49 -07:00
wjblanke
8905dc69ac
make sure we set the syncto height correctly when we roll back instea… (#11389)
* make sure we set the syncto height correctly when we roll back instead of noop

* don't error out if height is -1 on rollback
2022-05-02 15:55:45 -07:00
Mariano Sorgente
6fe7116730
Fix race condition, blockchain can change before acquiring the lock (#11324) 2022-05-02 13:02:38 -07:00
dustinface
e9e536925c
farmer|rpc: Add plot_count to get_pool_state RPC endpoint (#11364)
* farmer|rpc: Add `plot_count` to `get_pool_state` RPC endpoint

* Test `plot_count` of `get_pool_state` farmer RPC endpoint
2022-05-02 13:02:00 -07:00
dustinface
984c442b5e
util: Implement Paginator class as interface to access a list by pages (#11247)
* util: Implement `Paginator` class as interface to access a list by pages

* Be less restrictive about page sizes and refactor tests

* Make the pages based of 0 instead of 1 and some more test refactoring

* More tests

* Adjust workflows after rebase

* Introduce `Paginator.create`

* `<=` instead of `- 1`
2022-04-29 18:35:47 -07:00
dustinface
387abc3d8e
farmer|rpc: Some changes to get_harvesters{_summary} RPC endpoints (#11342)
* farmer|rpc: Always add `last_sync_time` in `Receiver.to_dict`

This changes the resonse of `get_harvesters` and
`get_harvesters_summary` to always include that field with value
`None`/`null` if the receiver is not yet synced.

* plot_sync: Add sync state to dict in `Receiver.to_dict`

* Add `total_plot_size` to `Receiver.to_dict`

* Refactor sync object assertions

* Access `_current_sync` directly

* Generate `syncing` data outside
2022-04-29 18:34:14 -07:00
Arvid Norberg
d3f2ae367d
remove unused functions on MerkleSet (#11362) 2022-04-29 08:17:37 -07:00
dustinface
270c69eb5d
plot_sync: Introduce receiver.Sync (#11267)
* plot_sync: Introduce `receiver.Sync`

* Use `dataclasses.replace`

Co-authored-by: Kyle Altendorf <sda@fstab.net>

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-04-29 08:16:39 -07:00
dustinface
b693aeb407
farmer|rpc: Introduce get_harvesters_summary RPC endpoint (#11245) 2022-04-28 19:36:21 -07:00
Arvid Norberg
3b4cbd42ce
lookup coins in large batches (#11328)
* lookup coins in large batches, rather than one at a time, in block_body_validation

* make get_coin_records use the cache. and make the blockchain change tracking also use the new batch lookup

* make building the sql query a bit more elegant

* fix assert

* fix test_blockchain assert tests

* add type annotation

* await aiosqlite.Cursor.execute() immediately
2022-04-28 19:34:43 -07:00
dustinface
6808d92243
cmds: Adjust stop daemon output (#11343)
* cmds: Adjust stop daemon output

Stopping the daemon without this patch prints:

```
daemon: {'ack': True, 'command': 'exit', 'data': {'success': True}, 
'destination': 'client', 'origin': 'daemon', 'request_id': 'some 
request_id hash here'}
```

Stopping the daemon with this patch prints: 

```
Daemon stopped
```

and if it fails:

```
Stop daemon failed {'ack': True, 'command': 'exit', 'data': {'success': 
True}, 
'destination': 'client', 'origin': 'daemon', 'request_id': 'some 
request_id hash here'}
```

* Simplify check

Co-authored-by: Kyle Altendorf <sda@fstab.net>

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-04-28 19:33:53 -07:00
Chris Marslender
acfdd75a20
Updated warning about CHIA_ROOT being set when running init (#11346)
* Remove warning about CHIA_ROOT being set when running init

* Update to still output the CHIA_ROOT if set, but not instruct the user to remove it

* Store chia_root in a var
2022-04-28 09:16:53 -07:00
Chris Marslender
f1b7eb0d2d
Forward "proof" event to metrics (#11345) 2022-04-28 07:31:22 -07:00
Kyle Altendorf
58e231e929
add daemon entry point (#11329)
* add daemon entry point

* remove argv parameter from chia.daemon.server.main()
2022-04-27 14:28:13 -07:00
Mariano Sorgente
23158c656e
Fix issue with missing coins (#11338)
* raise error when request fails

* Gather when cancelling sync

* Increase timeout

* Increase timeout even more
2022-04-27 14:07:41 -07:00
Mariano Sorgente
7630d220f5
Optimize BLS verification when public key is repeated (#11318)
* Optimize BLS verification when public key is repeated

* Fix accidental commit

* Lint

* Fix isort

* Address comments
2022-04-27 11:38:48 -07:00
Jack Nelson
bfb488b1b9
undo 9130 (#10278) 2022-04-25 16:05:35 -07:00
dustinface
c0f3d4d231
daemon|util: Don't remove from list while iterating (#11208)
* daemon: Iterate over a copy of `websockets`

Currently when it fails to send the response to one websocket it skips 
the next websocket in list becaue we remove from the list while 
iterating.

* util: Iterate over a copy of `peers_with_peak`

Currently when a peer is closed it skips the check for the next peer in 
the list becaue we remove from the list while iterating.
2022-04-22 10:55:57 -07:00