Commit Graph

229 Commits

Author SHA1 Message Date
dustinface
2ff987573d
Simplify ChiaServer.close_all_connections (#13916) 2022-11-16 13:57:30 -06:00
dustinface
5c861db42f
Make WSChiaConnection a dataclass (#13906)
Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-11-16 13:38:28 -06:00
dustinface
643b29e636
Drop broken message types logging (#13914)
Seems like the idea was to log an overview of message types for the 
currently active api calls whenever we have multiples of 100 running. 
This currently isn't working at all because `len(message_types) % 100 == 
0` will only trigger if there are 100 different message types in the 
counter (we don't even have that much different messages). A fix would 
be to check `message_types.total() % 100 == 0`. But then, if we hover 
around 100 active messages this will repeatedly trigger logs since we 
decrement for each processed message, im not sure this all makes much 
sense and we anyway log all received message types with `DEBUG` logs so 
i decided to just drop this. If someone thinks we should keep it with 
the fix or has ideas to make this a more reasonable log instead of 
dropping it, let me know :)
2022-11-16 10:44:48 -06:00
Almog De Paz
ac2203eda8
Refactor test utils for import availability outside of repo (#13880)
* refactor for import availability outside of repo

* refactor for import availability outside of repo

* small revert

* newline

* fix imports

* fix pre-commit issues

* types issort, adapt mypy ignore

* redundant imports

* move files from util to simulator

* run pre_commit

* remove setup_nodes/keyring from isort.cfg

* fix asyncio import
2022-11-15 18:51:05 -06:00
dustinface
8e2e51a8c8
Add optional outbound parameter to get_connections (#13879) 2022-11-15 17:40:16 -06:00
Earle Lowe
f30f0b3512
Call close callback for already closed connections (#13727)
* always call close callback

* Some further explorations

* some more explorations

* Add delay while connections are closed in test

* Add a delay to make sure connections drain

* Update chia/server/server.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Update tests/connection_utils.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* use log_exceptions context manager

* set close flag correctly

* Added callback type and other changes

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-11-15 16:23:06 -06:00
dustinface
2706f5995b
server: Enable and fix mypy in ws_connection.py (#13878)
* server: Enable and fix `mypy` in `ws_connection.py`

* Apply suggestions from code review

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Tweak error message

* Tweak formatting

* Make `WSChiaConnection.close_callback` optional

* Tweak assert message

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Don't provide a default for `close_callback`

* Adjust assertion

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-11-12 14:18:11 -06:00
Kyle Altendorf
5cf8a291f7
adjust approach to optionality around ApiMetadata (#13888) 2022-11-10 19:58:12 -06:00
dustinface
0befdec071
server: Introduce WSChiaConnection._get_extra_info (#13876) 2022-11-09 17:41:50 -06:00
dustinface
50b630aee9
server: Don't pass the queue into WSChiaConnection._wait_and_retry (#13877) 2022-11-09 14:57:59 -06:00
dustinface
8394857ca2
server: Return earlier in WSChiaConnection.__getattr__ (#13875)
* server: Return earlier in `WSChiaConnection.__getattr__`

* Tweak message names
2022-11-09 14:19:43 -06:00
dustinface
24961dd02c
server: Drop some unused code (#13867)
* Drop unused `Delivery` enum

* Drop unused `get_outgoing_connections`

* Drop `connection_utils.py`
2022-11-08 18:04:56 -06:00
dustinface
50d148eadd
Cleanup WSChiaConnection imports/usage (#13868) 2022-11-08 10:10:59 -06:00
hugepants
1220fcfd69
Revert rate limiting messages from warning back to debug (#13758)
In 1.5.1 this was changed from `debug` to `warning` causing a lot of excess noise in default logging as reported in https://github.com/Chia-Network/chia-blockchain/issues/13132
2022-11-05 17:48:07 -05:00
Kyle Altendorf
cb0fbe3208
switch to context manager for task timing (#13473) 2022-11-05 17:46:40 -05:00
dustinface
cf77f58bf2
server: Make ChiaServer a dataclass (#13574)
* server: Make `ChiaServer` a `dataclass`

* Use existing logger

Co-authored-by: Kyle Altendorf <sda@fstab.net>

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-11-03 11:28:23 -05:00
dustinface
7a5d9579ef
server: More use of ChiaServer.get_connections (#13573)
* Drop `get_full_node_connections` -> use `get_connections`

* Drop `connection_by_type`

I think the housekeeping required to have it isn't worth it since we
don't handle huge numbers of connections and `get_connections` should be
less enough overhead. Im open for complains though.
2022-10-05 11:21:38 -05:00
Kyle Altendorf
305875d0d4
use Logger.warning() instead of the deprecated Logger.warn() (#13588)
https://github.com/Chia-Network/chia-blockchain/actions/runs/3173913911/jobs/5170075702
```python-traceback
Traceback (most recent call last):
  File "/Users/runner/work/chia-blockchain/chia-blockchain/chia/server/node_discovery.py", line 299, in _connect_to_peers
    await self._query_dns(dns_address)
  File "/Users/runner/work/chia-blockchain/chia-blockchain/chia/server/node_discovery.py", line 237, in _query_dns
    self.log.warn(f"querying DNS introducer failed: {e}")
  File "/Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/logging/__init__.py", line 1492, in warn
    warnings.warn("The 'warn' method is deprecated, "
DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
```
2022-10-03 14:45:18 -05:00
dustinface
31f31296a7
server: Use WebServer in ChiaServer (#13545)
* server: Use `WebServer`

* Fix `yaml.representer.RepresenterError: ('cannot represent an object', 49405)`
2022-10-03 14:43:04 -05:00
dustinface
35acbabf80
server: Use get_extra_info to get the ssl_object (#13578) 2022-09-30 17:50:46 -05:00
Kyle Altendorf
2fc65e1178
more explicit and complete handling of api decorator data (#13542)
* more explicit and complete handling of api decorator data

* fix

* .message_class

* actually, those are different types...

* tweak

* simplify

* learn that functools.wraps copies random attributes
2022-09-30 17:47:56 -05:00
dustinface
c3ebc62246
server: Drop some ChiaServer attributes (#13513)
* Drop unused `ChiaServer.tasks`

* Drop unused `ChiaServer._tasks`
2022-09-30 17:41:51 -05:00
Kyle Altendorf
3b084a165b
configure isort to add the future annotations import (#13327)
* configure isort to add the future annotations import

* apply the new isort setting

* remove type ignores for new mypy (#13539)

https://pypi.org/project/mypy/0.981/

* another
2022-09-30 03:40:22 -05:00
dustinface
345698aebd
server: Create ssl contexts earlier + store them instead of all paths (#13512) 2022-09-29 12:14:20 -05:00
Kyle Altendorf
bbd032e1c9
embrace click 8 (#13559)
* click==8.1.3, black==22.8.0

* black

* click option creator hinting

* ignores for to-be-resolved issue

* remove unused import
2022-09-29 11:24:36 -05:00
Gene Hoffman
1224ef924f
Use codespell to catch some typos (#13557) 2022-09-28 04:27:34 -05:00
Kyle Altendorf
81837a9e1c
Refactor away from post init for UPnP (#11837)
* Refactor away from post init for UPnP

* refactor UPnP to not be reusable

* update upnp in datalayer too

* add if_needed parameter to UPnP.shutdown()

* simpler

* be more lenient about startup failures, like before

* too many ways for UPnP to fail right now, revert to a smaller refactor approach instead of fixing them all up
2022-09-26 11:19:33 -05:00
Kyle Altendorf
23ff56579a
parse api method hints earlier, and less, using get_type_hints() (#13509)
* parse api method hints earlier, and less, using get_type_hints()

* exploring (last)

* cleanup

* tidy comment

* be descriptive
2022-09-26 11:17:44 -05:00
Kyle Altendorf
3342a2a6a2
less _ for used variables (#13536) 2022-09-26 11:16:57 -05:00
dustinface
d78c122505
Drop redundant RPC server instances in DataLayer related RPC tests (#13518) 2022-09-26 11:16:25 -05:00
dustinface
77932fcc11
util: Refactor select_port to not load the config (#13514)
* util: Refactor `select_port` to not load the config

* Fix `prefer_ipv6` config lookup

* Use `str2bool`
2022-09-23 17:47:26 -05:00
Kyle Altendorf
1dbf575a3e
RpcApiProtocol and RpcServiceProtocol (#11522)
* provide a skeleton hint protocol for rpc apis

* more

* more self.custom_get_connections = None

* fix RpcApiProtocol.service hinting

* catch up with RpcApiProtocol._state_changed()

* FullNode.server is now non-optional.  'ish

* drop another todo

* custom_get_connections -> get_connections

* just make get_connections() a mandatory method with a helper provided

* lots

* try again

* ._start_with_fingerprint()

* fill out the Service generic parameter in start scripts

* ... instead of pass

* add docstrings to the protocols

* remove no longer needed hasattr/not None checks

* remove outdated comment

* refactor Harvester._is_shutdown to ._shut_down
2022-09-21 22:49:01 -05:00
Kyle Altendorf
b9e6fd577e
Remove kwargs from Service.start() (#12294) 2022-09-16 16:22:53 -05:00
dustinface
70fe981fb3
cmds: Implement chia beta (#12389)
* cmds: Implement `chia beta` for the beta test program

* Unhide and document all `beta` subcommands

* Refactor all subcommands

* Introduce `chia beta configure`

* Introduce `chia beta status`

* Test all `chia beta` commands

* Use a separate file logger for beta logs

* Write the plotting call args to the log file

* Sort potential submissions

* Some refactoring around log file log handler creation

* JSON dump the plotting args
2022-09-12 16:57:31 -05:00
Mariano Sorgente
87a3e93cc4
Ms.mypy enable (#13320)
* Remove mempool.additions

* Don't re run the program, and remove program from mempool item

* Removals only stores item ids, and stores a list

* Move pending cache down to prevent cache dos

* Separate validation from adding to pool, and remove mypy exceptions

* Fix bug with replacing

* Add to mypy

* Revert cbgui

* precommit fail

* Properly update the seen dict

* lint error

* Fix mempool bug

* Enable mypy for a bunch of files

* Update after merge with main

* 99/260 remaining

* Address comments

* Enable more mypy

* Merge conflict mypy file

* More fixes

* pytest.ini and cb-gui

* One more flake8 fix

* Flake8 and tests

* More test fixes

* isort

* Make LGTM happy

* Update chia/full_node/block_store.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Update chia/full_node/full_node_store.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Update chia/wallet/util/peer_request_cache.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Apply all suggestions from PR comments

* Revert cb-gui

* 100 back to 1000, minor tweaks

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-09-08 12:57:15 -05:00
Amine Khaldi
a615dee862
Merge commit '27df8e28e3c2082f8959607c07188e56395c6715' into checkpoint/main_from_release_1.6.0_27df8e28e3c2082f8959607c07188e56395c6715 2022-09-01 21:25:38 +01:00
Amine Khaldi
08d9e9b768
Merge commit '95ca86cba85bc58ff45ad5c7a87b008bc3e48af0' into checkpoint/main_from_release_1.6.0_95ca86cba85bc58ff45ad5c7a87b008bc3e48af0 2022-09-01 16:06:45 +01:00
Kyle Altendorf
27df8e28e3
fill in data layer certs at startup (#13225)
* fill in data layer certs at startup

* tidy

* don't overwrite

* add overwrite parameter to write_ssl_cert_and_key()
2022-08-30 02:57:36 -05:00
Kyle Altendorf
95ca86cba8
Fill missing services in config for data layer (#13220)
* fill missing services in config for data layer

* adjust

* here too
2022-08-29 15:59:18 -05:00
Amine Khaldi
eeb2277955
Merge commit '12fcaf0e982590efcf26a954c67cd760035d9e6e' into checkpoint/main_from_release_1.6.0_12fcaf0e982590efcf26a954c67cd760035d9e6e 2022-08-29 17:45:49 +01:00
Arvid Norberg
49ad81b5a0
instrumentation of asyncio tasks (#13046)
* Add support for instrumentation of asyncio tasks, to measure wall-clock time (as opposed to CPU time)

* fine tune output
2022-08-22 22:20:55 -05:00
Amine Khaldi
2836dd9574
Merge commit 'e5297f411d716b384e0514fd57aeb8536eda0527' into checkpoint/long_lived_atari_from_main_e5297f411d716b384e0514fd57aeb8536eda0527 2022-08-19 19:51:55 +01:00
Kyle Altendorf
bd7ed405eb
No traceback for OSError 113 in WSChiaConnection.outbound_handler() (#13016)
* No traceback for OSError 113 in WSChiaConnection.outbound_handler()

```python-traceback
Aug 16 21:24:08 fullnode chia_full_node[58416]: 2022-08-16T21:24:08.489 full_node full_node_server        : ERROR    Exception: [Errno 113] No route to host with 121.236.62.254
Aug 16 21:24:08 fullnode chia_full_node[58416]: 2022-08-16T21:24:08.490 full_node full_node_server        : ERROR    Exception Stack: Traceback (most recent call last):
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/farm/chia-blockchain/chia/server/ws_connection.py", line 262, in outbound_handler
Aug 16 21:24:08 fullnode chia_full_node[58416]:     await self._send_message(msg)
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/farm/chia-blockchain/chia/server/ws_connection.py", line 416, in _send_message
Aug 16 21:24:08 fullnode chia_full_node[58416]:     await self.ws.send_bytes(encoded)
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/farm/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/web_ws.py", line 315, in send_bytes
Aug 16 21:24:08 fullnode chia_full_node[58416]:     await self._writer.send(data, binary=True, compress=compress)
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/farm/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 688, in send
Aug 16 21:24:08 fullnode chia_full_node[58416]:     await self._send_frame(message, WSMsgType.BINARY, compress)
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/farm/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 659, in _send_frame
Aug 16 21:24:08 fullnode chia_full_node[58416]:     await self.protocol._drain_helper()
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/farm/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/base_protocol.py", line 87, in _drain_helper
Aug 16 21:24:08 fullnode chia_full_node[58416]:     await asyncio.shield(waiter)
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/usr/lib/python3.8/asyncio/selector_events.py", line 848, in _read_ready__data_received
Aug 16 21:24:08 fullnode chia_full_node[58416]:     data = self._sock.recv(self.max_size)
Aug 16 21:24:08 fullnode chia_full_node[58416]: OSError: [Errno 113] No route to host
Aug 16 21:24:08 fullnode chia_full_node[58416]:
```

* Update ws_connection.py

* refactor
2022-08-17 01:03:10 -05:00
Kyle Altendorf
5f1e483511
make web socket timeout a warning with a traceback (#12824) 2022-08-16 13:45:40 -05:00
Amine Khaldi
34d6197890
Merge commit '0bc54ae08b6655ec5242a8b21460d34b5e4a38c6' into checkpoint/long_lived_atari_from_main_0bc54ae08b6655ec5242a8b21460d34b5e4a38c6 2022-08-15 13:55:39 +01:00
Amine Khaldi
afd2725997
Merge commit 'f0a0f071169ef5b892261d07bf15ca701ca6e5b0' into checkpoint/long_lived_atari_from_main_f0a0f071169ef5b892261d07bf15ca701ca6e5b0 2022-08-11 13:12:19 +01:00
Amine Khaldi
87d89693b1
Merge commit '11b210809bd3c7e3e0ec199df5548f49abd6c377' into checkpoint/long_lived_atari_from_main_11b210809bd3c7e3e0ec199df5548f49abd6c377 2022-08-11 13:00:29 +01:00
Kyle Altendorf
bd5a7e4163
just raise, not raise e when reraising (#12825) 2022-08-10 15:39:51 -05:00
dustinface
7c9fb654a3
server|tests: Delay the harvester -> farmer connection in some tests (#12862)
* Introduce `Service.add_peer`

* Delay the harvester/farmer connection in some tests to avoid flakiness
2022-08-10 04:57:34 -05:00
Amine Khaldi
5b8451303a
Reflect the change on datalayer related code. 2022-08-08 19:13:07 +01:00