chia-blockchain/chia/seeder/start_crawler.py

90 lines
2.9 KiB
Python
Raw Normal View History

from __future__ import annotations
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
import logging
import pathlib
import sys
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
from multiprocessing import freeze_support
from typing import Any, Dict, Optional
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
from chia.consensus.constants import ConsensusConstants
from chia.consensus.default_constants import DEFAULT_CONSTANTS
from chia.rpc.crawler_rpc_api import CrawlerRpcApi
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
from chia.seeder.crawler import Crawler
from chia.seeder.crawler_api import CrawlerAPI
from chia.server.outbound_message import NodeType
from chia.server.start_service import RpcInfo, Service, async_run
from chia.types.aliases import CrawlerService
from chia.util.chia_logging import initialize_service_logging
from chia.util.config import load_config, load_config_cli
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
from chia.util.default_root import DEFAULT_ROOT_PATH
from chia.util.misc import SignalHandlers
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
# See: https://bugs.python.org/issue29288
"".encode("idna")
SERVICE_NAME = "seeder"
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
log = logging.getLogger(__name__)
def create_full_node_crawler_service(
root_path: pathlib.Path,
config: Dict[str, Any],
consensus_constants: ConsensusConstants,
connect_to_daemon: bool = True,
) -> CrawlerService:
service_config = config[SERVICE_NAME]
crawler_config = service_config["crawler"]
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
crawler = Crawler(
service_config,
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
root_path=root_path,
constants=consensus_constants,
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
)
api = CrawlerAPI(crawler)
network_id = service_config["selected_network"]
rpc_info: Optional[RpcInfo[CrawlerRpcApi]] = None
if crawler_config.get("start_rpc_server", True):
rpc_info = (CrawlerRpcApi, crawler_config.get("rpc_port", 8561))
return Service(
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
root_path=root_path,
config=config,
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
node=api.crawler,
peer_api=api,
node_type=NodeType.FULL_NODE,
advertised_port=service_config["port"],
2022-07-13 23:02:08 +03:00
service_name="full_node",
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
upnp_ports=[],
on_connect_callback=crawler.on_connect,
network_id=network_id,
rpc_info=rpc_info,
connect_to_daemon=connect_to_daemon,
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
)
async def async_main() -> int:
# TODO: refactor to avoid the double load
config = load_config(DEFAULT_ROOT_PATH, "config.yaml")
service_config = load_config_cli(DEFAULT_ROOT_PATH, "config.yaml", SERVICE_NAME)
config[SERVICE_NAME] = service_config
overrides = service_config["network_overrides"]["constants"][service_config["selected_network"]]
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
updated_constants = DEFAULT_CONSTANTS.replace_str_to_bytes(**overrides)
initialize_service_logging(service_name=SERVICE_NAME, config=config)
service = create_full_node_crawler_service(DEFAULT_ROOT_PATH, config, updated_constants)
async with SignalHandlers.manage() as signal_handlers:
await service.setup_process_global_state(signal_handlers=signal_handlers)
await service.run()
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
return 0
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
def main() -> int:
Chia Seeder (#8991) * initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <straya@chia.net> Co-authored-by: Florin Chirica <fchirica96@gmail.com> Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
2021-11-28 05:30:25 +03:00
freeze_support()
return async_run(async_main())
if __name__ == "__main__":
sys.exit(main())