chia-blockchain/build_scripts/pyinstaller.spec

225 lines
5.7 KiB
RPMSpec
Raw Normal View History

# -*- mode: python ; coding: utf-8 -*-
import importlib
import os
import pathlib
import platform
import sysconfig
from PyInstaller.utils.hooks import collect_submodules, copy_metadata
THIS_IS_WINDOWS = platform.system().lower().startswith("win")
Add plotters. (#8497) * Initial commit add plotters. * Lint. * Add progress for bladebit. * Address some review comments. * Oops... * Add install option. * Change chiapos to fit the old standard. * Update chia/plotters/bladebit.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/install_plotter.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Lint. * Remove farmerkey as required. * Chia plotters chiapos works with no arguments. * Added get_plotters RPC to support the GUI (#8530) * Added 'get_plotters' daemon RPC. Probes installed/available plotters on behalf of the GUI. * Linter fix * Minor type tweak * Tweaks. * Run with default arguments all plotters. * Fix bug. * Update chia/plotters/bladebit.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Change bladebit repo. * Update chia/plotters/bladebit.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Re-added the connect-to-daemon hidden option * Use connect_to_daemon value for madmax and bladebit plot key resolution. * Updated --tmp_dir, --tmp_dir2, --final_dir options to match old options from chia plots create * Add CONNECT_TO_DAEMON as a valid option for madmax/bladebit * Thread multiplier should be an int * Passing params for madmax to start_plotting. Still needs cleanup/refactoring. * Update chia/plotters/bladebit.py Co-authored-by: Kyle Altendorf <sda@fstab.net> * Update chia/plotters/madmax.py Co-authored-by: Kyle Altendorf <sda@fstab.net> * Filename option -z. * Factor out calling the plotter. * First attempt refactor install scripts. * Switch to exec. * Attempt to fix mypy warning. * Remove filename. * Increase RLIMIT_NOFILE for madmax on non-Windows platforms * Add trailing path separator to madmax tmpdir/tmp2dir/finaldir arguments (required by madmax) * Fixes to support madmax plotting from the GUI. Writing output from the plotters now includes a flush to ensure the plotter log (used by the GUI) is updated frequently. * Handle madmax's tmptoggle option internally when plotting with the GUI * Formatting and linter fix * Fixed the -i option for bladebit * Construct BladeBit plotting options * Cleanup code for building plotter command line options * Added a post-processing step after each plotting job completes. Adds the final_dir plot directory as necessary. * Fix plotter root path * Reverting prior checkin. Need to figure out how to handle CHIA_ROOT being overridden * BladeBit support for Windows * BladeBit's --memory-json option is used to check memory requirements * Madmax Windows support * Plotters directory is now under CHIA_ROOT * Madmax version detection * BladeBit will default to 0 threads. BladeBit will max-out available threads with this configuration. * LGTM fixes * Module definition for chia.plotters to resolve mypy issues with chiapos (package vs chiapos.py) * Updated BladeBit build script to account for 1.2.0 changes. Replaced remaining subprocess.run calls with run_command. Use BladeBit's reported memory requirement instead of hardcoded value. * Show a disclaimer when using thirdparty plotter * Test adding mac madmax plotter to the installers * Get latest madmax from the latest GH release * Fix bad var name * m1 madmax * Add linux/linux arm * pip install -e for arm installer, so that its consistent with the other platforms when looking for additional files * madmax + windows * Get madmax with Invoke-WebRequest * Use the correct windows slashes * add madmax to the list of windows binaries * Check if madmax exists on windows install and move it to site packages if it does * Make sure windows has .exe extension for madmax * Update azure to get latest version of madmax from GH releases * Bladebit for linux/linux arm * Fix error with binaries.extend * Bundle bladebit + windows * Fix download url for bladebit * Check for bladebit in windows script. move to correct directory if it exists * Detect and use packaged plotters * Removed unnecessary import * Updating the branch to use chiaplotters_gui for installer verification * Removed a change that was intended for debugging only * Remove disclaimer * Updated for new madmax plotter with k33, k34 support. Updated chia-blockchain-gui submodule * Fixed typo * Package the chia_plot_k34 executable * Boink * Revert "Boink" This reverts commit 8d13c071100e99af8ec05ccb26eb2e9b411a6939. * Additional chia_plot_k34 spots that I missed * pyinstaller.spec fix for chia_plot_k34 * Windows installer fix for chia_plot_k34.exe * Restoring chia-blockchain-gui submodule to 047ce16 (as in main) * Update to chiapos 1.0.6 Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> Co-authored-by: Jeff Cruikshank <jeff@chia.net> Co-authored-by: Kyle Altendorf <sda@fstab.net> Co-authored-by: Chris Marslender <chrismarslender@gmail.com> Co-authored-by: Earle Lowe <30607889+emlowe@users.noreply.github.com>
2021-10-29 01:37:46 +03:00
THIS_IS_MAC = platform.system().lower().startswith("darwin")
ROOT = pathlib.Path(importlib.import_module("chia").__file__).absolute().parent.parent
def solve_name_collision_problem(analysis):
"""
There is a collision between the `chia` file name (which is the executable)
and the `chia` directory, which contains non-code resources like `english.txt`.
We move all the resources in the zipped area so there is no
need to create the `chia` directory, since the names collide.
Fetching data now requires going into a zip file, so it will be slower.
It's best if files that are used frequently are cached.
A sample large compressible file (1 MB of `/dev/zero`), seems to be
about eight times slower.
Note that this hack isn't documented, but seems to work.
"""
zipped = []
datas = []
for data in analysis.datas:
if str(data[0]).startswith("chia/"):
zipped.append(data)
else:
datas.append(data)
# items in this field are included in the binary
analysis.zipped_data = zipped
# these items will be dropped in the root folder uncompressed
analysis.datas = datas
keyring_imports = collect_submodules("keyring.backends")
# keyring uses entrypoints to read keyring.backends from metadata file entry_points.txt.
keyring_datas = copy_metadata("keyring")[0]
version_data = [
copy_metadata(name)[0]
for name in ["chia-blockchain", "chiapos"]
]
block_cipher = None
SERVERS = [
2022-02-08 05:29:54 +03:00
"data_layer",
"wallet",
"full_node",
"harvester",
"farmer",
"introducer",
"timelord",
]
if THIS_IS_WINDOWS:
hidden_imports_for_windows = ["win32timezone", "win32cred", "pywintypes", "win32ctypes.pywin32"]
else:
hidden_imports_for_windows = []
hiddenimports = [
*collect_submodules("chia"),
*keyring_imports,
*hidden_imports_for_windows,
]
binaries = []
if os.path.exists(f"{ROOT}/madmax/chia_plot"):
binaries.extend([
(
f"{ROOT}/madmax/chia_plot",
"madmax"
)
])
if os.path.exists(f"{ROOT}/madmax/chia_plot_k34",):
binaries.extend([
(
f"{ROOT}/madmax/chia_plot_k34",
"madmax"
)
])
if os.path.exists(f"{ROOT}/bladebit/bladebit"):
Add plotters. (#8497) * Initial commit add plotters. * Lint. * Add progress for bladebit. * Address some review comments. * Oops... * Add install option. * Change chiapos to fit the old standard. * Update chia/plotters/bladebit.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/install_plotter.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Lint. * Remove farmerkey as required. * Chia plotters chiapos works with no arguments. * Added get_plotters RPC to support the GUI (#8530) * Added 'get_plotters' daemon RPC. Probes installed/available plotters on behalf of the GUI. * Linter fix * Minor type tweak * Tweaks. * Run with default arguments all plotters. * Fix bug. * Update chia/plotters/bladebit.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Change bladebit repo. * Update chia/plotters/bladebit.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Re-added the connect-to-daemon hidden option * Use connect_to_daemon value for madmax and bladebit plot key resolution. * Updated --tmp_dir, --tmp_dir2, --final_dir options to match old options from chia plots create * Add CONNECT_TO_DAEMON as a valid option for madmax/bladebit * Thread multiplier should be an int * Passing params for madmax to start_plotting. Still needs cleanup/refactoring. * Update chia/plotters/bladebit.py Co-authored-by: Kyle Altendorf <sda@fstab.net> * Update chia/plotters/madmax.py Co-authored-by: Kyle Altendorf <sda@fstab.net> * Filename option -z. * Factor out calling the plotter. * First attempt refactor install scripts. * Switch to exec. * Attempt to fix mypy warning. * Remove filename. * Increase RLIMIT_NOFILE for madmax on non-Windows platforms * Add trailing path separator to madmax tmpdir/tmp2dir/finaldir arguments (required by madmax) * Fixes to support madmax plotting from the GUI. Writing output from the plotters now includes a flush to ensure the plotter log (used by the GUI) is updated frequently. * Handle madmax's tmptoggle option internally when plotting with the GUI * Formatting and linter fix * Fixed the -i option for bladebit * Construct BladeBit plotting options * Cleanup code for building plotter command line options * Added a post-processing step after each plotting job completes. Adds the final_dir plot directory as necessary. * Fix plotter root path * Reverting prior checkin. Need to figure out how to handle CHIA_ROOT being overridden * BladeBit support for Windows * BladeBit's --memory-json option is used to check memory requirements * Madmax Windows support * Plotters directory is now under CHIA_ROOT * Madmax version detection * BladeBit will default to 0 threads. BladeBit will max-out available threads with this configuration. * LGTM fixes * Module definition for chia.plotters to resolve mypy issues with chiapos (package vs chiapos.py) * Updated BladeBit build script to account for 1.2.0 changes. Replaced remaining subprocess.run calls with run_command. Use BladeBit's reported memory requirement instead of hardcoded value. * Show a disclaimer when using thirdparty plotter * Test adding mac madmax plotter to the installers * Get latest madmax from the latest GH release * Fix bad var name * m1 madmax * Add linux/linux arm * pip install -e for arm installer, so that its consistent with the other platforms when looking for additional files * madmax + windows * Get madmax with Invoke-WebRequest * Use the correct windows slashes * add madmax to the list of windows binaries * Check if madmax exists on windows install and move it to site packages if it does * Make sure windows has .exe extension for madmax * Update azure to get latest version of madmax from GH releases * Bladebit for linux/linux arm * Fix error with binaries.extend * Bundle bladebit + windows * Fix download url for bladebit * Check for bladebit in windows script. move to correct directory if it exists * Detect and use packaged plotters * Removed unnecessary import * Updating the branch to use chiaplotters_gui for installer verification * Removed a change that was intended for debugging only * Remove disclaimer * Updated for new madmax plotter with k33, k34 support. Updated chia-blockchain-gui submodule * Fixed typo * Package the chia_plot_k34 executable * Boink * Revert "Boink" This reverts commit 8d13c071100e99af8ec05ccb26eb2e9b411a6939. * Additional chia_plot_k34 spots that I missed * pyinstaller.spec fix for chia_plot_k34 * Windows installer fix for chia_plot_k34.exe * Restoring chia-blockchain-gui submodule to 047ce16 (as in main) * Update to chiapos 1.0.6 Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> Co-authored-by: Jeff Cruikshank <jeff@chia.net> Co-authored-by: Kyle Altendorf <sda@fstab.net> Co-authored-by: Chris Marslender <chrismarslender@gmail.com> Co-authored-by: Earle Lowe <30607889+emlowe@users.noreply.github.com>
2021-10-29 01:37:46 +03:00
binaries.extend([
(
f"{ROOT}/bladebit/bladebit",
"bladebit"
)
])
if os.path.exists(f"{ROOT}/bladebit/bladebit_cuda"):
binaries.extend([
(
f"{ROOT}/bladebit/bladebit_cuda",
"bladebit"
)
])
if THIS_IS_WINDOWS:
chia_mod = importlib.import_module("chia")
dll_paths = pathlib.Path(sysconfig.get_path("platlib")) / "*.dll"
binaries = [
(
dll_paths,
".",
),
(
"C:\\Windows\\System32\\msvcp140.dll",
".",
),
(
"C:\\Windows\\System32\\vcruntime140_1.dll",
".",
),
Add plotters. (#8497) * Initial commit add plotters. * Lint. * Add progress for bladebit. * Address some review comments. * Oops... * Add install option. * Change chiapos to fit the old standard. * Update chia/plotters/bladebit.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/install_plotter.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Lint. * Remove farmerkey as required. * Chia plotters chiapos works with no arguments. * Added get_plotters RPC to support the GUI (#8530) * Added 'get_plotters' daemon RPC. Probes installed/available plotters on behalf of the GUI. * Linter fix * Minor type tweak * Tweaks. * Run with default arguments all plotters. * Fix bug. * Update chia/plotters/bladebit.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Change bladebit repo. * Update chia/plotters/bladebit.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Update chia/plotters/plotters.py Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> * Re-added the connect-to-daemon hidden option * Use connect_to_daemon value for madmax and bladebit plot key resolution. * Updated --tmp_dir, --tmp_dir2, --final_dir options to match old options from chia plots create * Add CONNECT_TO_DAEMON as a valid option for madmax/bladebit * Thread multiplier should be an int * Passing params for madmax to start_plotting. Still needs cleanup/refactoring. * Update chia/plotters/bladebit.py Co-authored-by: Kyle Altendorf <sda@fstab.net> * Update chia/plotters/madmax.py Co-authored-by: Kyle Altendorf <sda@fstab.net> * Filename option -z. * Factor out calling the plotter. * First attempt refactor install scripts. * Switch to exec. * Attempt to fix mypy warning. * Remove filename. * Increase RLIMIT_NOFILE for madmax on non-Windows platforms * Add trailing path separator to madmax tmpdir/tmp2dir/finaldir arguments (required by madmax) * Fixes to support madmax plotting from the GUI. Writing output from the plotters now includes a flush to ensure the plotter log (used by the GUI) is updated frequently. * Handle madmax's tmptoggle option internally when plotting with the GUI * Formatting and linter fix * Fixed the -i option for bladebit * Construct BladeBit plotting options * Cleanup code for building plotter command line options * Added a post-processing step after each plotting job completes. Adds the final_dir plot directory as necessary. * Fix plotter root path * Reverting prior checkin. Need to figure out how to handle CHIA_ROOT being overridden * BladeBit support for Windows * BladeBit's --memory-json option is used to check memory requirements * Madmax Windows support * Plotters directory is now under CHIA_ROOT * Madmax version detection * BladeBit will default to 0 threads. BladeBit will max-out available threads with this configuration. * LGTM fixes * Module definition for chia.plotters to resolve mypy issues with chiapos (package vs chiapos.py) * Updated BladeBit build script to account for 1.2.0 changes. Replaced remaining subprocess.run calls with run_command. Use BladeBit's reported memory requirement instead of hardcoded value. * Show a disclaimer when using thirdparty plotter * Test adding mac madmax plotter to the installers * Get latest madmax from the latest GH release * Fix bad var name * m1 madmax * Add linux/linux arm * pip install -e for arm installer, so that its consistent with the other platforms when looking for additional files * madmax + windows * Get madmax with Invoke-WebRequest * Use the correct windows slashes * add madmax to the list of windows binaries * Check if madmax exists on windows install and move it to site packages if it does * Make sure windows has .exe extension for madmax * Update azure to get latest version of madmax from GH releases * Bladebit for linux/linux arm * Fix error with binaries.extend * Bundle bladebit + windows * Fix download url for bladebit * Check for bladebit in windows script. move to correct directory if it exists * Detect and use packaged plotters * Removed unnecessary import * Updating the branch to use chiaplotters_gui for installer verification * Removed a change that was intended for debugging only * Remove disclaimer * Updated for new madmax plotter with k33, k34 support. Updated chia-blockchain-gui submodule * Fixed typo * Package the chia_plot_k34 executable * Boink * Revert "Boink" This reverts commit 8d13c071100e99af8ec05ccb26eb2e9b411a6939. * Additional chia_plot_k34 spots that I missed * pyinstaller.spec fix for chia_plot_k34 * Windows installer fix for chia_plot_k34.exe * Restoring chia-blockchain-gui submodule to 047ce16 (as in main) * Update to chiapos 1.0.6 Co-authored-by: Jeff Cruikshank <paninaro@gmail.com> Co-authored-by: Jeff Cruikshank <jeff@chia.net> Co-authored-by: Kyle Altendorf <sda@fstab.net> Co-authored-by: Chris Marslender <chrismarslender@gmail.com> Co-authored-by: Earle Lowe <30607889+emlowe@users.noreply.github.com>
2021-10-29 01:37:46 +03:00
(
f"{ROOT}\\madmax\\chia_plot.exe",
"madmax"
),
(
f"{ROOT}\\madmax\\chia_plot_k34.exe",
"madmax"
),
(
f"{ROOT}\\bladebit\\bladebit.exe",
"bladebit"
),
(
f"{ROOT}\\bladebit\\bladebit_cuda.exe",
"bladebit"
),
]
datas = []
datas.append((f"{ROOT}/chia/util/english.txt", "chia/util"))
datas.append((f"{ROOT}/chia/util/initial-config.yaml", "chia/util"))
for path in sorted({path.parent for path in ROOT.joinpath("chia").rglob("*.hex")}):
datas.append((f"{path}/*.hex", path.relative_to(ROOT)))
datas.append((f"{ROOT}/chia/ssl/*", "chia/ssl"))
datas.append((f"{ROOT}/mozilla-ca/*", "mozilla-ca"))
datas.extend(version_data)
pathex = []
def add_binary(name, path_to_script, collect_args):
analysis = Analysis(
[path_to_script],
pathex=pathex,
binaries=binaries,
datas=datas,
hiddenimports=hiddenimports,
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False,
)
solve_name_collision_problem(analysis)
binary_pyz = PYZ(analysis.pure, analysis.zipped_data, cipher=block_cipher)
binary_exe = EXE(
binary_pyz,
analysis.scripts,
[],
exclude_binaries=True,
name=name,
debug=False,
bootloader_ignore_signals=False,
strip=False,
)
collect_args.extend(
[
binary_exe,
analysis.binaries,
analysis.zipfiles,
analysis.datas,
]
)
COLLECT_ARGS = []
add_binary("chia", f"{ROOT}/chia/cmds/chia.py", COLLECT_ARGS)
add_binary("daemon", f"{ROOT}/chia/daemon/server.py", COLLECT_ARGS)
for server in SERVERS:
add_binary(f"start_{server}", f"{ROOT}/chia/server/start_{server}.py", COLLECT_ARGS)
add_binary("start_crawler", f"{ROOT}/chia/seeder/start_crawler.py", COLLECT_ARGS)
add_binary("start_seeder", f"{ROOT}/chia/seeder/dns_server.py", COLLECT_ARGS)
add_binary("start_data_layer_http", f"{ROOT}/chia/data_layer/data_layer_server.py", COLLECT_ARGS)
add_binary("start_data_layer_s3_plugin", f"{ROOT}/chia/data_layer/s3_plugin_service.py", COLLECT_ARGS)
add_binary("timelord_launcher", f"{ROOT}/chia/timelord/timelord_launcher.py", COLLECT_ARGS)
COLLECT_KWARGS = dict(
strip=False,
upx_exclude=[],
name="daemon",
)
coll = COLLECT(*COLLECT_ARGS, **COLLECT_KWARGS)