git.iko.soy/typeable/nixpkgs
on master help sign in
python3Packages.dask: fix sandboxed builds

Importing dask.dataframe in a sandboxed build results in a TypeError like this: File "/nix/store/nv60iri29bia4szhhcvsdxgsci4wxvp6-python3.8-dask-2021.03.0/lib/python3.8/site-packages/dask/dataframe/io/csv.py", line 392, in <module> AUTO_BLOCKSIZE = auto_blocksize(TOTAL_MEM, CPU_COUNT) File "/nix/store/nv60iri29bia4szhhcvsdxgsci4wxvp6-python3.8-dask-2021.03.0/lib/python3.8/site-packages/dask/dataframe/io/csv.py", line 382, in auto_blocksize blocksize = int(total_memory // cpu_count / memory_factor) TypeError: unsupported operand type(s) for //: 'int' and 'NoneType' This occurs because dask.dataframe has a non-deterministic component which generates an automatic chunk-size based on system information. This went unnoticed because the dask tests were disabled. Changes: - add a patch making the chunk-size inference more robust - re-enable the tests Resolves #120307

commitceeaf2d0669cc5a4d86e6313cc0fdb6774659884 authorNoah D. Brenowitz <nbren12@gmail.com> dateFri 23 Apr 2021, 00:50 (6 years ago) parentc21475e
1 file changed+17−1
pkgs/development/python-modules/dask/default.nix+17−1
pkgs/development/python-modules/dask/default.nix+17−1
@@ -1,6 +1,7 @@
11{ lib
22, bokeh
33, buildPythonPackage
4+, fetchpatch
45, fetchFromGitHub
56, fsspec
67, pytestCheckHook
@@ -42,7 +43,7 @@ buildPythonPackage rec {
4243 distributed
4344 ];
4445
45− doCheck = false;
46+ doCheck = true;
4647
4748 checkInputs = [
4849 pytestCheckHook
@@ -52,6 +53,16 @@ buildPythonPackage rec {
5253
5354 dontUseSetuptoolsCheck = true;
5455
56+ patches = [
57+ # dask dataframe cannot be imported in sandboxed builds
58+ # See https://github.com/dask/dask/pull/7601
59+ (fetchpatch {
60+ url = "https://github.com/dask/dask/commit/9ce5b0d258cecb3ef38fd844135ad1f7ac3cea5f.patch";
61+ sha256 = "sha256-1EVRYwAdTSEEH9jp+UOnrijzezZN3iYR6q6ieYJM3kY=";
62+ name = "fix-dask-dataframe-imports-in-sandbox.patch";
63+ })
64+ ];
65+
5566 postPatch = ''
5667 # versioneer hack to set version of github package
5768 echo "def get_versions(): return {'dirty': False, 'error': None, 'full-revisionid': None, 'version': '${version}'}" > dask/_version.py
@@ -66,8 +77,13 @@ buildPythonPackage rec {
6677 disabledTests = [
6778 "test_annotation_pack_unpack"
6879 "test_annotations_blockwise_unpack"
80+ # this test requires features of python3Packages.psutil that are
81+ # blocked in sandboxed-builds
82+ "test_auto_blocksize_csv"
6983 ];
7084
85+ pythonImportsCheck = [ "dask.dataframe" "dask" "dask.array" ];
86+
7187 meta = with lib; {
7288 description = "Minimal task scheduling abstraction";
7389 homepage = "https://dask.org/";