git.iko.soy/typeable/nixpkgs
on master help sign in
foundationdb: rework python bindings, build system

FoundationDB uses Python at build time for some code generation. However, it also has the official python bindings inside the source code too, and the code for the Python bindings has some of it auto-generated at compile time. This made building python packages unattractive: we want to use the source code generated from the FoundationDB build, but we don't want to rebuild it. Previously we would override the 'python' input to the FoundationDB module, but this meant we would do a complete rebuild, as it was a necessary build time dependency, even though the resulting generated code itself would not change. Furthermore, FoundationDB versions < 6.0 don't properly support Python 3 *for the build system*, though the bindings supported it, so that caused build failures. But the first effect is the worst: it meant building separate python2 and python3 packages implied two complete rebuilds of a single FoundationDB version. This meant rather than 3 FDB builds, we'd do 3*N where N = the number of major Python versions we support. Finally, because we did not use pip to generate a wheel that we install with metadata recorded for the installation, the FoundationDB python package couldn't be used as an input to other setup.py-based packages: there would be no recorded metadata in the dist-info folder which would say this is the foundationdb package. This greatly limits its utility. To fix all this, we do a few things: - Apply some patches to fix the build system with Python 3.x for older FoundationDB versions. (This is nice if end-users have overridden the global Python version for some reason.) - Move python directly into nativeBuildInputs, so it is only a build time dependency. - Take the python source code from the ./bindings directory and tar it up use later after the build is done, so we get to keep the generated code. This is the new 'pythonsrc' output from the build. This code doesn't change based on whether or not the input or resulting package is using Python 2 or 3, it's totally deterministic. - The build system also patches up the python source code a little, so it can be installed directly with setup.py (it needs a little stuff that it normally expects the build system to do.) - Rework the python package to a separate file that uses buildPythonPackage directly. Because the source code is already prepared, it needs almost nothing else. Furthermore, this kills the override itself for the foundationdb package, meaning rebuilds are no longer needed. - This package is very simple and just uses foundationdb.pythonsrc as its source input. It also ensures a link to libfdb_c.so can be found by ctypes (using substituteInPlace) - python-packages.nix now just uses callPackage directly. The net effect of this is, most importantly, that python packages do not imply a full rebuild of the server source code: building python2 and python3 packages from a version of FoundationDB now does not need to override the foundationdb python input, reducing the number of needless builds. They instead just run setup.py with the given version as input. The second biggest effect is that wheel metadata is recorded correctly, meaning dependent-python-packages that want to use the FoundationDB bindings e.g. from PyPi should now work fine with buildPythonPackage. Signed-off-by: Austin Seipp <aseipp@pobox.com>

commit6054dabc11de51bfd8688089c731e39bbb40910b authorAustin Seipp <aseipp@pobox.com> dateThu 15 Nov 2018, 17:03 (8 years ago) parent32948a6
3 files changed+55−21
pkgs/servers/foundationdb/default.nix+28−10 pkgs/servers/foundationdb/python.nix+24 pkgs/top-level/python-packages.nix+3−11
pkgs/servers/foundationdb/default.nix+28−10
@@ -1,5 +1,5 @@
11{ stdenv49
2−, lib, fetchurl, fetchFromGitHub
2+, lib, fetchurl, fetchpatch, fetchFromGitHub
33
44, which, findutils, m4, gawk
55, python, openjdk, mono58, libressl
@@ -47,8 +47,8 @@ let
4747 inherit rev sha256;
4848 };
4949
50− nativeBuildInputs = [ gawk which m4 findutils mono58 ];
51− buildInputs = [ python openjdk libressl boost ];
50+ nativeBuildInputs = [ python openjdk gawk which m4 findutils mono58 ];
51+ buildInputs = [ libressl boost ];
5252
5353 patches =
5454 [ # For 5.2+, we need a slightly adjusted patch to fix all the ldflags
@@ -57,10 +57,24 @@ let
5757 then ./ldflags-6.0.patch
5858 else ./ldflags-5.2.patch)
5959 else ./ldflags-5.1.patch)
60− ] ++
60+ ]
6161 # for 6.0+, we do NOT need to apply this version fix, since we can specify
6262 # it ourselves. see configurePhase
63− (lib.optional (!lib.versionAtLeast version "6.0") ./fix-scm-version.patch);
63+ ++ (lib.optional (!lib.versionAtLeast version "6.0") ./fix-scm-version.patch)
64+ # Versions less than 6.0 have a busted Python 3 build due to an outdated
65+ # use of 'print'. Also apply an update to the six module with many bugfixes,
66+ # which is in 6.0+ as well
67+ ++ (lib.optional (!lib.versionAtLeast version "6.0") (fetchpatch {
68+ name = "update-python-six.patch";
69+ url = "https://github.com/apple/foundationdb/commit/4bd9efc4fc74917bc04b07a84eb065070ea7edb2.patch";
70+ sha256 = "030679lmc86f1wzqqyvxnwjyfrhh54pdql20ab3iifqpp9i5mi85";
71+ }))
72+ ++ (lib.optional (!lib.versionAtLeast version "6.0") (fetchpatch {
73+ name = "import-for-python-print.patch";
74+ url = "https://github.com/apple/foundationdb/commit/ded17c6cd667f39699cf663c0e87fe01e996c153.patch";
75+ sha256 = "11y434w68cpk7shs2r22hyrpcrqi8vx02cw7v5x79qxvnmdxv2an";
76+ }))
77+ ;
6478
6579 postPatch = ''
6680 # note: this does not do anything for 6.0+
@@ -106,7 +120,6 @@ let
106120
107121 installPhase = ''
108122 mkdir -vp $out/{bin,libexec/plugins} $lib/{lib,share/java} $dev/include/foundationdb
109− mkdir -vp $python/lib/${python.libPrefix}/site-packages
110123
111124 '' + lib.optionalString (!lib.versionAtLeast version "6.0") ''
112125 # we only copy the TLS library on < 6.0, since it's compiled-in otherwise
@@ -123,10 +136,15 @@ let
123136 cp -v ./bindings/java/foundationdb-client.jar $lib/share/java/fdb-java.jar
124137
125138 # python
139+ cp LICENSE ./bindings/python
140+ substitute ./bindings/python/setup.py.in ./bindings/python/setup.py \
141+ --replace 'VERSION' "${version}"
142+ rm -f ./bindings/python/setup.py.in
126143 rm -f ./bindings/python/fdb/*.pth # remove useless files
127− cp -R ./bindings/python/fdb $python/lib/${python.libPrefix}/site-packages/fdb
128− # symlink a copy of the shared object into place, so that impl.py can load it
129− ln -sv $lib/lib/libfdb_c.so $python/lib/${python.libPrefix}/site-packages/fdb/libfdb_c.so
144+ rm -f ./bindings/python/*.rst ./bindings/python/*.mk
145+
146+ cp -R ./bindings/python/ tmp-pythonsrc/
147+ tar -zcf $pythonsrc --transform s/tmp-pythonsrc/python-foundationdb/ ./tmp-pythonsrc/
130148
131149 # binaries
132150 for x in fdbbackup fdbcli fdbserver fdbmonitor; do
@@ -140,7 +158,7 @@ let
140158 ln -sfv $out/bin/fdbbackup $out/libexec/backup_agent
141159 '';
142160
143− outputs = [ "out" "lib" "dev" "python" ];
161+ outputs = [ "out" "lib" "dev" "pythonsrc" ];
144162
145163 meta = with stdenv.lib; {
146164 description = "Open source, distributed, transactional key-value store";
pkgs/servers/foundationdb/python.nixnew file+24
@@ -0,0 +1,24 @@
1+{ buildPythonPackage, lib, foundationdb }:
2+
3+buildPythonPackage rec {
4+ pname = "foundationdb";
5+ version = foundationdb.version;
6+
7+ src = foundationdb.pythonsrc;
8+ unpackCmd = "tar xf $curSrc";
9+
10+ patchPhase = ''
11+ substituteInPlace ./fdb/impl.py \
12+ --replace libfdb_c.so "${foundationdb.lib}/lib/libfdb_c.so"
13+ '';
14+
15+ doCheck = false;
16+
17+ meta = with lib; {
18+ description = "Python bindings for FoundationDB";
19+ homepage = https://www.foundationdb.org;
20+ license = with licenses; [ asl20 ];
21+ maintainers = with maintainers; [ thoughtpolice ];
22+ };
23+}
24+
pkgs/top-level/python-packages.nix+3−11
@@ -11052,17 +11052,9 @@ EOF
1105211052
1105311053 rfc7464 = callPackage ../development/python-modules/rfc7464 { };
1105411054
11055− foundationdb51 = (toPythonModule (pkgs.fdbPackages.override {
11056− inherit python;
11057− }).foundationdb51).python;
11058−
11059− foundationdb52 = (toPythonModule (pkgs.fdbPackages.override {
11060− inherit python;
11061− }).foundationdb52).python;
11062−
11063− foundationdb60 = (toPythonModule (pkgs.fdbPackages.override {
11064− inherit python;
11065− }).foundationdb60).python;
11055+ foundationdb51 = callPackage ../servers/foundationdb/python.nix { foundationdb = pkgs.foundationdb51; };
11056+ foundationdb52 = callPackage ../servers/foundationdb/python.nix { foundationdb = pkgs.foundationdb52; };
11057+ foundationdb60 = callPackage ../servers/foundationdb/python.nix { foundationdb = pkgs.foundationdb60; };
1106611058
1106711059 libtorrentRasterbar = (toPythonModule (pkgs.libtorrentRasterbar.override {
1106811060 inherit python;