mitmproxy was buffering all requests (except pythonhosted.*), causing
pip to time out when getting big packages from custom URLs.
Setting flow.response.stream in the responseheaders hook fixes this.
* fetchPipMetadata: set meta.mainProgram
* fetchPipMetadata: deduplicate test fixtures
* fetchPipMetadata: add is_direct to lock file
* devShell: proof of concept
* python-local-development: add editables example
* devshell: simplify editables interface
* buildPythonPackage: remove editable option again
* devshell: use findRoot for editables
* devshell: add support for .whl sources
* devshell: avoid polluting the shells environment
* devshell: make editable.nix flatter
because there's only 1 attr left now
* devshell: fix pyEnv by filtering editables
from the environment.
* devshell: skip existing editables
* devshell: remove patched pyEnv, rewrite sys.path
* devshell: add special case for root package
* python-local-development: improve editable example
* python-local-development: filter source
* devshell: reset site_dir on each load
* pip: don't ignoreCollisions in pyEnv
* pip: default to no sitecustomize.py...
and teach the devshell how to load it
* pip: rewrite editable in python
* editable: use shutil.copytree
* editable: refactor into functions
* editables: add suport for console_scripts
* improve pip editables interface
- change editables interface to use bool or path
- expose public.shellHook for composition
- use shellHook by default in public.devShell
- extend example to include a script
* improve edtiable support:
- remove dependency on root package build
- always set root package as editable by default
- compute dit-info dir by calling a packages build backend or extracting directly from the wheel
- only pass required sources to editables shell hook
- add build inputs of all editables to teh dev shell
- move editables state to .dream2nix/python to remove likelyhood of collision with other ecosystems state
- delete editables state if editables configuration changed
* refine python editable support:
- never copy dependency sources
- force user to specify a local path
---------
Co-authored-by: DavHau <hsngrmpf+github@gmail.com>
The pypi proxy filters packages, which can end up corrupting the pip
cache. For an example of what this would look like, see
https://github.com/pypi/warehouse/issues/14457 This commit prevents the
the cache from storing our modified response by adding a 'Vary: *'
header. Judging from the caching code, this is the least intrusive
approach to prevent caching ('Cache-Control: no-cache' deletes existing
caches for example).
The pip locker [sometimes created a URL without hash][1] to reflect that there was a local source.
This produced the following error when trying to evaluate that derivation:
… from call site
at /nix/store/qdzdi6qvrqxfqxhi3j70d5dsl9f5jpby-source/modules/dream2nix/pip/default.nix:86:28:
85| mkDerivation = {
86| src = l.mkDefault (fetchers.${metadata.sources.${config.name}.type} metadata.sources.${config.name});
| ^
87| doCheck = l.mkDefault false;
… while calling 'url'
at /nix/store/qdzdi6qvrqxfqxhi3j70d5dsl9f5jpby-source/modules/dream2nix/pip/default.nix:57:11:
56| fetchers = {
57| url = info: l.fetchurl {inherit (info) url sha256;};
| ^
58| git = info: config.deps.fetchgit {inherit (info) url sha256 rev;};
error: value is null while a string was expected
Of course, one cannot call `builtins.fetchurl` without a `sha256` argument.
Now, local dependencies are specified exactly like that in the lock file. Now, we'll fetch nothing when dealing with local dependencies.
Also, their evaluation when generating `preFixup` attribute is delayed until build time. This was a bigger problem due to the previous bug, but in any case it should still help to reduce evaluation overhead.
FWIW, sometimes local sources might not really exit in the derivation source tree until build time. For example, when building aggregated sources with meta-repo management tools such as Mr. Chef.
[1]: 40b65e4598/pkgs/fetchPipMetadata/src/fetch_pip_metadata/lock_file_from_report.py (L72C28-L72C28)
For many users not having a snapshot date is a valid use case.
Also many users will expect their lock file to be updated to the latest dependency versions via a nix run .#package.config.lock.refresh. so it can be the default behavior.
When you had a pip git requirement such as `git+https://github.com/OCA/maintainer-tools`, `fetch_pip_metadata` was failing with:
```
Traceback (most recent call last):
File "/nix/store/v28wx7mr5m9wasbnmsij4dqfrgl0d28q-python3.10-fetch_pip_metadata/bin/.fetch_pip_metadata-wrapped", line 9, in <module>
sys.exit(fetch_pip_metadata())
File "/nix/store/v28wx7mr5m9wasbnmsij4dqfrgl0d28q-python3.10-fetch_pip_metadata/lib/python3.10/site-packages/fetch_pip_metadata/__init__.py", line 126, in fetch_pip_metadata
lock = lock_file_from_report(report, project_root=args.project_root)
File "/nix/store/v28wx7mr5m9wasbnmsij4dqfrgl0d28q-python3.10-fetch_pip_metadata/lib/python3.10/site-packages/fetch_pip_metadata/lock_file_from_report.py", line 240, in lock_file_from_report
name, package = lock_entry_from_report_entry(install, project_root)
File "/nix/store/v28wx7mr5m9wasbnmsij4dqfrgl0d28q-python3.10-fetch_pip_metadata/lib/python3.10/site-packages/fetch_pip_metadata/lock_file_from_report.py", line 154, in lock_entry_from_report_entry
info = lock_info(download_info)
File "/nix/store/v28wx7mr5m9wasbnmsij4dqfrgl0d28q-python3.10-fetch_pip_metadata/lib/python3.10/site-packages/fetch_pip_metadata/lock_file_from_report.py", line 132, in lock_info_from_vcs
sha256 = nix_prefetch_git(url, rev)
File "/nix/store/v28wx7mr5m9wasbnmsij4dqfrgl0d28q-python3.10-fetch_pip_metadata/lib/python3.10/site-packages/fetch_pip_metadata/lock_file_from_report.py", line 29, in nix_prefetch_git
subprocess.run(
File "/nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/python3.10/subprocess.py", line 503, in run
with Popen(*popenargs, **kwargs) as process:
File "/nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/python3.10/subprocess.py", line 971, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/python3.10/subprocess.py", line 1863, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'nix-prefetch-git'
Traceback (most recent call last):
File "/nix/store/0715rzxpxka6b12x7nl6i25gbxp9m80g-refresh/bin/refresh", line 37, in <module>
lock_data = run_refresh_scripts(refresh_scripts)
File "/nix/store/0715rzxpxka6b12x7nl6i25gbxp9m80g-refresh/bin/refresh", line 33, in run_refresh_scripts
refresh_scripts[name] = run_refresh_script(value["script"])
File "/nix/store/0715rzxpxka6b12x7nl6i25gbxp9m80g-refresh/bin/refresh", line 19, in run_refresh_script
subprocess.run(
File "/nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/nix/store/2qfpia31lh9dajnf9s6dm0rp5dw109d3-script.sh']' returned non-zero exit status 1.
```
@moduon MT-1075
* Make pip lock script use its own python version
To make the lock script independent of the python version used in the
user's project, it needs to use its own python environment. This was
already the case for the mitm proxy, but not yet for the lock script
itself. This commit ensures that the lock script uses its own python
environment, independent of the user python version, and shares it with
the mitm proxy.
* Fix IOKit dependency in htop example
* Add hash for git based packages
* Support install git dependencies
* Add test for pip with git dependency
* Update lockfiles for python example projects
This does not change any of the hashes or versions, just adds the type:
"url" property.
---------
Co-authored-by: DavHau <hsngrmpf+github@gmail.com>
- add flake-module.nix
- remove custom logic to find the repo root
- use findRoot script to find repo root
- use gitMinimal instead of git
- use separate python version for lock script than the python version to lock for. This prevents rebuilds and allows defining a standalone fetchPipMetadata package
When parsing the pip report, parsing some requirements could fail, as the packaging library used for the parsing was different from the one used by pip internally.
This makes sure that always the pip internals packaging library is used, which should prevent inconsistencies like that.
pip does allow cycles and they are common in larger dependency trees. Instead of crashing on them, we should allow the user to deal with them.
As a result of this, it can happen that during the installation of individual dependencies, some dep(s) might be missing, which then can be fixed by including them manually or with a `--no-deps`
