0229ed4506
Summary: In coverage collection mode a special module loader is prepended to `sys.meta_path`. In very specific conditions this module loader can end up returning a loader pointing to a _completely wrong module_. When importing symbols from the wrong module errors occur. The conditions to trigger the bug are: - running in coverage collection mode, enabling the custom loader - the test binary is a zip (e.g. par_style=fastzip) - having a module name where the end part matches the name of a builtin Python module When these conditions were met, the special loader would return the builtin Python module instead of the expected module. E.g. when loading a module like `myapp.somemod.platform` in a zip style binary. The custom loader first calls `imp.find_module()` to find the module it wants to return a wrapped loader for. This fails for modules included in the test binary, because the builtin `imp` module can not load from zips. This was the trigger leading to the call to the buggy code. When the initial call to `imp.find_module()` failed, the custom loader would try a second call, asking the internal loader to this time try any path on `sys.path`. For most module names this call would also fail, making the custom loader return `None`, after which Python tries other loaders on `sys.path`. However, when the final part of the module that was asked to load matches the name of a Python builtin module, then the second call to the `imp` module would succeed, returning a loader for the builtin module. E.g. `platform` when asking for `myapp.somemod.platform`. This diff fixes the issue by removing the broken second call to the internal loader. This will never have worked, we just have not triggered or noticed triggering the wrong loading before. Differential Revision: D20798119 fbshipit-source-id: dffb54e308106a81af21b63c5ee64c6ca2041920 |
||
|---|---|---|
| .. | ||
| CMake | ||
| getdeps | ||
| manifests | ||
| specs | ||
| .gitignore | ||
| docker_build_with_ccache.sh | ||
| docker_builder.py | ||
| docker_enable_ipv6.sh | ||
| fbcode_builder_config.py | ||
| fbcode_builder.py | ||
| getdeps.py | ||
| LICENSE | ||
| make_docker_context.py | ||
| parse_args.py | ||
| README.docker | ||
| README.md | ||
| shell_builder.py | ||
| shell_quoting.py | ||
| travis_docker_build.sh | ||
| utils.py | ||
Easy builds for Facebook projects
This is a Python 2.6+ library designed to simplify continuous-integration (and other builds) of Facebook projects.
For external Travis builds, the entry point is travis_docker_build.sh.
Using Docker to reproduce a CI build
If you are debugging or enhancing a CI build, you will want to do so from host or virtual machine that can run a reasonably modern version of Docker:
./make_docker_context.py --help # See available options for OS & compiler
# Tiny wrapper that starts a Travis-like build with compile caching:
os_image=ubuntu:18.04 \
gcc_version=7 \
make_parallelism=2 \
travis_cache_dir=~/travis_ccache \
./travis_docker_build.sh &> build_at_$(date +'%Y%m%d_%H%M%S').log
IMPORTANT: Read fbcode_builder/README.docker before diving in!
Setting travis_cache_dir turns on ccache,
saving a fresh copy of ccache.tgz after every build. This will invalidate
Docker's layer cache, foring it to rebuild starting right after OS package
setup, but the builds will be fast because all the compiles will be cached.
To iterate without invalidating the Docker layer cache, just cd /tmp/docker-context-* and interact with the Dockerfile normally. Note
that the docker-context-* dirs preserve a copy of ccache.tgz as they
first used it.
What to read next
The *.py files are fairly well-documented. You might want to peruse them in this order:
- shell_quoting.py
- fbcode_builder.py
- docker_builder.py
- make_docker_context.py
As far as runs on Travis go, the control flow is:
- .travis.yml calls
- travis_docker_build.sh calls
- docker_build_with_ccache.sh
This library also has an (unpublished) component targeting Facebook's internal continuous-integration platform using the same build-step DSL.
Contributing
Please follow the ambient style (or PEP-8), and keep the code Python 2.6
compatible -- since fbcode_builder's only dependency is Docker, we want to
allow building projects on even fairly ancient base systems. We also wish
to be compatible with Python 3, and would appreciate it if you kept that
in mind while making changes also.