Summary:
We use drive letter substitutions to workaround Windows
filename length limitations. On my personal Windows system running
python 3.8.2 realpath inside an ssh session manages to resolve to
the full filename which causes the boost build to fail.
Avoid that!
Reviewed By: chadaustin
Differential Revision: D21507851
fbshipit-source-id: 1220c1c85d2124ddc51f42cefff2ce00e10c55c9
Summary:
For large projects, with lots of tests, running all the tests can take a lot
of time, but for quick development iteration, only a subset of the tests may
be needed to run.
On non-Windows platforms, this can be easily achieved by manually executing
the individual tests binaries and use the builtin filtering mechanism of that
test binary to achieve the goal. On Windows, this can quickly become
impossible as DLLs might not be available, and the right PATH would need to
be manually specified by hand to execute the tests binaries[0].
To solve this, let's simply provide a unified way of running specific tests
by passing in a regexp. Both testpilot and CTest do support regex to execute
specific tests. My understanding is that cargo doesn't yet allows regex, but
will in the future.
[0]: And a missing DLLs would produce no output when executed from
PowerShell, which makes this very confusing.
Reviewed By: wez
Differential Revision: D21484774
fbshipit-source-id: ee32e950e25bb2a498a2b364a447955a917b0590
Summary:
Previously, the Windows build was litered with warnings of the form (typo included):
warning C4250: 'C1': inherits 'C2::C2::method' via dominance
note: see declaration of 'C2::method'
Microsoft doesn't offer any recommendation, and the internet suggest that the
right `using` should silence it. That's unfortunately not the case, adding:
using C2::method
In `C1` doesn't do anything, and the compiler still complains :(.
Since the warning appears to be non-actionable, and looks more like a
"notice" than a warning, let's just silence it.
Reviewed By: wez
Differential Revision: D21395095
fbshipit-source-id: ae661b3ed61303e6361b8a15d9e7c6b9627ea8c1
Summary: This also unblocks the MacOS Mononoke builds, so enabling them back
Reviewed By: farnz
Differential Revision: D21455422
fbshipit-source-id: 4eae10785db5b93b1167f580a1c887ee4c8a96a2
Summary:
The breakage has been fixed, so bring back the manifest, but only the Linux one, because the Mac version is failing due to another issue.
Also to make it easier to debug issues on GitHub Actions separate out the dependencies build from Mononoke and rust-shed builds.
Reviewed By: krallin
Differential Revision: D21448412
fbshipit-source-id: 68d89c858d1692727a7fd66bca114920e6dfb4dc
Summary:
fboss-oss build links to hash that corresponds to tag v4.4.0 released on Jan 11 2016
```
repo_url = https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git
rev = 92a0236a3cdf3438000834121b7ea8a09f1f52b1
```
The change is to update the iproute2 version to ```v4.12.0``` (July 5 2017) to match with the version used internally to Facebook
Reviewed By: shri-khare
Differential Revision: D21411714
fbshipit-source-id: fac606396e284193bb7199cf60d2601594bfa5f0
Summary:
D21364132 accidentally broke this; we can't run the fetcher
for projects for which we pulled the build out of cache, because there
is no source to update in that case.
This commit adjusts the logic so that we write out a marker file
to indicate that we installed a build from cache and to look for
that file being present to gate the new update logic.
Reviewed By: lnicco
Differential Revision: D21419122
fbshipit-source-id: 304670848add22531d88549d66f22c40ff255140
Summary: The oss jobs are causing problems to the developers now, lets disable them temporarily.
Reviewed By: StanislavGlebik
Differential Revision: D21400898
fbshipit-source-id: f7a3567056633d9eef98a8d05a37cd029c9e506c
Summary:
When the commit hash changed in fbsource, we would correctly decide
that we'd need to rebuild first-party projects but we would incorrectly skip
running the fetcher.update method. This would mean that we'd not perform the
shipit run and that our shipit tree would diverge from the source tree.
This commit resolves this by performing the fetcher.update but ignoring
the source update status in this case.
Reviewed By: xavierd
Differential Revision: D21364131
fbshipit-source-id: b4001e549c7d3f27aa4a21b19893c9bb7c0f6d1f
Summary:
We didn't do a great job of recognizing that we'd need to
build a project when one of its dependencies had changed: we relied
chiefly on the dependency hash for this and could fail to handle
changes in individual source files.
This commit helps to improve this situation by checking to see if
any installed files in the dependencies of a manifest are newer than
the most recent built time of a given manifest. If so, we'll trigger
a build. We try to be reasonably smart about deciding when to trigger
a cmake reconfigure if it looks like cmake files in the deps have
been changed.
Reviewed By: xavierd
Differential Revision: D21364132
fbshipit-source-id: 7534496e10d1f532aa9cf865900ace84a8785327
Summary:
Only run cmake reconfigure for .cmake, .cmake.in and CMakeLists.txt
files changes.
Previously we would reconfigure for any change to a file with a path that
matched `cmake` which could result in false positives in cases where
you may be iterating on .py or .c files in shared cmake directories.
This also reclassifies non-cmake files under fbcode_builder/CMake as source
files so that we run cmake for those; previously they would cause a
reconfigure and build, now they just cause a build.
Reviewed By: xavierd
Differential Revision: D21364133
fbshipit-source-id: a1231f657d6c6056b269656c677d3449d8715cf6
Summary:
Our linter really wants to include formatting changes unrelated
to my diff stack.
This is a formatting only change to avoid clouding my diffs; no functional
effect.
Reviewed By: xavierd
Differential Revision: D21364519
fbshipit-source-id: 7670dd4154e788f593f256aabdfdeef6d17aeec4
Summary:
Previously getdeps would remove the entire top-level `CMakeFiles` directory
from the build output when it wanted to invalidate the CMake cache. This
directory is used to keep all of the compiled object files for any libraries
or executables defined in the top-level CMakeLists.txt file. Blowing away
this directory forces all of these sources to be re-compiled, even if this was
not necessary. This is particularly problematic for folly, which compiles all
of its source files via rules in the top-level CMakeLists.txt target file.
I did have the code still blow away the CMake error and output logs in this
directory: in the past I have seen situations where CMake would not update
these files on new CMake runs if they already existed.
Reviewed By: wez
Differential Revision: D21360668
fbshipit-source-id: 6fcd1a8e371d756114fbab60d8636be8cd5f8978
Summary:
Update the manifest file for folly to indicate a dependency on lz4.
folly does not require lz4 be available, but it will use it if it is found at
configure time.
getdeps is unfortunately not strict about providing projects only with the
dependencies that they require at build time. This causes it to sometimes
make lz4 available to folly (if you are also building another project that
requires lz4), and sometimes not. This ends up causing changes in
folly-config.h depending on which projects you are building, forcing all of
the folly sources to be recompiled.
In the future we perhaps should update getdeps to consistently only pass in
include directories for dependencies actually listed in the manifest file.
However, specifying that folly depends on lz4 also works to mitigate this
particular issue for now, and it is also generally desirable to build folly
with lz4 support.
Reviewed By: wez
Differential Revision: D21359995
fbshipit-source-id: aaf61671b7750d6c47e3613c732d220b3311b5ba
Summary:
In the initial stages of the windows port we had
problems building rocksdb on windows, so we disabled it.
These days we're able to build it and detect it--we even
require it in the cmake code, but hadn't gotten around
to telling the rest of the code that we can use it.
This commit re-enables it in the build but leaves sqlite
as the default engine until we're able to perform some
benchmarking.
Rocksdb itself has some build issues on Windows; it doesn't
use cmake to locate dependencies, so even though we built
snappy it doesn't know how to find it without modifying the
source:
https://github.com/facebook/rocksdb/blob/master/thirdparty.inc#L4
For that reason, we disable the use of Snappy in the Windows build.
However, in the version of rocksdb that we were using, it would
default to trying to use Snappy even though it wasn't compiled in
and throw an exception.
I've upgraded to a newer version of rocksdb that will simply not
use compression if no compression was enabled at build time.
Given that we mostly store relatively small objects, I'm assuming
that the lack of compression is fine for now.
Reviewed By: xavierd
Differential Revision: D21319896
fbshipit-source-id: 2a2d06d4bd5382706e9220f9b4a2de99dc18311d
Summary:
Upgrade to the 1.10.0 release. This includes some new features like the
`GTEST_SKIP()` macro.
Reviewed By: genevievehelsel
Differential Revision: D21309360
fbshipit-source-id: 163db628fc99aaa786aeb207f35c7d6295cb5e25
Summary:
Update the generated `run_cmake.py` script to tell ctest to print the test
output on failure. Also pass in a `-j` flag to run tests in parallel by
default.
These flags are already passed in by default when running `getdeps.py test`;
this simply updates this developer utility script to do the same.
Reviewed By: wez
Differential Revision: D21307806
fbshipit-source-id: 42045b0f9362494042c79bc946a1004ff8ad98b6
Summary:
On Linux the debug info sections in projects downstream from folly and
thrift are huge due to a lot of C++ template usage.
We build in RelWithDebInfo mode as that is most useful when debugging
locally and because the build times are long, but most casual consumers
of the binaries via GH actions really don't care about debug info,
and this should save ~180MB of size in (for example) the watchman
executables.
Pull Request resolved: https://github.com/facebook/watchman/pull/809
Refs: https://github.com/facebook/watchman/issues/804
Test Plan:
Review the artifact size on the linux build in this PR (see
https://github.com/facebook/watchman/actions/runs/91688952) and confirm that it
is a bit smaller than 180MB; now under 3 MB.
Reviewed By: simpkins
Differential Revision: D21318302
Pulled By: wez
fbshipit-source-id: f78bc5e735dd78771e0604abae64c0a23cf9158d
Summary:
Rather than have a single main.yml file containing all off the different
builds, split that up so that we have one file per build environment
(linux, mac, windows).
This has a couple of advantages:
* It is quicker to see the status of just one of the platforms
* Artifact collection for one platform is not blocked pending completion
of the builds for all systems
* It's a little easier to understand what is happening for a single platform
To support having multiple files I've changed the output-file option to
be an output-dir.
I've included the rm of main.yml in this commit. Once this gets
imported back to the FB internal system I'll amend in an update to
the helper script that updates all of our opensource builds and run
and amend that.
Pull Request resolved: https://github.com/facebook/folly/pull/1360
Test Plan:
the GH action status on this PR should show three different
actions running, one for each platform.
I updated and ran
`fbcode/opensource/fbcode_builder/getdeps/facebook/update-all-github-actions.sh`
to regenerate all the actions files for FB.
Reviewed By: yfeldblum
Differential Revision: D21310991
Pulled By: wez
fbshipit-source-id: 604ef652c8f746781a4b410c6b996cdee4524e0d
Summary:
This commit resolves an issue with our zipapp executables
on Windows that meant that the only reliable way to start them was
to use the fully path to the executable.
The root cause is that the __wargv array is produced by parsing the
process command line into an array, and if you ran `watchman-wait -h`
__wargv[0] would have `watchman-wait` rather than the fully qualified
path to the executable that the zipapp plumbing requires.
The fix is to ask the system for the fully qualified path and ensure that
that gets set as both argv[0] AND argv[1].
Reviewed By: xavierd
Differential Revision: D21190350
fbshipit-source-id: eeb95084592d30a028a93b2b03877f8cc6c72729
Summary:
The environment changed since I tested D20740410 and now it appears
that we'll need to re-export a versioned variable in order for cmake
to detect boost on the GH actions hosts.
Pull Request resolved: https://github.com/facebook/folly/pull/1359
Test Plan:
the GH actions status of this diff.
The workflow was updated via:
```
python3 build/fbcode_builder/getdeps.py generate-github-actions folly --output-file .github/workflows/main.yml
```
Reviewed By: yfeldblum
Differential Revision: D21307640
Pulled By: yfeldblum
fbshipit-source-id: 1555cbcade822775379cd9054be37fdbc17b4d93
Summary:
From the outset, we wanted to be sure that getdeps was able
to source and build the dependencies so that we knew that we'd have
a repeatable build. This came at the cost of build times: having
to build boost on each CI run is a bit of a chore.
This commit adds three new elements to the manifest files:
* `rpms` - a list of RPM names that are all required to be present
in order to consider the dependency satisfied
* `debs` - like `rpms` above, but scoped to debian package names
* `preinstalled.env` - a list of environment variables that if they
are all set and non-empty will satisfy the dependency.
A new `--allow-system-packages` option to getdeps enables the new
logic that looks for system packages; it is off by default, but
enabled in the generated GitHub Actions workflows.
A new `install-system-deps` subcommand is provided that will attempt
to install the system packages needed to satisfy the build. This
typically needs to be run via sudo and is thus broken out separately
from the main getdeps build flow.
I made a pass over the manifest files and added package names that
satisfy the build on ubuntu-18 and fedora-31.
shri-khare: I renamed the `Python3.7.6` manifest to just `python` as
part of this change; the version of python that it pulls in through
the normal build is the same and I believe that an equal or newer
version of python3 is available in the GH actions builder.
The `preinstalled.env` is used only by the boost manifest: it references
the name of an environment variable that is set by the github
windows hosts and that points to a pre-built and pre-installed
copy of boost. Since there is no package manager that we can
easily query for this sort of thing, probing from the environment
seems like a reasonable and fast way to check for this. We
may need to evolve this over time to become more feature rich,
but this seems like a good starting point.
This commit has the potential to save 20 minutes of build time
from each public CI build just due to the boost dependency alone!
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: yfeldblum
Differential Revision: D20740410
fbshipit-source-id: 6c38019449c54465127656c3d18a6ff1f30adaea
Summary:
I noticed that copytree was taking forever and realized
that it wasn't issuing a prefetch call so I started looking in here;
this commit teaches getdeps how to recognize and EdenFS repo on
Windows but skips calling prefetch on Windows.
Currently the prefetch implementation triggers some very slow
processing in mercurial that is slower to start than just
enumerating the files in the opensource build.
It turned out that my original problem was just that my credentials
had expired and we weren't surfacing that error on Windows yet.
Reviewed By: simpkins
Differential Revision: D20755905
fbshipit-source-id: 8d3695cdd1f04199d1d409895482b8c706285d5f
Summary: The configerator structs are used in many top level functions in Mononoke and are required in order to build all the code on github
Reviewed By: ahornby
Differential Revision: D21130546
fbshipit-source-id: 7f17d92173f5ecf7c3406ae4202359a0db8df84a
Summary:
- Indicate that the EDEN_VERSION_OVERRIDE environment variable affects the
build.
- Exclude eden/config.py from the shipit path map, as this file is
auto-generated in open source builds.
- Include pexpect as a dependency, as this is needed for some of the
integration tests.
Reviewed By: genevievehelsel
Differential Revision: D21000163
fbshipit-source-id: 8eec378f66487229c995f637c4787eae530c7845
Summary:
This diff extracts the fbsource commit hash and the date of that
commit and maintains that in place of just the commit hash that
we were previously extracting.
This data is exported into the environment that we pass on to
builders so that it is available if they choose to use it.
In a follow on diff I'll use this to default a version number
in the watchman project.
Reviewed By: fanzeyi
Differential Revision: D20949666
fbshipit-source-id: dc12bffe5f0efc4297b15ba0140c4c67a23ab0fd
Summary:
In coverage collection mode a special module loader is prepended to
`sys.meta_path`. In very specific conditions this module loader can end up
returning a loader pointing to a _completely wrong module_. When importing
symbols from the wrong module errors occur.
The conditions to trigger the bug are:
- running in coverage collection mode, enabling the custom loader
- the test binary is a zip (e.g. par_style=fastzip)
- having a module name where the end part matches the name of a builtin Python
module
When these conditions were met, the special loader would return the builtin
Python module instead of the expected module. E.g. when loading a module like
`myapp.somemod.platform` in a zip style binary.
The custom loader first calls `imp.find_module()` to find the module it wants
to return a wrapped loader for. This fails for modules included in the test
binary, because the builtin `imp` module can not load from zips. This was the
trigger leading to the call to the buggy code.
When the initial call to `imp.find_module()` failed, the custom loader would
try a second call, asking the internal loader to this time try any path on
`sys.path`. For most module names this call would also fail, making the custom
loader return `None`, after which Python tries other loaders on `sys.path`.
However, when the final part of the module that was asked to load matches the
name of a Python builtin module, then the second call to the `imp` module would
succeed, returning a loader for the builtin module. E.g. `platform` when asking
for `myapp.somemod.platform`.
This diff fixes the issue by removing the broken second call to the internal
loader. This will never have worked, we just have not triggered or noticed
triggering the wrong loading before.
Differential Revision: D20798119
fbshipit-source-id: dffb54e308106a81af21b63c5ee64c6ca2041920
Summary:
I was testing vs2019 vs vs2017 and realized that
we weren't reconfiguring when the toolchain was changed;
this resolves that.
Reviewed By: genevievehelsel
Differential Revision: D20795118
fbshipit-source-id: db80f090367cacfcc6b53887b77cf949f9cef0e6
Summary:
On Windows the build artifacts cannot be easily run directly from the build
output directory without installing them. The `$PATH` environment variable
needs to be set correctly so that the runtime library dependencies can be
found.
This updates the builder code to emit a `run.ps1` wrapper script in the build
output directory that sets `$PATH` to support running build artifacts directly
from the build directory.
Additionally, this updates the CMake-specific builder to set properly when
running the tests with `ctest`.
Reviewed By: wez
Differential Revision: D20688290
fbshipit-source-id: 5d0f4d685692bca7e37370bd88309cf7634d87f0
Summary:
This still requires support from EdenFS in order to do much
of use, but it takes us a step closer:
* Pull in cpptoml when building with Eden support
* On Windows, when we locate the `.eden` directory, load and parse
the config file in order to determine the socket path
* If the EdenView constructor throws, treat it as a terminal error
so that we don't fallback to the regular filesystem watcher.
This is important because current EdenFS builds don't implement
the journal thrift API endpoint yet.
Reviewed By: pkaush
Differential Revision: D20504752
fbshipit-source-id: 48bbad49f1641698aa7d7b85674e3ddf4d4e617d
Summary:
We have a global `--install-prefix` argument that can be used to set
the prefix for all projects, but that is only suitable if you are running with
sufficient privileges to install each of the deps to that location during the
build. Cmake dependency resolution won't work from the build directory in that
situation; it can only see the final installed location and it will error out
if those files are not present, or link against the currently installed version
instead of the version we just built; not great!
This commit adds a project specific `--project-install-prefix` that can be used
on just the leaf project in a set of deps. That sidesteps the dependency
concern because only the last stage is built in that mode. This option
can technically be applied to an arbitrary set of projects, but in light
of the above, in practice it only makes sense to use it for the final
cmake project. Only the CMakeBuilder respects this option.
In the watchman repo, this commit adjusts the autogen.sh script to allow
specifying the installation prefix; it defaults to `/usr/local` as you
might expect.
refs: https://github.com/facebook/watchman/issues/760
Reviewed By: yfeldblum
Differential Revision: D20674439
fbshipit-source-id: 52799dbd47f3c295e2d6469ee2b74cedeaa20138
Summary:
Not sure if this is a transient issue, but the URL we've been using to obtain boost has started to return 403 errors.
Switch to a sourceforge download link instead.
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: chadaustin
Differential Revision: D20739351
fbshipit-source-id: 47483c675d59201a410c9d2a8f6db0f63ea5da69
Summary:
My recent change to ensure that we were using python3 to launch everything failed on windows: the GH actions environment has `python.exe` in the path and it is python version 3. There is no such thing as `python3` in that environment, although there is a `python2`.
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: chadaustin
Differential Revision: D20740411
fbshipit-source-id: 0e40590ccedc18e327ebb84901e2509588fdd0ff
Summary: This diff makes it possible to relay on the thrift structures from configerator in OSS.
Reviewed By: ahornby
Differential Revision: D20279457
fbshipit-source-id: 39df1c7a6f78b10a0a5bdeeebe476249ab674cc8
Summary:
This takes some pressure off both cpu and memory
on a laptop.
Reviewed By: pkaush
Differential Revision: D20562474
fbshipit-source-id: a058c71c47f25c3a2b3c1e34a0d0caf83e642021
Summary:
On linux we didn't account for the `--final-install-prefix` argument
which meant that the binaries would always be rewritten to be relative to
the destdir.
This commit fixes that.
Refs: https://github.com/facebook/watchman/issues/760
(this doesn't deal with the compiled in statedir being in the scratch path though)
Reviewed By: simpkins
Differential Revision: D20659749
fbshipit-source-id: 1e8e198a58361882249c33a67f54a7d97b849257
Summary:
The GH actions defaults for git prevent it from being able
to checkout the fbthrift repo due the length of the java related
files in the fbthrift repo.
This commit tells git to use long filenames and allows the checkout
to succeed.
Reviewed By: fanzeyi
Differential Revision: D20659750
fbshipit-source-id: 060b36d312d52a0769cf2f2dd9af60f7446f94a8
Summary:
Ensure that we are referencing python3 in the paths
that we generate for the github actions workflows, and remove
some shebangs that influence how our internal linters process
the python code.
Reviewed By: fanzeyi
Differential Revision: D20659747
fbshipit-source-id: 6f300f8e91edf7701bb27babc7b1418958cf0a10
Summary: Actually use vendored Rust crates when building with CMake. The `--offline` option make sure cargo do not go to the Internet for missing crates.
Reviewed By: simpkins
Differential Revision: D20542672
fbshipit-source-id: ab4af40150c6af8b531a75f503a4fa848b3914da
Summary:
If you have built the `eden/scm` subdirectory with `make local` in your
repository it will have left a bunch of build output artifacts in the source
tree. Update the getdeps shipit path map for eden to explicitly exclude many
of these build artifacts.
In particular:
- Exclude `eden/scm/build/` since this directory can contains a lot of files.
- Exclude all `*.pyc` files since there may be a reasonable number of these.
- Exclude several Cargo-related files since these will cause problems when
trying to build Rust code.
Reviewed By: chadaustin
Differential Revision: D20570720
fbshipit-source-id: a60dec4854ae470fdb58e9651fd8a3b910c76004
Summary:
OpenNSA has prebuilt SDK but kernel modules have to be built from sources. Add
a utility script to build and extend packaging script.
In future, we can consider folding this build into fbcode_builder itself.
Differential Revision: D20549883
fbshipit-source-id: f9475b7e0223e9f357117d7d7d27df8904fa1d73
Summary: Updated fmt version to be on par with buck build. It was causing inconsistencies.
Reviewed By: vitaut
Differential Revision: D20528011
fbshipit-source-id: d9e04ed2c28b839eaeff24120162c4db4732fa55
Summary:
This enables fetching blobs from the LFS server. For now, this is limited to
fetching them, but the protocol specify ways to also upload. That second part
will matter for commit cloud and when pushing code to the server.
One caveat to this code is that the LFS server is not mocked in tests, and thus
requests are done directly to the server. I chose very small blobs to limit the
disruption to the server, by setting a test specific user-agent, we should be
able to monitor traffic due to tests and potentially rate limit it.
Reviewed By: DurhamG
Differential Revision: D20445628
fbshipit-source-id: beb3acb3f69dd27b54f8df7ccb95b04192deca30
