Summary:
In the initial stages of the windows port we had
problems building rocksdb on windows, so we disabled it.
These days we're able to build it and detect it--we even
require it in the cmake code, but hadn't gotten around
to telling the rest of the code that we can use it.
This commit re-enables it in the build but leaves sqlite
as the default engine until we're able to perform some
benchmarking.
Rocksdb itself has some build issues on Windows; it doesn't
use cmake to locate dependencies, so even though we built
snappy it doesn't know how to find it without modifying the
source:
https://github.com/facebook/rocksdb/blob/master/thirdparty.inc#L4
For that reason, we disable the use of Snappy in the Windows build.
However, in the version of rocksdb that we were using, it would
default to trying to use Snappy even though it wasn't compiled in
and throw an exception.
I've upgraded to a newer version of rocksdb that will simply not
use compression if no compression was enabled at build time.
Given that we mostly store relatively small objects, I'm assuming
that the lack of compression is fine for now.
Reviewed By: xavierd
Differential Revision: D21319896
fbshipit-source-id: 2a2d06d4bd5382706e9220f9b4a2de99dc18311d
Summary:
Upgrade to the 1.10.0 release. This includes some new features like the
`GTEST_SKIP()` macro.
Reviewed By: genevievehelsel
Differential Revision: D21309360
fbshipit-source-id: 163db628fc99aaa786aeb207f35c7d6295cb5e25
Summary:
Update the generated `run_cmake.py` script to tell ctest to print the test
output on failure. Also pass in a `-j` flag to run tests in parallel by
default.
These flags are already passed in by default when running `getdeps.py test`;
this simply updates this developer utility script to do the same.
Reviewed By: wez
Differential Revision: D21307806
fbshipit-source-id: 42045b0f9362494042c79bc946a1004ff8ad98b6
Summary:
On Linux the debug info sections in projects downstream from folly and
thrift are huge due to a lot of C++ template usage.
We build in RelWithDebInfo mode as that is most useful when debugging
locally and because the build times are long, but most casual consumers
of the binaries via GH actions really don't care about debug info,
and this should save ~180MB of size in (for example) the watchman
executables.
Pull Request resolved: https://github.com/facebook/watchman/pull/809
Refs: https://github.com/facebook/watchman/issues/804
Test Plan:
Review the artifact size on the linux build in this PR (see
https://github.com/facebook/watchman/actions/runs/91688952) and confirm that it
is a bit smaller than 180MB; now under 3 MB.
Reviewed By: simpkins
Differential Revision: D21318302
Pulled By: wez
fbshipit-source-id: f78bc5e735dd78771e0604abae64c0a23cf9158d
Summary:
Rather than have a single main.yml file containing all off the different
builds, split that up so that we have one file per build environment
(linux, mac, windows).
This has a couple of advantages:
* It is quicker to see the status of just one of the platforms
* Artifact collection for one platform is not blocked pending completion
of the builds for all systems
* It's a little easier to understand what is happening for a single platform
To support having multiple files I've changed the output-file option to
be an output-dir.
I've included the rm of main.yml in this commit. Once this gets
imported back to the FB internal system I'll amend in an update to
the helper script that updates all of our opensource builds and run
and amend that.
Pull Request resolved: https://github.com/facebook/folly/pull/1360
Test Plan:
the GH action status on this PR should show three different
actions running, one for each platform.
I updated and ran
`fbcode/opensource/fbcode_builder/getdeps/facebook/update-all-github-actions.sh`
to regenerate all the actions files for FB.
Reviewed By: yfeldblum
Differential Revision: D21310991
Pulled By: wez
fbshipit-source-id: 604ef652c8f746781a4b410c6b996cdee4524e0d
Summary:
This commit resolves an issue with our zipapp executables
on Windows that meant that the only reliable way to start them was
to use the fully path to the executable.
The root cause is that the __wargv array is produced by parsing the
process command line into an array, and if you ran `watchman-wait -h`
__wargv[0] would have `watchman-wait` rather than the fully qualified
path to the executable that the zipapp plumbing requires.
The fix is to ask the system for the fully qualified path and ensure that
that gets set as both argv[0] AND argv[1].
Reviewed By: xavierd
Differential Revision: D21190350
fbshipit-source-id: eeb95084592d30a028a93b2b03877f8cc6c72729
Summary:
The environment changed since I tested D20740410 and now it appears
that we'll need to re-export a versioned variable in order for cmake
to detect boost on the GH actions hosts.
Pull Request resolved: https://github.com/facebook/folly/pull/1359
Test Plan:
the GH actions status of this diff.
The workflow was updated via:
```
python3 build/fbcode_builder/getdeps.py generate-github-actions folly --output-file .github/workflows/main.yml
```
Reviewed By: yfeldblum
Differential Revision: D21307640
Pulled By: yfeldblum
fbshipit-source-id: 1555cbcade822775379cd9054be37fdbc17b4d93
Summary:
From the outset, we wanted to be sure that getdeps was able
to source and build the dependencies so that we knew that we'd have
a repeatable build. This came at the cost of build times: having
to build boost on each CI run is a bit of a chore.
This commit adds three new elements to the manifest files:
* `rpms` - a list of RPM names that are all required to be present
in order to consider the dependency satisfied
* `debs` - like `rpms` above, but scoped to debian package names
* `preinstalled.env` - a list of environment variables that if they
are all set and non-empty will satisfy the dependency.
A new `--allow-system-packages` option to getdeps enables the new
logic that looks for system packages; it is off by default, but
enabled in the generated GitHub Actions workflows.
A new `install-system-deps` subcommand is provided that will attempt
to install the system packages needed to satisfy the build. This
typically needs to be run via sudo and is thus broken out separately
from the main getdeps build flow.
I made a pass over the manifest files and added package names that
satisfy the build on ubuntu-18 and fedora-31.
shri-khare: I renamed the `Python3.7.6` manifest to just `python` as
part of this change; the version of python that it pulls in through
the normal build is the same and I believe that an equal or newer
version of python3 is available in the GH actions builder.
The `preinstalled.env` is used only by the boost manifest: it references
the name of an environment variable that is set by the github
windows hosts and that points to a pre-built and pre-installed
copy of boost. Since there is no package manager that we can
easily query for this sort of thing, probing from the environment
seems like a reasonable and fast way to check for this. We
may need to evolve this over time to become more feature rich,
but this seems like a good starting point.
This commit has the potential to save 20 minutes of build time
from each public CI build just due to the boost dependency alone!
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: yfeldblum
Differential Revision: D20740410
fbshipit-source-id: 6c38019449c54465127656c3d18a6ff1f30adaea
Summary:
I noticed that copytree was taking forever and realized
that it wasn't issuing a prefetch call so I started looking in here;
this commit teaches getdeps how to recognize and EdenFS repo on
Windows but skips calling prefetch on Windows.
Currently the prefetch implementation triggers some very slow
processing in mercurial that is slower to start than just
enumerating the files in the opensource build.
It turned out that my original problem was just that my credentials
had expired and we weren't surfacing that error on Windows yet.
Reviewed By: simpkins
Differential Revision: D20755905
fbshipit-source-id: 8d3695cdd1f04199d1d409895482b8c706285d5f
Summary: The configerator structs are used in many top level functions in Mononoke and are required in order to build all the code on github
Reviewed By: ahornby
Differential Revision: D21130546
fbshipit-source-id: 7f17d92173f5ecf7c3406ae4202359a0db8df84a
Summary:
- Indicate that the EDEN_VERSION_OVERRIDE environment variable affects the
build.
- Exclude eden/config.py from the shipit path map, as this file is
auto-generated in open source builds.
- Include pexpect as a dependency, as this is needed for some of the
integration tests.
Reviewed By: genevievehelsel
Differential Revision: D21000163
fbshipit-source-id: 8eec378f66487229c995f637c4787eae530c7845
Summary:
This diff extracts the fbsource commit hash and the date of that
commit and maintains that in place of just the commit hash that
we were previously extracting.
This data is exported into the environment that we pass on to
builders so that it is available if they choose to use it.
In a follow on diff I'll use this to default a version number
in the watchman project.
Reviewed By: fanzeyi
Differential Revision: D20949666
fbshipit-source-id: dc12bffe5f0efc4297b15ba0140c4c67a23ab0fd
Summary:
In coverage collection mode a special module loader is prepended to
`sys.meta_path`. In very specific conditions this module loader can end up
returning a loader pointing to a _completely wrong module_. When importing
symbols from the wrong module errors occur.
The conditions to trigger the bug are:
- running in coverage collection mode, enabling the custom loader
- the test binary is a zip (e.g. par_style=fastzip)
- having a module name where the end part matches the name of a builtin Python
module
When these conditions were met, the special loader would return the builtin
Python module instead of the expected module. E.g. when loading a module like
`myapp.somemod.platform` in a zip style binary.
The custom loader first calls `imp.find_module()` to find the module it wants
to return a wrapped loader for. This fails for modules included in the test
binary, because the builtin `imp` module can not load from zips. This was the
trigger leading to the call to the buggy code.
When the initial call to `imp.find_module()` failed, the custom loader would
try a second call, asking the internal loader to this time try any path on
`sys.path`. For most module names this call would also fail, making the custom
loader return `None`, after which Python tries other loaders on `sys.path`.
However, when the final part of the module that was asked to load matches the
name of a Python builtin module, then the second call to the `imp` module would
succeed, returning a loader for the builtin module. E.g. `platform` when asking
for `myapp.somemod.platform`.
This diff fixes the issue by removing the broken second call to the internal
loader. This will never have worked, we just have not triggered or noticed
triggering the wrong loading before.
Differential Revision: D20798119
fbshipit-source-id: dffb54e308106a81af21b63c5ee64c6ca2041920
Summary:
I was testing vs2019 vs vs2017 and realized that
we weren't reconfiguring when the toolchain was changed;
this resolves that.
Reviewed By: genevievehelsel
Differential Revision: D20795118
fbshipit-source-id: db80f090367cacfcc6b53887b77cf949f9cef0e6
Summary:
On Windows the build artifacts cannot be easily run directly from the build
output directory without installing them. The `$PATH` environment variable
needs to be set correctly so that the runtime library dependencies can be
found.
This updates the builder code to emit a `run.ps1` wrapper script in the build
output directory that sets `$PATH` to support running build artifacts directly
from the build directory.
Additionally, this updates the CMake-specific builder to set properly when
running the tests with `ctest`.
Reviewed By: wez
Differential Revision: D20688290
fbshipit-source-id: 5d0f4d685692bca7e37370bd88309cf7634d87f0
Summary:
This still requires support from EdenFS in order to do much
of use, but it takes us a step closer:
* Pull in cpptoml when building with Eden support
* On Windows, when we locate the `.eden` directory, load and parse
the config file in order to determine the socket path
* If the EdenView constructor throws, treat it as a terminal error
so that we don't fallback to the regular filesystem watcher.
This is important because current EdenFS builds don't implement
the journal thrift API endpoint yet.
Reviewed By: pkaush
Differential Revision: D20504752
fbshipit-source-id: 48bbad49f1641698aa7d7b85674e3ddf4d4e617d
Summary:
We have a global `--install-prefix` argument that can be used to set
the prefix for all projects, but that is only suitable if you are running with
sufficient privileges to install each of the deps to that location during the
build. Cmake dependency resolution won't work from the build directory in that
situation; it can only see the final installed location and it will error out
if those files are not present, or link against the currently installed version
instead of the version we just built; not great!
This commit adds a project specific `--project-install-prefix` that can be used
on just the leaf project in a set of deps. That sidesteps the dependency
concern because only the last stage is built in that mode. This option
can technically be applied to an arbitrary set of projects, but in light
of the above, in practice it only makes sense to use it for the final
cmake project. Only the CMakeBuilder respects this option.
In the watchman repo, this commit adjusts the autogen.sh script to allow
specifying the installation prefix; it defaults to `/usr/local` as you
might expect.
refs: https://github.com/facebook/watchman/issues/760
Reviewed By: yfeldblum
Differential Revision: D20674439
fbshipit-source-id: 52799dbd47f3c295e2d6469ee2b74cedeaa20138
Summary:
Not sure if this is a transient issue, but the URL we've been using to obtain boost has started to return 403 errors.
Switch to a sourceforge download link instead.
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: chadaustin
Differential Revision: D20739351
fbshipit-source-id: 47483c675d59201a410c9d2a8f6db0f63ea5da69
Summary:
My recent change to ensure that we were using python3 to launch everything failed on windows: the GH actions environment has `python.exe` in the path and it is python version 3. There is no such thing as `python3` in that environment, although there is a `python2`.
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: chadaustin
Differential Revision: D20740411
fbshipit-source-id: 0e40590ccedc18e327ebb84901e2509588fdd0ff
Summary: This diff makes it possible to relay on the thrift structures from configerator in OSS.
Reviewed By: ahornby
Differential Revision: D20279457
fbshipit-source-id: 39df1c7a6f78b10a0a5bdeeebe476249ab674cc8
Summary:
This takes some pressure off both cpu and memory
on a laptop.
Reviewed By: pkaush
Differential Revision: D20562474
fbshipit-source-id: a058c71c47f25c3a2b3c1e34a0d0caf83e642021
Summary:
On linux we didn't account for the `--final-install-prefix` argument
which meant that the binaries would always be rewritten to be relative to
the destdir.
This commit fixes that.
Refs: https://github.com/facebook/watchman/issues/760
(this doesn't deal with the compiled in statedir being in the scratch path though)
Reviewed By: simpkins
Differential Revision: D20659749
fbshipit-source-id: 1e8e198a58361882249c33a67f54a7d97b849257
Summary:
The GH actions defaults for git prevent it from being able
to checkout the fbthrift repo due the length of the java related
files in the fbthrift repo.
This commit tells git to use long filenames and allows the checkout
to succeed.
Reviewed By: fanzeyi
Differential Revision: D20659750
fbshipit-source-id: 060b36d312d52a0769cf2f2dd9af60f7446f94a8
Summary:
Ensure that we are referencing python3 in the paths
that we generate for the github actions workflows, and remove
some shebangs that influence how our internal linters process
the python code.
Reviewed By: fanzeyi
Differential Revision: D20659747
fbshipit-source-id: 6f300f8e91edf7701bb27babc7b1418958cf0a10
Summary: Actually use vendored Rust crates when building with CMake. The `--offline` option make sure cargo do not go to the Internet for missing crates.
Reviewed By: simpkins
Differential Revision: D20542672
fbshipit-source-id: ab4af40150c6af8b531a75f503a4fa848b3914da
Summary:
If you have built the `eden/scm` subdirectory with `make local` in your
repository it will have left a bunch of build output artifacts in the source
tree. Update the getdeps shipit path map for eden to explicitly exclude many
of these build artifacts.
In particular:
- Exclude `eden/scm/build/` since this directory can contains a lot of files.
- Exclude all `*.pyc` files since there may be a reasonable number of these.
- Exclude several Cargo-related files since these will cause problems when
trying to build Rust code.
Reviewed By: chadaustin
Differential Revision: D20570720
fbshipit-source-id: a60dec4854ae470fdb58e9651fd8a3b910c76004
Summary:
OpenNSA has prebuilt SDK but kernel modules have to be built from sources. Add
a utility script to build and extend packaging script.
In future, we can consider folding this build into fbcode_builder itself.
Differential Revision: D20549883
fbshipit-source-id: f9475b7e0223e9f357117d7d7d27df8904fa1d73
Summary: Updated fmt version to be on par with buck build. It was causing inconsistencies.
Reviewed By: vitaut
Differential Revision: D20528011
fbshipit-source-id: d9e04ed2c28b839eaeff24120162c4db4732fa55
Summary:
This enables fetching blobs from the LFS server. For now, this is limited to
fetching them, but the protocol specify ways to also upload. That second part
will matter for commit cloud and when pushing code to the server.
One caveat to this code is that the LFS server is not mocked in tests, and thus
requests are done directly to the server. I chose very small blobs to limit the
disruption to the server, by setting a test specific user-agent, we should be
able to monitor traffic due to tests and potentially rate limit it.
Reviewed By: DurhamG
Differential Revision: D20445628
fbshipit-source-id: beb3acb3f69dd27b54f8df7ccb95b04192deca30
Summary:
Next OpenNSA release with increased stack size (needed for FBOSS) is available.
Start linking with it.
Differential Revision: D20062997
fbshipit-source-id: 9938270c322087ac3990861aa1ddd3b9ea1148ac
Summary:
Ideally we'd be using mkscratch, but this still isn't shipped
to our Windows systems.
Pick a path that is more friendly to our corp windows environment by default.
Reviewed By: pkaush
Differential Revision: D20342277
fbshipit-source-id: c85bccee6701adc03b26c92ba217b18bd684257a
Summary:
Replace the methods to get CPU and memory usage statistics:
- For the memory: use `VmRSS` of `/proc/[pid]/status`: http://man7.org/linux/man-pages/man5/proc.5.html
- For the CPU%: calculate the process is occupied how much percentage of the CPU time, use `getrusage()`: http://man7.org/linux/man-pages/man2/getrusage.2.html
- Implemented like the sigar: https://our.intern.facebook.com/intern/diffusion/FBS/browse/master/third-party/sigar/src/sigar.c?commit=4f945812675131ea64cb3d143350b1414f34a351&lines=111-169
- Formula:
- CPU% = `process used time` during the period / `time period` * 100
- `time period` = current query timestamp - last query timestamp
- `process used time` = current `process total time` - last query `process total time`
- `process total time` = CPU time used in user mode + CPU time used in system mode // get from the API `ru_utime` and `ru_stime`
Remove the `fbzmq::ResourceMonitor` and `sigar`:
- Change and rename the UT
- `ResourceMonitorTest.cpp` -> `SystemMetricsTest.cpp`
- `ResourceMonitor` -> `SystemMetricsTest` in `openr/tests/OpenrSystemTest.cpp`
- Remove `ResourceMonitor` code and dependency for `Watchdog` and `ZmqMonitor`
- Remove `sigar` dependency used in building
Reviewed By: saifhhasan
Differential Revision: D20049944
fbshipit-source-id: 00b90c8558dc5f0fb18cc31a09b9666a47b096fe
Summary:
This diff adds fb303 dependency to Open/R. fb303 is FB's open-sourced project
which provides basic infrastructure for thrift service e.g. base functions,
counters supports etc.
Using this in Open/R will allow both internal & external users leverage counters
via thrift APIs
Reviewed By: xiangxu1121
Differential Revision: D20134243
fbshipit-source-id: 23782ec99dc749dd27dc19f2b2b023f1c229608b
Summary:
The SAI diag shell implementation requires Python3.7.6.
Previous diff in the stack added Python3.7.6 manifest.
Make use of it to build diag shell and link with Python lib.
Differential Revision: D20064315
fbshipit-source-id: c855f8a9f554066e8150ce1d867828a84161961b
Summary:
As titled.
FBOSS depends on Python3.7.6, subsequent diff in the stack adds Python3.7.6
depedency for FBOSS.
Differential Revision: D20064316
fbshipit-source-id: 27b328dc25e326f3927ea6cb003fb7bb45732d61
Summary:
We need to add a separate manifest for SAI 1.4 and build that too, but that is
for later diffs, for now, we need 1.5, so change it.
Differential Revision: D20019016
fbshipit-source-id: 8630340ac7c8b7d94e6b768a2efa96266b2cc3e0
Summary:
The Buck build of RocksDB supports LZ4, so if an Eden instance's
object cache ends up compressed with LZ4 compression, the opensource
getdeps build cannot open it. Enable LZ4 in the open source getdeps
build of rocksdb.
Reviewed By: pkaush
Differential Revision: D19961878
fbshipit-source-id: 9a4f37e12bd2b40a0c5dcf2dd1684876886fae26
Summary: There are downstream oss builds showing build failure caused by introduction of metadata. We want to make them proceed to build so we won't get errors in CI.
Reviewed By: iahs
Differential Revision: D19900007
fbshipit-source-id: 4201448f7980b53e407fd2bc1c74ed4ffa8e18c1
Summary:
Take the README.md from
7ead0e29e4/README.md
and apply it on Eden repo.
Re-add the Cargo.toml file that declares Cargo workspace.
Re-add fbcode_builder/getdeps manifest for Mononoke
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/13
Test Plan:
./build/fbcode_builder/getdeps.py build mononoke
./build/fbcode_builder/getdeps.py test mononoke
Reviewed By: ahornby
Differential Revision: D19833059
Pulled By: lukaspiatkowski
fbshipit-source-id: fb37e13306c0b9969a7c4e52b05e1a66a577022f
Summary:
Instead of fetching a tarball for the vendored Rust crates, let's just use
what's present in third-party/rust.
Reviewed By: wez, fanzeyi
Differential Revision: D19770580
fbshipit-source-id: ca6615a5f5c672d9a5527ad47feab6bdd33a39db
Summary: Mononoke is becoming part of Eden repository, remove it's getdeps manifest until we finalize the move.
Reviewed By: krallin
Differential Revision: D19769090
fbshipit-source-id: 9b471686728e9ff28317f9157f5f8a1834c5f5e4
