Summary:
Since FilteredFS uses an entirely separate `eden clone` command, none of the existing Hg tests run using a FilteredBackingStore-backed repo. To ensure we have proper test coverage for FilteredFS, we should add a FilteredFS mixin type that runs all existing tests with the new FilteredBackingStore setup.
Introducing these mixins revealed a number of broken tests (some legit issues with FilteredFS, others are just incompatibilities with new FilteredBackingStore Root/Object Ids). We will disable all of these broken tests and incrementally fix them in the next diffs in this stack.
Note: A small number of tests are already broken/flakey for non-FilteredFS tests. They are "ignored" on other diff signals since they are marked flakey/broken in TestX, but the FilteredHg counterpart won't be marked accordingly (due to no run history). I'm not planning to fix those existing flakey/broken tests just yet.
Reviewed By: kmancini
Differential Revision: D51287692
fbshipit-source-id: 8135cb9e5e9dba08be48a4814dfc3405122127b8
Summary:
X-link: https://github.com/facebookincubator/velox/pull/7542
ties all of the pieces together. The bulk of the net-new logic is in `OverlayFile`, with the LMDB stuff being ported from other implementations or just delegating calls to other classes.
Reviewed By: kmancini
Differential Revision: D46914322
fbshipit-source-id: 3434b71c92ece6b94a3c08828df286b04152d50f
Summary:
On Linux gcc does not have the 'no-nullability-completeness' warning flag. This change removes that flag from Linux builds to eliminate warning messages like these:
```
At global scope:
cc1plus: warning: unrecognized command line option ‘-Wno-nullability-completeness’
```
Reviewed By: kmancini
Differential Revision: D51220414
fbshipit-source-id: 377a781872d50a1f183edc6635fdcc6a877a7c53
Summary: When looking at the getdeps builds I noticed some simple warnings.
Reviewed By: xavierd
Differential Revision: D47490053
fbshipit-source-id: 4d720aded376d34a63e7e3c39d50db682f0bdc17
Summary:
Win32 provides a few defines that make safely using the Windows API
easier. STRICT is one of them. Move it into the build system alongside
NOMINMAX and WIN32_LEAN_AND_MEAN.
Reviewed By: xavierd
Differential Revision: D45823992
fbshipit-source-id: c6849f039198072faaef88c1792a51c60b1c4b50
Summary: We can limit the -DWIN32_LEAN_AND_MEAN flag to Windows.
Reviewed By: xavierd
Differential Revision: D45823417
fbshipit-source-id: 47fab430363b4b4e7f256002e5a0043494154d06
Summary:
Adding blake3 for the upcoming migration to it.
bypass-github-export-checks
Reviewed By: chadaustin
Differential Revision: D43567300
fbshipit-source-id: a0c81b42309cab65cf68be9be0530c86e0e91e95
Summary:
We somehow have two different RustStaticLibrary.cmake in different places (one in eden repo and the other one in the shared opensource builder).
This diff merges them and switches Eden into using the shared CMake function (for the features option).
This diff also adds the features option for rust_executable funciton, which will be used in the next diff.
Reviewed By: kmancini
Differential Revision: D39038491
fbshipit-source-id: 99d61a1d5450010b345107a9ec5c761b62004aa6
Summary:
libatomic should not be linked on macOS, and it looks like RocksDB has
fixed the way it links to lz4 in its CMake build.
Reviewed By: fanzeyi
Differential Revision: D36726495
fbshipit-source-id: f2bce30d54372da0a8638a8a4a33e74ad62a4bd9
Summary:
The reason we can't use -latomic as a linker flag is because it must show up on the link line after libraries that depend on it. Instead, have every executable and shared library in the EdenFS CMake build pull in libatomic.
This fixes the EdenFS CMake build on some systems.
Reviewed By: xavierd
Differential Revision: D36450490
fbshipit-source-id: 2fe7dc4497554dc8e009cb3c744695684ff08f6e
Summary:
Our systemd code is unused, and is causing a bunch of tests to fail. In the
case where systemd is needed again in the future, its shape will be
significantly different than what we have today, so there even less need to
keep this version around.
Reviewed By: fanzeyi
Differential Revision: D35829902
fbshipit-source-id: 6013f5fe7c3debdcfbb9929d2748819a96403fe7
Summary: changes `ImportPriorityKind` and members in `ObjectFetchContext` to `uint8_t`, these should be minimized for use in `HgImportTraceEvent`
Reviewed By: xavierd
Differential Revision: D35269554
fbshipit-source-id: f36752b41e653338704316a8a75e2bbc72317e4a
Summary:
Stack context:
To get a build setup for edencommon I picked an eden library to move over
to edencommon.
I picked process name cache. I would like to have this library in watchman
so that I can log spawning process command lines as well as client command
lines.
Diff:
process name cache depends on Synchronized so I am moving that over in this diff.
Reviewed By: chadaustin
Differential Revision: D34218021
fbshipit-source-id: 528e7b4b12476b33a294a342e6e4ee13c92c7361
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/110
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/27
Make it so that changes to rust-shed or other common rust source are used locally vendored, so they don't need to be pushed to github before they are visible in a build.
There was already some support for cargo vendoring in getdeps, but it was limited to dependencies between manifests built with cargo builder. This wasn't enough to build something like eden (cmake is main entry point, with later calls cargo) or eden_scm (make is main entry point, with later calls to cargo), so this diff adds a cargo prepare step for getdeps other primary build systems.
The cargo vendoring is done by using a cargo config file to point to the source files used by getdeps. It has two modes:
1. per crate, existing mode which is already automatic for cargo to cargo manifest dependencies. To use it for a non cargo build manifest, add crate.pathmap
2. per git url, existing mode which was only use for crates.io third-party crates, now can be enabled by setting cargo.cargo_config_file
Reviewed By: yancouto
Differential Revision: D33895469
fbshipit-source-id: 7b13c0b679532492a336ce217de875c25fe1be90
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
The headers are included in the Windows devkits, thus we do not need to include
an outdated version in the repository.
Reviewed By: chadaustin
Differential Revision: D34092871
fbshipit-source-id: f8ea381442edf5fa1f143e756d3ccac986035fc8
Summary:
With Facebook having been renamed Meta Platforms, we need to change the license
headers.
Reviewed By: fanzeyi
Differential Revision: D33407812
fbshipit-source-id: b11bfbbf13a48873f0cea75f212cc7b07a68fb2e
Summary:
There is a race in here. We read the config, then mutate it, then put the new
config in place. However, if something else comes and mutates the config
between the time where we read it and write it again, we'll clobber its
changes.
Concretely, if you're running lots of `eden clone` or `eden rm` concurrently,
then the current mechanism will sometimes "forget" some directories or add back
directories you removed, which will e.g. break `eden list` (and cause those
commands to sometimes fail as well).
This diff fixes that by wrapping the critical section with a file lock.
Reviewed By: chadaustin, xavierd
Differential Revision: D32251727
fbshipit-source-id: 9eee57c7a824286bb4f0e7911e5ccd9d769c6daa
Summary:
Put code using the usage service behind an `EDEN_HAVE_USAGE_SERVICE` macro.
Previously the C++ code was simply guarded by a `__linux__` check, and the
CMake code did not have a guard at all. This caused builds from the GitHub
repository to fail on Linux, since the code attempted to use the usage service
client which was not available.
Reviewed By: xavierd
Differential Revision: D30797846
fbshipit-source-id: 32a0905d0e1d594c3cfb04a466aea456d0bd6ca1
Summary:
Remove some `EDEN_HAVE_*` config definitions that are no longer referenced
anywhere.
Reviewed By: xavierd
Differential Revision: D30797847
fbshipit-source-id: e2d9e0049cfd984c0814f1ac61ffddb1185e9904
Summary:
Reenables dynamicconfig loading with eden backingstore. Previously it
broke edenfs-windows-release, but we believe the
opensource/fbcode_builder/manifests/eden tweak has fixed it.
Reviewed By: xavierd
Differential Revision: D29561192
fbshipit-source-id: 775dd21d177f3baa09b0192e7d3f7231008c3766
Summary: This is breaking the Windows release, reverting.
Reviewed By: fanzeyi
Differential Revision: D29339787
fbshipit-source-id: 22d8ff5db5619194e4597754dc37343cf0bc3286
Summary:
From what I can see, this was added when EdenFS had a Mononoke store, which is
now long gone, thus we should be able to remove the Curl dependency altogether.
Reviewed By: fanzeyi
Differential Revision: D28037816
fbshipit-source-id: 834f7db64bab5dda1748ad2f033c27a2854b0ba4
Summary:
Currently we have limited test coverage of the NFS code. Let's start running
our integration tests on NFS mounts. We already duplicate tests to run them on
both Git and Hg repos using a python decorator. We can update this decorator to
run a copy of tests on an nfs mount.
This covers most of the tests, but a few tests do not use this decorator. See next
change.
Note some tests are currently broken, so I am using the same skip list functionality
we use for windows so we use a uniform framework.
Reviewed By: xavierd
Differential Revision: D27874662
fbshipit-source-id: c7d425830b691e395b5228d0e0f797f67987b4ec
Summary:
On macOS, the compiler shows tons of warnings due to our use of pointers where
nullability attributes aren't set, and the compiler complains about it. This
makes finding legitimate warnings/errors almost impossible, thus for now let's
disable these.
Reviewed By: fanzeyi
Differential Revision: D27862116
fbshipit-source-id: 8ea2c27a0cfd6cbd484fca6be5f15c772d9f4589
Summary:
On Windows, we've stored the configs in C:/ProgramData/Facebook/eden for a
while now, let's make sure the compiled in values reflect that.
Reviewed By: genevievehelsel
Differential Revision: D24864358
fbshipit-source-id: 2ff27554b36713d31a89b4d1802e6d7c8707cde5
Summary:
We use Re2 in D22877942 for parsing multiple path prefix data fetch logging,
this introduces the dependency for eden's opensource builds.
Reviewed By: chadaustin
Differential Revision: D23431175
fbshipit-source-id: 44b399e92cb89ba1403295ecd10bc8f8d769b02c
Summary:
A future diff will unify all config loading into configparser::hg, but
to do so we need dynamicconfig to live in configparser, so it can load
dynamicconfigs. Let's move everything in.
Reviewed By: quark-zju
Differential Revision: D22587237
fbshipit-source-id: 5613094175b6e1597aa113ee3e6d92ce7ec79f6d
Summary:
I think LZ4 was disabled accidentally in our rocksdb getdeps build in
D21319896. Enable it again on macOS and Linux, because otherwise this
breaks people with EdenFS mounts containing LZ4-compressed proxy
hashes.
Reviewed By: xavierd
Differential Revision: D21990356
fbshipit-source-id: b9166c2992ae51f09de3fa9a4f114143aa008f43
Summary:
CMake description of `JOB_POOL` is:
"A pool is a named integer property and defines the maximum number of
concurrent jobs which can be started by a rule assigned to the pool."
Not very clear by it looks like putting all the Rust jobs in it prevent CMake
from spawning concurrent jobs when these are running. This should help in
reducing the strain on the system while compiling, while not increasing
compile time.
Reviewed By: wez
Differential Revision: D21595135
fbshipit-source-id: e718c92a237274a9edbc35417644a46bdfde5617
Summary:
Previously, the Windows build was litered with warnings of the form (typo included):
warning C4250: 'C1': inherits 'C2::C2::method' via dominance
note: see declaration of 'C2::method'
Microsoft doesn't offer any recommendation, and the internet suggest that the
right `using` should silence it. That's unfortunately not the case, adding:
using C2::method
In `C1` doesn't do anything, and the compiler still complains :(.
Since the warning appears to be non-actionable, and looks more like a
"notice" than a warning, let's just silence it.
Reviewed By: wez
Differential Revision: D21395095
fbshipit-source-id: ae661b3ed61303e6361b8a15d9e7c6b9627ea8c1
Summary:
When using our vendored set of crates, cmake doesn't
have any dependency information to use to invalidate the Cargo.lock
file when we update crate versions. In addition, since we're
vendoring from a local directory, cargo itself doesn't seem to
want to re-assess the dependencies in that same situation, leading
to confusing error messages like this when we want to build rust
targets:
```
error: failed to select a version for the requirement `anyhow = "= 1.0.26"`
candidate versions found which didn't match: 1.0.28
```
This commit addresses this issue by removing the `Cargo.lock` that
may be alongside the `Cargo.toml` prior to invoking `cargo`.
`cargo` is pretty quick at recomputing the deps so this has
neglible overhead.
Reviewed By: xavierd
Differential Revision: D21394363
fbshipit-source-id: 547db2e2395a47aed77d9597e659eb2d96e274dd
Summary:
Previously we only included `basic_test.py` and `hg/status_test.py` in the
integration tests during CMake-based builds. This updates the code to now
include all of the test files, with just a few exclusions based on platform
type and what dependencies were available at build time.
Reviewed By: wez
Differential Revision: D21239912
fbshipit-source-id: b8826d249a6323ac3bcc555c9ceba54a4cbcfde9
Summary:
In the initial stages of the windows port we had
problems building rocksdb on windows, so we disabled it.
These days we're able to build it and detect it--we even
require it in the cmake code, but hadn't gotten around
to telling the rest of the code that we can use it.
This commit re-enables it in the build but leaves sqlite
as the default engine until we're able to perform some
benchmarking.
Rocksdb itself has some build issues on Windows; it doesn't
use cmake to locate dependencies, so even though we built
snappy it doesn't know how to find it without modifying the
source:
https://github.com/facebook/rocksdb/blob/master/thirdparty.inc#L4
For that reason, we disable the use of Snappy in the Windows build.
However, in the version of rocksdb that we were using, it would
default to trying to use Snappy even though it wasn't compiled in
and throw an exception.
I've upgraded to a newer version of rocksdb that will simply not
use compression if no compression was enabled at build time.
Given that we mostly store relatively small objects, I'm assuming
that the lack of compression is fine for now.
Reviewed By: xavierd
Differential Revision: D21319896
fbshipit-source-id: 2a2d06d4bd5382706e9220f9b4a2de99dc18311d
Summary:
This updates the top-level CMakeLists.txt file to compute package version
information, and expose this to C++ code in `eden-config.h`, and to Python
code in a new `eden/config.py` module.
Previously we exposed an `EDEN_VERSION` macro for the C++ code in
`eden-config.h`, but this was not initialized or used anywhere. Now the
top-level CMakeLists.txt file computes appropriate version information and
exposes the package name, version, release, commit ID, and build time in these
configuration files.
The version selection logic in CMakeLists.txt based largely on the code that
wez wrote for watchman in D20636833.
Reviewed By: wez
Differential Revision: D21000164
fbshipit-source-id: db1a1035f1eefec058bbad558d35e113005e454e
Summary: Actually use vendored Rust crates when building with CMake. The `--offline` option make sure cargo do not go to the Internet for missing crates.
Reviewed By: simpkins
Differential Revision: D20542672
fbshipit-source-id: ab4af40150c6af8b531a75f503a4fa848b3914da
Summary: All the crates are present in third-party/rust, let's use it in the OSS build instead of fetching a tarball of all the crates.
Reviewed By: fanzeyi
Differential Revision: D19770783
fbshipit-source-id: f0d74bb0807be207d599d4868f907d38099c7f5b
Summary:
We've had a small proportion of our users run into problems
with hdiutil and diskimages-helper, where those components get into
an unhappy state and effectively block operating on the redirections.
This diff introduces a new utility that is intended to replace the
use of disk image files with APFS volumes that we mount in the
appropriate places.
The intention is that we will teach `eden redirect` to use this tool
when available, rather than disk images.
macOS's security model is weird: it is perfectly valid for a non-privileged
user to create and delete APFS volumes in the APFS storage container,
but root privs are required to mount it into the VFS.
The intent is that we deploy this utility setuid root to minimize
the fan out--this way we won't need to teach the priv helper about
this kind of redirection.
There are a couple of subcommands demonstrated in the test plan.
Reviewed By: chadaustin
Differential Revision: D19323850
fbshipit-source-id: 35556f841e49e5c4b77679b756af9093222f4500
Summary:
The Rust vendored crates script also depends on another script and the
lfs-pointers file which specifies the ID to use for the download.
Update the RustStaticLibrary.cmake file to make sure that the vendored crates
are re-downloaded if either of these files change.
Reviewed By: wez
Differential Revision: D18846757
fbshipit-source-id: dd67d8d954a048501f0bdaddbd78147d39a1da5f
Summary:
Now that the fb-mercurial sources are available in the Eden repository, update
our CMake build files to always build them. This moves the build logic from
the centralized `FBMercurialFeatures.cmake` file into `CMakeLists.txt` files
in the appropriate subdirectories.
Reviewed By: chadaustin
Differential Revision: D18588011
fbshipit-source-id: ded9decde5c2ec766aae0bb0f4f5b021d1044a98
Summary:
Remove the standalone fb-mercurial-rust target that was an internal-only
dependency for the Eden build. This build step is now done entirely in the
Eden build.
Reviewed By: fanzeyi
Differential Revision: D18623943
fbshipit-source-id: c62a1155ddd1c0a6b2270c472176ba25194c6145
Summary:
Update the CMake build to always build the Rust datapack libraries, even on
Windows. This allows us to completely eliminate the `EDEN_HAVE_RUST_DATAPACK`
checks.
Note that I did leave the `EDEN_HAVE_RUST_DATAPACK` macro in place, as we
still do not build the Rust datapack code on Mac when building with Buck.
Reviewed By: fanzeyi
Differential Revision: D18588008
fbshipit-source-id: 1a4c9ceec5372d0e6a7313d2eb87edabd1e60a96
Summary:
Update Eden's top-level CMakeLists.txt file to build the Rust datapack
libraries. Previously these were built by invoking CMake separately inside th
`eden/scm` subdirectory. Now that the code has been combined into a single
location we can use a single CMake invocation to drive the build of both these
components.
The old code did not build the Rust datapack code on Windows, and this diff
does not change that behavior. I'm not aware of any reason to skip building
this on Windows, so I plan to enable building this code on Windows in a
subsequent diff.
Reviewed By: pkaush
Differential Revision: D18588006
fbshipit-source-id: 20f4f0ea9fef8595a9dd35a21115952b2808c824
