Summary:
The kernel has some legacy logic to attempt to clear the setuid and
setgid mode bits upon writes, but it is racy and disabled by default
in libfuse3. Now that we don't support setgid and setuid bits at all,
opt out of the legacy kernel behavior.
Reviewed By: wez
Differential Revision: D21334075
fbshipit-source-id: bdef12c1958a5e9bd2649c2bcb54975b0b4e78d6
Summary:
We'll be adding a bunch of Facebook specific configuration and values
here. Let's move it to someplace not open source.
Reviewed By: quark-zju
Differential Revision: D21241038
fbshipit-source-id: 2ac9cdce40b1b46f15f171d9d1f6b6692dcd29bf
Summary:
Implements an ensure_location_supersets function who's goal is to
verify that a given config location specifies the exact same configs as a given
set of other locations. Any inconsistencies are removed from the config and
reported to the caller.
This will be used to ensure our dynamic configs match our existing rc file
configs exactly, before we delete the file configs.
Reviewed By: quark-zju
Differential Revision: D21240837
fbshipit-source-id: e2c8ec054a3696d2cf02e65c212ad886c5117253
Summary:
The `parents` template currently returns "meaningfulparents". This sometimes
returns the null revision as the second parent, making templates think this is a
merge commit when it is not. Make sure we filter this out.
Reviewed By: quark-zju
Differential Revision: D21347776
fbshipit-source-id: af83fd5cff381850ac39d97b5b2f4c77033fe2fb
Summary:
Make all the things that only Lua hooks needed (hook type etc) optional.
With this done, configs can be cleaned up to not contain redundant data.
Reviewed By: ikostia
Differential Revision: D21349614
fbshipit-source-id: 1c72c2082b8c002e3feb41d1d720a41d21afaae5
Summary:
In the initial stages of the windows port we had
problems building rocksdb on windows, so we disabled it.
These days we're able to build it and detect it--we even
require it in the cmake code, but hadn't gotten around
to telling the rest of the code that we can use it.
This commit re-enables it in the build but leaves sqlite
as the default engine until we're able to perform some
benchmarking.
Rocksdb itself has some build issues on Windows; it doesn't
use cmake to locate dependencies, so even though we built
snappy it doesn't know how to find it without modifying the
source:
https://github.com/facebook/rocksdb/blob/master/thirdparty.inc#L4
For that reason, we disable the use of Snappy in the Windows build.
However, in the version of rocksdb that we were using, it would
default to trying to use Snappy even though it wasn't compiled in
and throw an exception.
I've upgraded to a newer version of rocksdb that will simply not
use compression if no compression was enabled at build time.
Given that we mostly store relatively small objects, I'm assuming
that the lack of compression is fine for now.
Reviewed By: xavierd
Differential Revision: D21319896
fbshipit-source-id: 2a2d06d4bd5382706e9220f9b4a2de99dc18311d
Summary: `cargo autocargo` should normally produce no changes on `master`. The features of the `log` crate was updated in D21303891 without re-running autocargo. This fixes it.
Reviewed By: dtolnay
Differential Revision: D21349799
fbshipit-source-id: ce487bc5989e179673297350249593103b4d34dd
Summary:
Now we download files in 200K chunks, and it might be confusing - it's unclear
whether hg is making any progress or not.
This diff makes it clearer.
Note I'm not printing progress before first fetch to avoid breaking the tests
Reviewed By: farnz
Differential Revision: D21348756
fbshipit-source-id: 05e5169114adf2b99a74b37d933755a644214a42
Summary: Mercurial Infinitepush normally records received bundles into the `forwardfillerqueue`, which is later tailed by the commit cloud `forwardfiller` in order be replayed onto Mononoke. Now I am adding a reverse filler, which means that we will have two such queues and sync in two directions. Therefore, in order to avoid infinite loops we need to distinguish cross-backend bundle replay from genuine pushes. I propose to use `crossbackendsync` bundle2 param to indicate that no recording is needed.
Reviewed By: krallin
Differential Revision: D21255446
fbshipit-source-id: 70f6efe1331bd3c7fd3aca61d486d350d93086dc
Summary:
Previously we weren't showing help at all when running "hg cloud --help". This
diff should fix it
Reviewed By: ikostia
Differential Revision: D21347825
fbshipit-source-id: 1d9d11e2f9fe18a03b5d2cd8bd316fe9a218347c
Summary: Let's make it tunable, more context in D21300885
Reviewed By: ikostia, krallin
Differential Revision: D21347636
fbshipit-source-id: 4bc91effdd212a3f5c4a0c3d4952f52a1cf417d7
Summary:
Path elements longer than 255 bytes cannot be materialized on most filesystems,
so allowing them isn't very useful.
We can't normally receive a push for this (because the client would have to
have the file on their isk), but they could get created through a
specially-crafted bundle or accidentally through a Thrift API (e.g. create
commit in SCS).
Reviewed By: farnz
Differential Revision: D21328145
fbshipit-source-id: cb32d6df0aa60863665a651f1a33c69c75c6a880
Summary:
In vs-code if you have a well-formed url you can command-click it
to open it, e.g., when you see it in your terminal. But this URL is not
well-formed because it's missing the protocol. So let's add it.
Reviewed By: quark-zju
Differential Revision: D21336742
fbshipit-source-id: dd2de2d5177d3a2542c4a22f0099f28b97c79d06
Summary:
Update the Windows code to use the normal StartupLogger.h and .cpp files.
Previously the Windows code defined its own separate StartupLogger.h file.
The class declaration looked largely like a copy of the POSIX
DaemonStartupLogger code, but most of the methods were not actually defined
anywhere. The actual functionality appears identical to the POSIX
ForegroundStartupLogger, so this code just switches to using that in most
cases.
Reviewed By: xavierd
Differential Revision: D21332570
fbshipit-source-id: 0585a38d8f37dc93459d380aa277082c35cbadfc
Summary:
Currently, Mononoke's configs are loaded at startup and only refreshed
during restart. There are some exceptions to this, including throttling limits.
Other Mononoke services (such as the LFS server) have their own implementations
of hot reloadable configs, however there isn't a universally agreed upon method.
Static configs makes it hard to roll out features gradually and safely. If a
bad config option is enabled, it can't be rectified until the entire tier is
restarted. However, Mononoke's code is structured with static configs in mind
and doesn't support hot reloading. Changing this would require a lot of work
(imagine trying to swap out blobstore configs during run time) and wouldn't
necessarily provide much benefit.
Instead, add a subset of hot reloadable configs called tunables. Tunables are
accessible from anywhere in the code and are cheap to read as they only require
reading an atomic value. This means that they can be used even in hot code
paths.
Currently tunables support reloading boolean values and i64s. In the future,
I'd like to expand tunables to include more functionality, such as a rollout
percentage.
The `--tunables-config` flag points to a configerator spec that exports a
Tunables thrift struct. This allows differents tiers and Mononoke services to
have their own tunables. If this isn't provided, `MononokeTunables::default()`
will be used.
This diff adds a proc_macro that will generate the relevant `get` and `update`
methods for the fields added to a struct which derives `Tunables`. This struct is
then stored in a `once_cell` and can be accessed using `tunables::tunables()`.
To add a new tunable, add a field to the `MononokeTunables` struct that is of
type `AtomicBool` or `AtomicI64`. Update the relevant tunables configerator
config to include your new field, with the exact same name.
Removing a tunable from `MononokeTunables` is fine, as is removing a tunable
from configerator.
If the `--tunables-config` path isn't passed, then a default tunables config
located at `scm/mononoke/tunables/default` will be loaded. There is also the
`--disable-tunables` flag that won't load anything from configerator, it
will instead use the `Tunable` struct's `default()` method to initialise it.
This is useful in integration tests.
Reviewed By: StanislavGlebik
Differential Revision: D21177252
fbshipit-source-id: 02a93c1ceee99066019b23d81ea308e4c565d371
Summary:
FUSE_HANDLE_KILLPRIV expects that any write() call is handled by
clearing the setuid and setgid bits in the userspace. To avoid
implementing that behavior, disallow setting setuid or setgid in the
first place.
Reviewed By: xavierd
Differential Revision: D21333703
fbshipit-source-id: eb084ee8b00afe74c0da26e41c32c2cb742723da
Summary: I'm about to add a new parameter to `scribe_commit_queue`; first asyncify it to modernise
Reviewed By: krallin
Differential Revision: D21288044
fbshipit-source-id: d1a4bb052b3c055383dd9d9df5fe36d61b14bdfe
Summary: If the bundle file has 1G in size, we probably don't want to waste 1G of memory just to see that it is too big.
Reviewed By: markbt
Differential Revision: D21177359
fbshipit-source-id: 026b5ab40e6dbddba3d7142a8e34256d127bf82c
Summary:
Building on the previous two commits, this adds a test which performs the following steps:
- does an infintiepush push to a Mercurial server
- looks into the `forwardfillerqueue`
- runs commitcloud forwardfiller's `fill-one` to replay the bundle to Mononoke
- verifies that this action causes the commit to appear in Mononoke
As this test uses `getdb.sh` from Mercurial test suite, it needs to be whitelisted from network blackholing (Note: we whitelist mononoke tests, which run with `--mysql` automatically, but this one is different, so we need to add it manually. See bottom diff of the stack for why we don't use `--mysql` and ephemeral shards here).
Reviewed By: krallin
Differential Revision: D21325071
fbshipit-source-id: d4d6cbdb10a2bcf955ee371278bf2bbbd5f5122c
Summary:
Microwave doesn't normally allow writes, which can cause cache warmup to fail
if master has underived commits. So, let's go back in bookmarks history to
whatever is most recent and derived. We can do so using the existing logic we
use in the warm bookmarks cache.
Reviewed By: farnz
Differential Revision: D21325485
fbshipit-source-id: 11e758cd512a22e02704ac34458fead18c284c20
Summary:
This updates cache_warmup to new futures, since I'd like to do a little bit of
work in there. This also changes a bit of functionality: until now we were
roundtripping through a hg changeset id un-necessarily so that updates it to
just use the bcs id instead.
Reviewed By: mitrandir77
Differential Revision: D21325484
fbshipit-source-id: 4d296c5fbe7c21dda681aed4ed9c572a38246893
Summary:
Upgrade to the 1.10.0 release. This includes some new features like the
`GTEST_SKIP()` macro.
Reviewed By: genevievehelsel
Differential Revision: D21309360
fbshipit-source-id: 163db628fc99aaa786aeb207f35c7d6295cb5e25
Summary:
Update the generated `run_cmake.py` script to tell ctest to print the test
output on failure. Also pass in a `-j` flag to run tests in parallel by
default.
These flags are already passed in by default when running `getdeps.py test`;
this simply updates this developer utility script to do the same.
Reviewed By: wez
Differential Revision: D21307806
fbshipit-source-id: 42045b0f9362494042c79bc946a1004ff8ad98b6
Summary:
Instead of multiple linear array scans to determine the type of a FUSE
request, use a single lookup table. This is a small optimization, but
it makes the code a bit nicer too.
Reviewed By: wez
Differential Revision: D21314720
fbshipit-source-id: 5b6700ad5bb8e353da1e457de1533e6a626e8f68
Summary:
Google Benchmark is easier to use, has more built-in functionality,
and more accurate default behavior than Folly Benchmark, so switch
EdenFS to use it.;
Reviewed By: simpkins
Differential Revision: D20273672
fbshipit-source-id: c90c49878592620a83d2821ed4bc75c20e599a75
Summary:
Cloning a repository with selectivepull enabled temporarily disables
selectivepull for the duration of the initial pull so that we can get a
streaming clone.
Once this is done, hide all of the commits in the repository by clearing the
visible heads. Selective pull will then populate the remote bookmarks with the
public heads that we do want.
Reviewed By: quark-zju
Differential Revision: D21301037
fbshipit-source-id: 565ae50439ed5405ce940a5675caeba912fe7083
Summary: If a scratch remotebookmark points to a draft commit that is not available locally (i.e., has been hidden on a different machine), then don't pull it into the local repository. Instead, just omit the remote bookmark.
Reviewed By: quark-zju
Differential Revision: D21287886
fbshipit-source-id: 84e7c6b52250709f7c88b07fccdbb61e044370c8
Summary:
Selective pull subscripts are lost during a push. This is because
`remotenames.expush` calls `selectivepullbookmarknames` with the
remote, not the remote name.
`remotenames.exclone` does the same, but in that case there is
no need for the remote name as the destination can't have any
subscriptions other than the default set.
Reviewed By: quark-zju
Differential Revision: D21283029
fbshipit-source-id: 52c2e7e301b8e95b4a252cbfe2ad9de168e81044
Summary:
Commit cloud remote bookmarks sync has several edge cases that don't work correctly.
This test demonstrates some that we will fix.
Reviewed By: quark-zju
Differential Revision: D21283028
fbshipit-source-id: 4e746476a0f3bf0ca7d7088f510451632a2ee075
Summary:
This is a fix after D21067809 broke the mononoke_api unittests and was not caught by sandcastle.
Prior to this change if `ctx.identities` were None then the permission was unconditionally denied, even when the permission checker is set to always_allow. After this change if the identities is None then the permission_checker is called with empty identity set thus making unit tests work properly.
Reviewed By: krallin
Differential Revision: D21324038
fbshipit-source-id: 1edd5fd2af6731d43cab06324b4d9fc02882c09e
Summary:
The systemd binary is installed in a different location on Debian-based
distributions (e.g., Ubuntu) from RedHat-based distributions. Update the
integration tests to look in both places.
Reviewed By: wez
Differential Revision: D21297591
fbshipit-source-id: 67e5a698080205b8624c1f38aa49e75c20b6a28c
Summary:
On Linux the debug info sections in projects downstream from folly and
thrift are huge due to a lot of C++ template usage.
We build in RelWithDebInfo mode as that is most useful when debugging
locally and because the build times are long, but most casual consumers
of the binaries via GH actions really don't care about debug info,
and this should save ~180MB of size in (for example) the watchman
executables.
Pull Request resolved: https://github.com/facebook/watchman/pull/809
Refs: https://github.com/facebook/watchman/issues/804
Test Plan:
Review the artifact size on the linux build in this PR (see
https://github.com/facebook/watchman/actions/runs/91688952) and confirm that it
is a bit smaller than 180MB; now under 3 MB.
Reviewed By: simpkins
Differential Revision: D21318302
Pulled By: wez
fbshipit-source-id: f78bc5e735dd78771e0604abae64c0a23cf9158d
Summary:
Our CMakeLists.txt file requests `ctest` to pass in a `CMAKE_SOURCE_DIR`
environment variable when running our integration tests via `ctest`. However
if a developer manually invokes the test binary this environment variable may
not be set.
For convenience, this updates the code to try looking at the default location
where getdeps will have put the source relative to the build directory (at
least for internal builds using shipit-converted fbsource sources).
Reviewed By: wez
Differential Revision: D21307807
fbshipit-source-id: d747c50d79d2d378721b68b18c256d5363e41f26
Summary:
Update the integration tests to attempt to find the Mercurial config files
from the source directory even in CMake-based builds. This directory will be
available when doing an internal build from fbsource via getdeps.py
If this directory isn't present (e.g., on a build from the github repository)
we will still fall back to using the Mercurial configuration files installed
on the system. In that case I did update the code to use the correct system
configuration path on Windows.
Reviewed By: wez
Differential Revision: D21307808
fbshipit-source-id: 917ba5e8548fed999fb06a29324be125cec83cac
