Summary:
Tl;DR Reduces costs of fuse request mertics by reducing lock contention.
D20922194 adds tracking for FUSE request metrics, this makes tracking those
metrics more efficient. Since every user request goes through the FUSE channel,
we want to reduce the cost of these metrics as much as possible (originally
mentioned in a comment D20922194).
The synchronization used to track the metrics is costly especially
when the lock is contended.
To reduce the cost, each FuseChannel will have a thread local copy of metrics.
Each will still be synchronized to allow for reading the metrics and for
Requests moved to other threads that will need to access the metrics. However,
the lock should be contended less often since only requests from a single fuse
channel thread will access it.
Reviewed By: chadaustin
Differential Revision: D21043792
fbshipit-source-id: ce58a0cbce334095976233bfac7578d39c81bb55
Summary:
`scmutil.revsingle` resolves the commit using the revset language layer,
which will trigger auto pull logic. The import helper does not want any
kind of auto pull logic. Therefore use `repo[rev]` instead, which resolves
the commit hash directly and bypasses the revset / auto pull logic.
Reviewed By: singhsrb
Differential Revision: D21196495
fbshipit-source-id: 5a51057a731523bbb643c7e264d6902dcfbb9059
Summary: Move the MutationEntry type to the Mercurial types crate. This will allow us to use it from Mononoke.
Reviewed By: quark-zju
Differential Revision: D20871338
fbshipit-source-id: 8de3bb8a2673673bc4c8a6cc7578a0a76358c14a
Summary:
Add debug logging and perf counters for the number of mutation entries stored
during `add_entries`, and the number of mutation entries fetched during
`all_predecessors`.
Reviewed By: StanislavGlebik
Differential Revision: D21065934
fbshipit-source-id: 9b2ff9720116e6a168706f994655daffb18d0ffc
Summary:
This commit adds the `add_entries` method to the mutation store, which
allows Mononoke to add new entries to the store for a given set of
commits.
It is expected that the client will provide all of the mutation entries related
to the commits it is sending. This may be too much information, as in normal
operation we will already know about the predecessors. In that case we can
avoid additional work by just storing the entries directly related to the new
commits.
If the client has made commits while offline, then some mutation entries may
refer to predecessors that are not known locally. In this case we must search
through the mutation history looking for the last commits we did know about,
and add all the in-between entries, too.
Reviewed By: StanislavGlebik
Differential Revision: D20287383
fbshipit-source-id: e5fb42bc4da7873c3a5aafd83684d374d9155bca
Summary:
Add the Mononoke Mercurial mutation store. This stores mutation information
for draft commits so that it can be shared between clients. The mutation
entries themselves are stored in a database, and the mutation store provides
abstractions for adding and querying them.
This commit adds the `all_predecessors` method to the mutation store, which
allows Mononoke to fetch all predecessors for a given set of commits. This
will be used to serve mutation information to clients who are pulling draft
commits.
Reviewed By: krallin
Differential Revision: D20287381
fbshipit-source-id: b4455514cb8a22bef2b9bf0229db87c2a0404448
Summary:
Make the default output for `scsc log` shorter by only including the first line of the commit message, and omitting less interesting fields like commit extras.
The full details are hidden behind a `--verbose` flag, similar to `hg log`.
Reviewed By: mitrandir77
Differential Revision: D21202318
fbshipit-source-id: f15a0f8737f17e3189ea1bbe282d78a9c7199dd9
Summary:
Multiple functions in cmdlib were looking for
`"mononoke-config-path"`. Make it into a constant and provide a helper function
to reduce duplication.
Further, update `read_common_config` to accept `impl AsRef<Path>` to make
calling the function easier.
Reviewed By: farnz
Differential Revision: D21202528
fbshipit-source-id: 96cad817ed47be0f207965ad2bc33af13ca8b5fd
Summary:
Add the `commit_history` method to the source control service.
This methods returns the history of the commit, directly from the changelog.
This differs from `commit_path_history` in that it includes all commits, whereas `commit_path_history` will skip commits that do not contain changes that affect the path.
Differential Revision: D21201705
fbshipit-source-id: dbf1f446c106620620343122176eccd5d809779c
Summary:
Add a method to get the history of a changeset. This differs from the history
of a changeset path, even the history of the root directory, in that all
changesets are included, even empty ones.
Differential Revision: D21179877
fbshipit-source-id: e19aac75fc40d8e9a3beb134e16a8cdfe882b791
Summary: Add a new unit test for `commit_path_history`. We will use this test to contrast against `commit_history`.
Differential Revision: D21179878
fbshipit-source-id: 318aa34d8d80f61c1e52d4053d5aead5a71e864c
Summary: The new MembershipChecker and PermissionChecker traits will generalize access to various permission/acl systems (like LDAP) and leave the implementation details hidden behind an object trait.
Reviewed By: StanislavGlebik
Differential Revision: D21067811
fbshipit-source-id: 3bccd931f8acdb6c1e0cff4cb71917c9711b590b
Summary: Make it possible to traverse fsnodes in walker.
Reviewed By: ahornby
Differential Revision: D21153883
fbshipit-source-id: 047ab73466f48048a34cb52e7e0f6d04cda3143b
Summary: File is getting too large - let's split it
Reviewed By: farnz
Differential Revision: D21180807
fbshipit-source-id: 43f0af8e17ed9354a575b8f4dac6a9fe888e8b6f
Summary:
The part of status that lists what files have changed is called
PendingChanges. This diff introduces the initial stub for PendingChanges. The
pending changes algorithm involves three parts:
1. Looking at files on the filesystem for changes.
2. Looking at files in the dirstate map for changes.
3. Looking at the content for any files that we were unsure of during steps 1
and 2.
This diff puts the basic state machine in place, and accepts the basic
information about the working copy (the root and what type of filesystem it is).
In the future we might have it detect what type of filesystem it is, but for now
this makes it easy.
Reviewed By: xavierd
Differential Revision: D20546898
fbshipit-source-id: a3030b7c846b3cb2fcba805b7fe4744df7c5764e
Summary:
treestate.get_filtered_keys passes directory paths to the filter
function and returns directory matches with a trailing '/' on the end. This
makes it difficult to act as a path normalization function when the caller
doesn't know if the path is a file or directory.
It seems like we can just strip the trailing '/' before exposing the strings to
the caller (both as filter inputs and as get_filtered_keys outputs).
This is useful in the following diff that adds a case normalization crate.
Reviewed By: xavierd
Differential Revision: D20880881
fbshipit-source-id: 6e9f419178b4e278844244bd6aff2fc10e09d2cd
Summary:
This logic will be used in a variety of places (update workers, status,
etc). Let's move it somewhere common.
Reviewed By: xavierd
Differential Revision: D20771623
fbshipit-source-id: b4de7c1d20055a10bbc1143d44c55ea1045ec62a
Summary:
In a later diff we'll need to be able to hand a reference to the
TreeState to the pending changes iterator. We'd like to be able to hand a
Rc<RefCell<TreeState>> but cpython requires that its fields implement Send. The
simplest solution is to use Arc<Mutex<_>>. Once we finish Rustifying all of this
code we can drop the cpython requirement that this work across threads and
downgrade this to a Rc<RefCell<_>>.
Reviewed By: xavierd
Differential Revision: D20546904
fbshipit-source-id: a4a1ce6973f53b3bb95f227616149f98fcd780e0
Summary:
PathAuditor will be needed for native status soon. Let's move it into
the workingcopy crate.
Reviewed By: xavierd
Differential Revision: D20546906
fbshipit-source-id: ef69f88ee828a72e82b5e944cc7913f391bd8a2f
Summary: Update the test to match the latest output.
Reviewed By: xavierd
Differential Revision: D21181146
fbshipit-source-id: 3b6190c9ae8e73309c0b7cb3c48d8a6646bc7fb4
Summary: This will help us debug slow commands
Reviewed By: xavierd
Differential Revision: D21075895
fbshipit-source-id: 3e7667bb0e4426d743841d8fda00fa4a315f0120
Summary:
Several of the tests started edenfs of fake_edenfs processes and didn't kill
them at the end of the test. This adds proper cleanup functions to try and
kill these processes when the test exits.
This also eliminates using pexpect in these tests, since it isn't actually
necessary here.
Reviewed By: wez
Differential Revision: D21084096
fbshipit-source-id: 4e92a99a5c398d4a78830ac51507ba34d7a6c0b1
Summary:
This updates the various systemd-related tests to use EdenTestCaseBase,
and allows us to delete the EnvironmentVariableMixin and
SystemdUserServiceManagerMixin classes.
The main goal of this clean-up is to make it easier to consolidate most of the
systemd-related code into just a few locations, so that we can more easily
disable the systemd-related tests in build environments where systemd is not
supported.
Reviewed By: wez
Differential Revision: D21084098
fbshipit-source-id: d5e05254c689c28751fe48d2dc38d722c7e77ed3
Summary:
This adds a new `EdenTestCaseBase` class to serve as the base class across a
number of our integration tests and some of our CLI tests.
The main goal of this is to allow eliminating many of the annoying `*Mixin`
classes used in a lot of our integration tests. These mixin classes are
annoying since they result in complicated multiple inheritance, and it can be
tricky to ensure that the method resolution order behaves the way you want.
The systemd tests in particular use a lot of mixins, which gets complicated.
These mixin classes are also awkward from a Python typing perspective, and the
systemd tests end up resorting to just declaring different APIs in several
places when `typing.TYPE_CHECKING` is True.
The fact that `EdenTestCaseBase` has a `contextlib.ExitStack` member variable
should make it easier for us to eliminate these mixins moving forward: rather
than using mixins that use inheritance and assume a `self.cleanUp()` method
exists, we can transition this code to standalone functions or context
managers, and they can take the `ExitStack` variable as an argument if
necessary.
Reviewed By: wez
Differential Revision: D21084097
fbshipit-source-id: 77ee457b7debe6f584f630e3e30f79fe634a2026
Summary:
If a commit operation results in a commit that is already present in the
repository, we must revive the commit, rather than create a new one. In this
case, if obsmarkers are disabled, we can't create a revive marker.
Reviewed By: quark-zju
Differential Revision: D21179879
fbshipit-source-id: 320c70b0ca1e0f384733e09e0e1dd77ac3a255f9
Summary:
Update the integration tests to only run git integration tests if EdenFS was
built with git support.
Reviewed By: genevievehelsel
Differential Revision: D21000162
fbshipit-source-id: 0a015ae93a9420f040be702096e0b378fe138e4c
Summary:
Update the code to work around the fact that Pyre refuses to unwrap Optional
member variables. Store the data in local variables so that Pyre will allow
unwrapping these objects after we check against None.
Reviewed By: genevievehelsel
Differential Revision: D21000165
fbshipit-source-id: 59e8430de30ec35213a53b06670b16c34be85593
Summary:
Update some of the eden integration tests to use the `eden.config` module
instead of trying to import the `__manifest__` module, which is only present
in Buck-based builds.
Reviewed By: wez
Differential Revision: D21000167
fbshipit-source-id: 347082a32e842752e5539a8e6c8addde97a16eee
Summary:
Update most locations in edenfsctl to report the version number that was built
into the edenfsctl binary at build time, rather than querying the RPM database
for the installed RPM version. The RPM behavior only works on to RedHat-based
Linux systems, and the currently running process doesn't necessarily have to
have come from the RPM.
The one place where we do still attempt to print the RPM version is in the
`edenfsctl rage` report, when running on Linux.
Reviewed By: wez
Differential Revision: D21000168
fbshipit-source-id: 0fb747e71b6950d74f22c458efa0dfcbd45270bd
Summary:
This updates the top-level CMakeLists.txt file to compute package version
information, and expose this to C++ code in `eden-config.h`, and to Python
code in a new `eden/config.py` module.
Previously we exposed an `EDEN_VERSION` macro for the C++ code in
`eden-config.h`, but this was not initialized or used anywhere. Now the
top-level CMakeLists.txt file computes appropriate version information and
exposes the package name, version, release, commit ID, and build time in these
configuration files.
The version selection logic in CMakeLists.txt based largely on the code that
wez wrote for watchman in D20636833.
Reviewed By: wez
Differential Revision: D21000164
fbshipit-source-id: db1a1035f1eefec058bbad558d35e113005e454e
Summary:
This exposes metrics for the live FUSE requests (the duration
of the longest outstanding request and the number of outstanding
requests).
Because FUSE is the interface through which the user mostly interacts
with the file system they provide good metrics to judge if the perfomance
of eden is normal, or there may be an issue.
Exposing these counters this way will send them to ods, so it will not only
allow for debuging current issues, but can be used to look back at previous
problems. This data could also be used for alerting or more proactive
remediation.
Metrics are exposed per checkout to allow seeing which checkout was
having issues. This data will aggregated in `eden top` to be used as
an overall health indicator, but should more information be needed it
will be logged in ods.
Reviewed By: chadaustin
Differential Revision: D20922194
fbshipit-source-id: 16208883417acb77b62bf712cfdd9068c5420303
Summary:
Without this change the following code will fail surprisingly:
In [1]: def f():
...: return m
...:
In [2]: f()
NameError: global name 'm' is not defined
Reviewed By: singhsrb
Differential Revision: D21140299
fbshipit-source-id: f0a6509badcd9314a33a5ca0c78a60bb847d63c7
Summary:
Make `ml`, `serv`, `bin`, `hex`, `util` available as they are handy.
Remove less useful names like `e`.
Reviewed By: singhsrb
Differential Revision: D21140300
fbshipit-source-id: ca012da6e33fad64b352c117595538e10d1fad83
Summary: Certain node types are not to be followed by any text/parts. This change makes parse_node repor an error if they are.
Reviewed By: ahornby
Differential Revision: D21129504
fbshipit-source-id: 4a4c2755d786e81ab43cec86d5fd5189ea0e138f
Summary: Small cleanup: WalkState type wraps an Arc, can replace with implementing the WalkVisitor trait for Arc<WalkVisitor>
Reviewed By: farnz
Differential Revision: D21129622
fbshipit-source-id: 12a8a8d3b5ba2e459658a3cc71021c8d700db3b8
Summary:
For some nodes like FileContent from a BonsaiChangset, the file path is not part of node identity, but it is important for tracking which nodes are related to which paths.
This change adds an optional path field to the OutGoingEdge so that it can be used in route generation and as part of edge identity for sampling.
Its optional as some walks don't need the paths, for example scrub.
Reviewed By: farnz
Differential Revision: D20835653
fbshipit-source-id: f609c953da8bfa0cdfdfb26328149d567c73dbc9
Summary: use WrappedPath to Arc all MPaths held in the graph. Doing it with a wrapper type rather than Option<Arc<MPath>> so have the option of using internalization instead later.
Reviewed By: krallin
Differential Revision: D20868508
fbshipit-source-id: e47458c4b274e4cd7bd0c400cd033eebc3c85d14
Summary: Report stats by the node type (e.g. FileContent, HgManifest etc) for blobstore usage when scrubbing so we can see how large each type is.
Reviewed By: StanislavGlebik
Differential Revision: D20564327
fbshipit-source-id: 55efd7671f893916d8f85fa9a93f95c97a098af4
Summary:
Correctly identify infinitepush without bookmarks as infinitepush instead of plain push.
Current behavior would sometimes pass `infinitepush` bundles through the `push` pipeline. Interestingly, this does not result in any user-visible effects at the moment. However, in the future we may want to diverge these pipelines:
- maybe we want to disable `push`, but enable `infinitepush`
- maybe there will be performance optimizations, applicable only to infinitepush
In any case, the fact that things worked so far is a consequence of a historical accident, and we may not want to keep it this way. Let's have correct identification.
Reviewed By: StanislavGlebik
Differential Revision: D18934696
fbshipit-source-id: 69650ca2a83a83e2e491f60398a4e03fe8d6b5fe
Summary:
In next diffs with permission_checker the panic is changed to anyhow::Error.
The previous behavior of this code was that when AclChecker updated failed
after 10s this fact was ignored and the hooks were simply not using ACLs. This
diff fixes it so that the server exits when AclChecheker update is timing out.
Reviewed By: johansglock
Differential Revision: D21155944
fbshipit-source-id: ab4a5071acbe6a1282a7bc5fdbf301b4bd53a347
