Summary:
The rate metric can be unreliable, and in some environment is never updated by
the time it is read by the test. Since this test cares about the number of
events, using count is a better metric, and more reliable.
Differential Revision: D30377128
fbshipit-source-id: f10c656567e3a3c07b66ecc6fc563a53199e3088
Summary:
The diff is giant, but it's just a one-line change to add the
nested-values feature to slog, we just have a whole bunch of projects dependent
on slog.
Reviewed By: dtolnay
Differential Revision: D30351289
fbshipit-source-id: b6c1c896b06cbdf23b1f92c0aac9a97aa116085d
Summary: `//common/rust/shed/cached_config` is the center of a dependency graph that all only uses old configerator because cached_config uses it. This diff switches all of these over to the new client.
Reviewed By: farnz
Differential Revision: D30357631
fbshipit-source-id: 9a9df74096aa38a06371c6bc787245af71175e48
Summary:
The `DerivedDataManager` will manage the ordering of derivation for derived
data, taking into account dependencies between types as well as the topological
ordering of the repository. It will replace the free functions in
`derived_data` as well as much of the `utils` crate.
This is the first step: it introduces the manager, although currently it only takes
over management of the derived data lease.
Reviewed By: mitrandir77
Differential Revision: D30281634
fbshipit-source-id: 04c3a34d97ea02cc8c26d34096cca341e800da9b
Summary:
In preparation for the derived data manager, ensure that derived data
mappings do not require a `BlobRepo` reference.
The main use for this was to log to scuba. This functionality is extracted out
to the new `BonsaiDerivedMappingContainer`, which now contains just enough
information to be able to log to scuba.
Reviewed By: mitrandir77
Differential Revision: D30135447
fbshipit-source-id: 1daa468a87f297adc531cb214dda3fa7fe9b15da
Summary:
We have mover only for files, and it doesn't quite work for directories - at
the very least directory can be None (i.e. root of the repo).
In the next diffs we'll start recording files and directories renames during
megarepo operations, so let's DirectoryMultiMover as a preparation for that.
Reviewed By: mitrandir77
Differential Revision: D30338444
fbshipit-source-id: 4fed5f50397a7d3d8b77f23552921d515a684604
Summary:
AOSP megarepo wants to create release branches from existing branches, and then update configs to follow only release-ready code.
Provide the primitive they need to do this, which takes an existing commit and config, and creates a new config that tracks the same sources. The `change_target_config` method can then be used to shift from mainline to release branch
Reviewed By: StanislavGlebik
Differential Revision: D30280537
fbshipit-source-id: 43dac24451cf66daa1cd825ada8f685957cc33c1
Summary:
Let's make it possible to query mutable renames from fastlog. For now this is a
very basic support i.e. we don't support middle-of-history renames, blame is
not supported etc.
Mutable rename logic is gated by a tunable, so we can roll it back quickly in
case of problems.
Reviewed By: ahornby
Differential Revision: D30279932
fbshipit-source-id: 0e8e329e8ab4d4980ab401bd103e6c97419d0f67
Summary:
Let's make it possible to build mutable renames using repo factories. It will
be used in the next diffs.
Differential Revision: D30279930
fbshipit-source-id: 57e873c69495e541daf943a47e6cb46fc19b221b
Summary:
When a user reports a slow EdenFS and a high network traffic is suspected, the
lack of Thrift stats makes it hard to fully validate this. Thus, let's collect
some stats and put them in the rage.
The collected stats are the exact same ones that `eden top` uses.
Reviewed By: chadaustin
Differential Revision: D30355746
fbshipit-source-id: 519a8e2c8b0c458daecdcc0813a8d7416d5362d6
Summary:
In the ASIC test environment, we've see cases where this isn't set, let's
simply skip the test in this case instead of failing.
Reviewed By: chadaustin
Differential Revision: D30353929
fbshipit-source-id: 956da6f8f12de025b8ca72e40097f1f9d50e6bf7
Summary:
With D30320515, EdenFS internally canonicalize all the mount paths passed to
it. As a result, the output of `eden list` may not match the path given to
`eden mount` if one of the directory leading to the mount point was a symlink.
Since some tests are comparing both, this can lead to some test failures.
To solve this, we simply need to make sure that the temporary directory for the
test is canonicalized.
Reviewed By: fanzeyi
Differential Revision: D30349411
fbshipit-source-id: 139d4be02b5783c6a439270845239acab6a6c955
Summary:
In the case where the path to the mount has symlinks, EdenFS would only accept
the path to it that was specified at mount time, even though another path may
refer to the same directory.
To solve this, we can simply normalize paths in all the Thrift endpoint to make
sure that EdenFS always refers to a mount point under its non-symlinked path.
Reviewed By: chadaustin
Differential Revision: D30320515
fbshipit-source-id: e578d059a3b1307d6b24c4b9bdb1ceb3b534c460
Summary:
We currently do repo lock checks in a loop during unbundle. However we now do a repo lock check in the bookmarks_movement::PushrebaseOntoBookmarkOp::run(), making the loop and check in repo_client unbundle redundant
Cons: It will no longer early terminate. Pros: database load should be reduced.
Reviewed By: StanislavGlebik
Differential Revision: D30331806
fbshipit-source-id: 16ee72e570184c20ac08d3fa6d8f9f333c91deb7
Summary:
During flush, there could be newly added vertexes in the master group,
overlay_map_next_id needs to be updated accordingly so lazy lookup
can still work. This is especially needed when the non-master group
needs to be re-assigned.
Reviewed By: andll
Differential Revision: D30314051
fbshipit-source-id: cce7080f62aec2617b8f3d7194864df41dfff7e0
Summary:
In case anything went wrong in the pull fast path, preserve `self` as unchanged
to avoid possible issues.
Reviewed By: andll
Differential Revision: D30314052
fbshipit-source-id: 127a7aeda8f27f7862d1ad92847c2e911131b4b4
Summary:
Take 2. This time with measured improvement (3x).
With WAL mode enabled in SQLite. The bottleneck focuses at flush time. WPA shows a lot of time are spent at flushing changes into the WAL file.
Setting `synchronous` to `OFF` in SQLite will make SQLite to not wait for flushing changes onto the disk. This saves us time and compromises with correctness only when the system crashes.
From SQLite documentation:
> With synchronous OFF (0), SQLite continues without syncing as soon as it has handed data off to the operating system. If the application running SQLite crashes, the data will be safe, but the database might become corrupted if the operating system crashes or the computer loses power before that data has been written to the disk surface. On the other hand, commits can be orders of magnitude faster with synchronous OFF.
https://sqlite.org/pragma.html#pragma_synchronous
Reviewed By: chadaustin
Differential Revision: D30314601
fbshipit-source-id: 87f6cd05149045313678328bb3fc7dd10e894e37
Summary:
The test is simplified from a user report. It requires a non-master commit to
have a lazy parent in the master group and is unknown locally.
Reviewed By: andll
Differential Revision: D30314055
fbshipit-source-id: 9e2bf04728bc6c524fa3e32b8d80935d5cd7e96f
Summary: Those env vars might affect tests. Reset them in tests.
Reviewed By: andll
Differential Revision: D30314056
fbshipit-source-id: 2cedbe41a332fea582e8ad7577b9d5eaae79c015
Summary: Will be used in the next change.
Reviewed By: andll
Differential Revision: D30314053
fbshipit-source-id: 6491bcfe8b5667075521d021b995e71aab08d7e7
Summary:
Will be used in tests to check what data is inserted into IdMap during clone or
pull.
Reviewed By: andll
Differential Revision: D30314054
fbshipit-source-id: 8a58473ef3d85263985c1be8a6c57a9a10b3fdb4
Summary:
In rare cases, the local IdMap might miss parents of draft roots.
This provides a way to emulate that in tests.
Reviewed By: andll
Differential Revision: D30314050
fbshipit-source-id: 09983604bf3259e62a41224a579774265ae0b272
Summary: This would help investigate some errors.
Reviewed By: andll
Differential Revision: D30314057
fbshipit-source-id: 070e75cf0a39180c544b49c9ed1292d8536040fa
Summary:
- This diff adds validation so that changesets that are not snapshots cannot have untracked or missing files.
- It removes the THIS IS A SNAPSHOT commit message.
- It makes the snapshot created by `hg snapshot createremote` be an actual snapshot.
Differential Revision: D30159184
fbshipit-source-id: 976968c0c2222f950a4a937aa805b25dc07c9207
Summary:
This diff adds some data to BonsaiChangeset that tells whether it is a snapshot or not.
For now, this marks every changeset as not being a snapshot. The next diff will add validation to snapshots, some tests, and mark the current `snapshot createremote` command as uploading snapshots.
Reviewed By: markbt
Differential Revision: D30158530
fbshipit-source-id: 9835450ac44e39ce8d653938f3a629f081247d2f
Summary:
This diff makes the snapshot command upload all types of files that were missing (added/untracked/missing/deleted), using the new types of file changes added on the previous diff.
Next steps:
- Add some indicator to Bonsai Changeset saying it is a snapshot. Verify only snapshots can have certain file changes (untracked/missing).
- Upload the files and the changeset inside an ephemeral bubble instead of in the main blobstore
- Start writing the `snapshot restore` command
Differential Revision: D30137673
fbshipit-source-id: 555238f1d64a5438cde35a843043884a939de4fe
Summary: I have my editor set up to format on save - let's unify this on the standard FB format for thrift files, so that I don't create junk.
Reviewed By: ahornby
Differential Revision: D30285082
fbshipit-source-id: 17b09635a2473174a92e29bb042432dbac44865a
Summary:
The current Mononoke Blobstore Trace scuba table is used with idea of having a record per blobstore and operation. This diff adds logging to the new scuba table of the combined multiplexed operations' outcome, like time spent on the `put` including sync-queue and blobstore write or tracking record of the "some failed others none" cases in `get/is_present`.
This helps to see the real time spent on writes and reads and to assess the impact of changes coming in `get` and `is_present`.
Reviewed By: ahornby
Differential Revision: D30248284
fbshipit-source-id: f79050ced32ba77bd2e220e242407bcd711a9b6d
Summary:
On Windows, it's hard to rename `.hg/store/segments/v1` away because the files
are (too easily) being currently used. Instead, let's write the new lazy
segments to `segments/v1next`, and try to rename it back later on follow-up
commands.
Reviewed By: DurhamG
Differential Revision: D30118952
fbshipit-source-id: e3edb588dccf1acb5f4ed106bbb979bcc8c6e67e
Summary:
OD was running `hg rage --preview` in the background periodically and that has caused edenfsctl creating paste on the user's behalf without them actually knowing.
This will make Mercurial to collect EdenFS rage in dry-run mode (i.e. do not create paste).
Reviewed By: quark-zju
Differential Revision: D30288390
fbshipit-source-id: 1e7f25648ca1b76f24264ee13ed98cf974148b0f
Summary:
Autocargo only allows 1 rust-library per Cargo.toml, but right now we have 3
per Thrift library so that doesn't work:
https://www.internalfb.com/intern/sandcastle/log/?instance_id=27021598231105145&step_id=27021602582167211&step_index=13&name=Run%20config
There's little benefit in Autocargo-ifying those rules anyway since they're of
use to Thrift servers and this doesn't work at all in our OSS builds, so let's
just see if we can just noop them. That'll make the crate not exist at all as a
dep, but even considering that it exists only to link to a C++ library that
Autocargo doesn'tk now how to build anyway, that seems OK?
drop-conflicts
Reviewed By: markbt
Differential Revision: D30304720
fbshipit-source-id: 047524985b2dadab8610267c05e3a1b3770e84e6
Summary:
This is basically a refactor.
Before this diff, `bubble.handle(main)` could be used to access things in bubble with fallback. With this diff, `bubble.wrap_repo_blobstore(main)` can be used for the same effect.
The difference is **the type**, which now is `RepoBlobstore` instead of `EphemeralHandle`. Both are blobstores and work the same way for fetching/putting, but on the following diffs I will want to replace some code (e.g. that creates a changeset) to use the ephemeral blobstore for snapshots, and in order to reuse the same code (which expects `RepoBlobstore`), we need the change of types.
This is part of BlobRepo refactoring as well, as what I'm gonna do is replace BlobRepo with a different facet container that has a RepoBlobstore inside.
Reviewed By: markbt
Differential Revision: D30282624
fbshipit-source-id: 4132797104ecd2596e7da91b1daacc1c6fc85934
Summary: Allow `manifest-tree::Entry` to be constructed directly by other crates, and introduce a `TryFrom` conversion to `manifest::List::Directory`. This will be used by the scmstore `BackingStore` implementation to avoid having to rely on the `TreeStore` trait, which does not support batching.
Reviewed By: kulshrax
Differential Revision: D30282738
fbshipit-source-id: 590350dd53217fa8a181e91b194abca753a8adbe
Summary: Add a new results adapter for `FileStore` (which will probably become the primary method of consuming batch errors in the future), which iterates over all keys, and explicitly encodes not found with `Option`, separate from the outer `Result`.
Reviewed By: kulshrax
Differential Revision: D30254000
fbshipit-source-id: 074c6ac4c279fdf72078dca17231e58d0c704956
Summary:
Add a `local` method to `TreeStore` like the one on `FileStore` which returns a `TreeStore` with only the local subset of backends.
Required for `BackingStore` implementation.
Reviewed By: kulshrax
Differential Revision: D30253980
fbshipit-source-id: 142f2d88454826ff9cb9c34f30b7d21bf62b297c
Summary: Add a new method, `edenapi_override`, to the `TreeStore` and `FileStore` builders for use in the `BackingStore` implementation.
Reviewed By: kulshrax
Differential Revision: D30253837
fbshipit-source-id: 4a42e83621fb2634024e4ee8529c26aeae0256a4
Summary:
Instrument all the fetching-related legacy API implementations (ie, not `get_logged_fetches` and `get_shared_mutable`) to track the number of calls, keys, and single-key calls.
This introduces an additional lock acquisition to each of these implementations (and it'd be awkward to merge it with the one in `fetch`), but I think that's probably fine.
For APIs which do not support a variable number of keys, I use `.call(0)` so we simply track the total number of API calls.
Reviewed By: DurhamG
Differential Revision: D30003444
fbshipit-source-id: 8756d2669ca038b3f6a08e211e44e8ccb9251312
Summary:
Adds tracking of FileStore write errors.
Write batches will still abort after the first failure as written.
Reviewed By: DurhamG
Differential Revision: D29997203
fbshipit-source-id: e1cc2ffc4a8d97ca935a7fc9aab30bde3dc548b2
Summary: Refactor the `FileStore::write_batch` method, extracting the various write cases into other methods.
Reviewed By: DurhamG
Differential Revision: D29997096
fbshipit-source-id: 8efac4b2efa2e2225d39583a5c6893efc5096fec
