Summary:
This test has been flaky practically since its creation, although below the threshold to be detected as such some of the times.
Looking at its journal, it failed its first stress run and more than half
of the stress runs thereafter (failed 1920 of them, passed 1519 of them).
The assumption that the test makes that we will always take the local path first
is not stable and has never been.
Match the test's expectations to the actual behaviour of mononoke for now.
If the behaviour is problematic, let's tackle it in its own time.
Note that this test was written here D34276252 in reaction to a change to
segmented changelog here D34041066 and the discussion on that diff is relevant
to establish the correctness of the code.
Reviewed By: RajivTS
Differential Revision: D44297278
fbshipit-source-id: 07f42fc11ca0aa637b59b5fc46263b65d624e0a4
Summary: After enabling remote derivation we've lost related to derivation logging in the parent scuba table (like SCS server). This diffs add log tags, which indicates when we requested remoted derivation and when it was finished. In case of the error we already had the logging in place.
Reviewed By: Croohand
Differential Revision: D44297304
fbshipit-source-id: 1e2f6e279e111ce01b2feaf950a11c7b69a314cf
Summary: useful for checking if an object is in the hgcache or not
Reviewed By: quark-zju
Differential Revision: D44316641
fbshipit-source-id: f16093ec1659a8f139b88e702dbd1d8646319c8a
Summary: In the next diff, we'll use this util function to acquire an exclusive handle to a file for a prjfs integration test.
Reviewed By: xavierd
Differential Revision: D44349708
fbshipit-source-id: ded7037266f539c681569f0c90f875717760fed5
Summary: We are running into fileNotification() failures way more often than we expect on Sandcastle (5% of certain jobs are failing). Let's add some logging here to see if the failures are also happening to users.
Reviewed By: xavierd
Differential Revision: D44350819
fbshipit-source-id: 3be7578dccc297071280f528b53270a63994a99d
Summary:
DurationScope has a pair of atomic reference count operations at the
beginning and end of every recorded duration. To avoid that overhead,
reference EdenStats with RefPtr and, in production EdenFS, store a
global, unowned EdenStats object.
This gives us the benefits of explicit dependencies and no global
state in tests, while avoiding atomic RMWs in the release build.
Reviewed By: xavierd
Differential Revision: D44323723
fbshipit-source-id: 1b3384d2e6a0a2959fd79774a8ba46afc4c176ca
Summary:
In Rust 1.68.0, the `derivable_impls` clippy lint [got smarter](https://github.com/rust-lang/rust-clippy/pull/10161).
Fix all instances of that lint in Mononoke, mute one that's impractical to address.
Reviewed By: markbt
Differential Revision: D44378068
fbshipit-source-id: 473a051a7001d9596db43f47b56cbad5f5db7efe
Summary:
In Rust 1.68.0, the `useless_conversion` clippy lint [got smarter](https://github.com/rust-lang/rust-clippy/pull/10020).
Fix all instances of that lint in Mononoke.
Reviewed By: markbt
Differential Revision: D44378069
fbshipit-source-id: acfd6c77c6a400830c378b4040661323e7232441
Summary:
Original commit changeset: 8c2f66457761
Original Phabricator Diff: D44111246
This seems to be causing stdout ordering issues too, possibly because of PYTHONUNBUFFERED environment variable
Reviewed By: fanzeyi
Differential Revision: D44356269
fbshipit-source-id: 707495d785987308d9c4617a582ccb8613ed5016
Summary: If an EdenAPI request is taking too long, we currently have no signal into this until it either completes or is cancelled. As for wireproto, add logs for long-running requests.
Reviewed By: liubov-dmitrieva
Differential Revision: D44340299
fbshipit-source-id: e24bed96505964c519ea85748207670ded3baacb
Summary:
When streams are cancelled we lose all of their stats, as they only get logged after the stream is polled to completion.
Make the stream stats callback synchronous (we don't actually use its async nature), which means we can call it from a `Drop` implementation, allowing us to log the stats even when the stream is not completed.
To distinguish the two cases, we add a "completed" flag, which is true if the stream was polled to completion.
This also makes `completion_time` optional, as we don't have a value for this if the stream was never polled at all when it got dropped.
`TimedStream` had an implicit fuse-like nature (once completed the stream would never yield anything again), which we rely upon, so make sure this property is maintained.
Reviewed By: liubov-dmitrieva
Differential Revision: D44331959
fbshipit-source-id: 5a7d9875a4d7fd4276c304b6c30ff45c6b990b38
Summary:
As Croohand pointed out on D44183928, if the shared future is cancelled, the `WeakShared` will linger in the collection until the next read of that key. If the read of that key never comes, it will never be cleaned up.
Switch to a scopeguard, that we move into the future. This will execute the code to remove the shared future whether the future succeeds, fails, or is dropped.
Reviewed By: Croohand
Differential Revision: D44342772
fbshipit-source-id: 37fb15d126d8289d61ab33f64f6e1cf9f7a8550d
Summary:
In production we use Logger to log to Scribe (e.g. to "mononoke_new_commit" category). It has WhenceScribeLogged enum field which can be "default", "prod", "sandbox", etc.
Without override, Logger sets "default" value that usually resolves to "prod", **but** it resolves to "sandbox" if code is run from devserver. "sandbox" means we log data to "/sandbox" category instead of usual place.
This is confusing/harmful for us since we only configure to use Logger while working with prod data. For example, if we run SCS locally and land to prod using "land_stack", the commit will go to prod while logging will go to sandbox category. On the other hand, if we run Mononoke Bootstrap to work with a test repo, we don't use Logger at all so the "sandbox" category has no benefit.
We can add more complexity to the Mononoke logging configuration, so we are able to override WhenceScribeLogged in any way we want. But for now, I suggest overriding it in place with "prod" value since we only use Logger in prod. Later we can even simplify the config by removing the old way of logging directly to Scribe and leave just two options: test (log to file) and prod (log via Logger using the "prod" override).
Reviewed By: markbt
Differential Revision: D44118890
fbshipit-source-id: 8d4b9faa6f1f2da1faa96db35dd567102ccde34d
Summary:
We use interngraph GraphQL queries for a few things (phabricator diff info, scmquery log integration).
First, move the graphql endpoint config from static to dynamic config and vary it based on domain.
Finally, set x-x2pagentd-inject-cat in GraphQL request header when using x2pagentd. This replaces the oauth token mechanism, even for existing VPN/corp use. Note that the off-vpn endpoint rejects requests with both x-x2pagentd-inject-cat header and access_token in the body. We can delete the oauth stuff once we are confident in the x2pagent usage here.
Reviewed By: quark-zju
Differential Revision: D44317674
fbshipit-source-id: 329d63207fd9a5b19a145e12b491e270b5924571
Summary:
Send graphql queries over the local proxy socket, if configured. This is a prerequisite for off-VPN compatible access.
I added an escape hatch config phabricator.use-unix-socket we can use to disable this if someone breaks.
Also, kill apparently dead phabricator_graphql_client_requests.py.
Reviewed By: quark-zju
Differential Revision: D44317673
fbshipit-source-id: 52b58ce7899ce7291b700a9fe206da6011f740e5
Summary: Compare file mtimes so we synchronously regenerate dynamic config if the user has modified /etc/mercurial/vpnless. This way things will work off-VPN immediately after writing "1" to /etc/mercurial/vpnless.
Reviewed By: evangrayk
Differential Revision: D44263217
fbshipit-source-id: 1f09591336b7dc1294e1f21fafebce7962859d70
Summary:
To support VPNless access, we need to fetch the remote config through x2pagentd, setting a CAT related header.
I tweaked the order of config loading so the remote config fetch has the system "auth_proxy.unix_socket_path" config setting available.
Reviewed By: quark-zju
Differential Revision: D44263216
fbshipit-source-id: 5b8e6c4c9c7a5ab22ecfcfa62a537623931a63bf
Summary:
ObjectStore.h includes too many headers. Replace several with forward
declarations.
Reviewed By: xavierd
Differential Revision: D44323736
fbshipit-source-id: 944cd72fd758df4363493f87a573d91245a45b6a
Summary:
All of these configs have been enabled by default for a bit, let's change their
default before we can fully remove them from the code.
Reviewed By: genevievehelsel
Differential Revision: D44234768
fbshipit-source-id: 922093a64b91c9a45f84349ea50afc1c08fd6a39
Summary:
XLOG(DFATAL) is non-fatal in production builds, so it's possible we'll fall through here. It's unclear what the compiler will do in this scenario, so let's make sure we return folly::unit to avoid any compiler weirdness.
This may be causing a frequent crash on Sandcastle hosts where we're panicking after logging this DFATAL error. The crash looks like it's coming from an invalid instruction, so falling through here could be the root cause.
Reviewed By: xavierd
Differential Revision: D44314543
fbshipit-source-id: 6d23ffa221d7af4e7be641349eab1eb9027443a1
Summary: Basename suffix skeleton manifests are not been warmed by the warm bookmark cache. This means requests that need these are slowed down while they are derived. Add a warmer for them.
Reviewed By: yancouto
Differential Revision: D43906545
fbshipit-source-id: e1f77b5fe2a7a4512bfc49bf8047afef9b441fa9
Summary: This doesn't appear used anymore (superseded by phabdiff and phabstatus extensions).
Reviewed By: evangrayk
Differential Revision: D44263215
fbshipit-source-id: 323da45474a6cd1f9df730fdce23f4f42e836fb8
Summary:
do not sample out slow requests in edenapi replay
we already don't sample them in mononoke test perf
if we also don't sample here, we will have more insight in those and also we will be able to replay
Reviewed By: clara-9
Differential Revision: D44258777
fbshipit-source-id: 597fc475062b23eb24b0c61024eea9046b808420
Summary: Deduplicate reads of large blobs by using a shared future for any in-flight reads. If another read for the same key comes in after then we will read it again, but there should be at most one read in flight for a particular key at a time.
Reviewed By: Croohand
Differential Revision: D44183928
fbshipit-source-id: d6911422a6e16b3dd246dd48cbce4390e6856940
Summary: Make it possible for tickets to become `'static` by using `std::borrow::Cow` for the borrowed values, and converting to the owned variant if needed. We will use this to shared futures that hold a ticket.
Reviewed By: Croohand
Differential Revision: D44262842
fbshipit-source-id: 2c549b2f2f3724c6485b41463b973273211dfe77
Summary:
The `awaited` boolean on the `Ticket` structure and corresponding `Drop` makes it impossible to move out of the fields of `Ticket`. This is inconvenient, especially since it only enables a unit test check.
Move the field to a structure that's internal to the `Ticket`. We still check for failure to await, as dropping the `Ticket` with a `CheckAwait` that hasn't been marked as awaited will still panic during tests when the inner `CheckAwait` gets dropped.
Reviewed By: Croohand
Differential Revision: D44270321
fbshipit-source-id: 29429c40243b31dcc1a5046969b85fb7a15fd0aa
Summary: This log was useful while we were testing the new commit graph, but now it is too noisy and should be removed.
Reviewed By: liubov-dmitrieva
Differential Revision: D44297423
fbshipit-source-id: c9969d4a0b4709ea2065f19508ae6b51a7ee6baf
Summary: We previously only logged the error code of any failed Python commands. This means we have no insight into why Python commands were failing.
Reviewed By: chadaustin
Differential Revision: D44111246
fbshipit-source-id: 8c2f66457761ddd8b90ff4fb307152cd27ffa00a
Summary:
During checkout, these 2 are called a large amount of time, let's make sure we
have some telemetry so we can understand how much time is spent doing
invalidation during checkout.
Reviewed By: chadaustin
Differential Revision: D44311711
fbshipit-source-id: 5af62fe7fd4b37972458bc545bfaa2f4b4d2ca53
Summary:
This makes lifetime easier to reason about, and allows easier use of
DurationScope (see next diff).
Reviewed By: chadaustin
Differential Revision: D44311712
fbshipit-source-id: 1f9f4cbcc59bafeb464e2b39a248958cedaf9ca6
Summary:
Running the benchmarks/random_writes.cpp benchmark on all power of 2 from 4KiB
to 1MiB shows that 16KiB has the best tradeoff between fast random writes and
fast streaming writes.
Reviewed By: chadaustin
Differential Revision: D44320693
fbshipit-source-id: 2d1839caca3c30acdb6ffc675710a68359d03a76
Summary:
This one slipped through D44263797 and allows for integration tests to be run
with Buck2 on macOS.
Reviewed By: fanzeyi
Differential Revision: D44315942
fbshipit-source-id: d2de0773ba68f13fca9e8d5c067b82653646c757
Summary:
I would like to refer to some performance data of `gca` in the Segmented
Changelog doc.
Initially I was trying to make `drawdag` fast since it can use an elegant
syntax `A001..A999` and can work for both git and hg formats
(`debugexportrevlog` can be used to create repos that the upstream Mercurial
can understand). However it turns out to be a lot of work to make `drawdag`
"just" fast so I ended up writing dedicated scripts to create repos with
millions of commits.
Reviewed By: muirdm
Differential Revision: D43373717
fbshipit-source-id: 5ea1753027cb7487099ed00007e4cc0beae8a09a
Summary:
Parents outside the requested set mislead PathHistory to skip a segment
incorrectly. Detect that and do not skip segment in this case.
Reviewed By: zzl0
Differential Revision: D44305349
fbshipit-source-id: 9c75c4f830e97ff0f54c992bef3ac3aa331edbe1
Summary:
Previously the pathhistory tests always use the entire commit graph to
initialize PathHistory. This diff adds a way to test it on a subset of
the commit graph.
Reviewed By: zzl0
Differential Revision: D44305350
fbshipit-source-id: 955af0243a89f4f6e8f13da970170021c51dafbf
Summary:
Sometimes the tests run so fast that we're being served stale version of the
bookmarks.
This may cause the test to occasionally take the fast path where it expects to take the slow path, and has caused this test to be flaky since its creation (although not always flaky enough to be marked as such).
Differential Revision: D44298928
fbshipit-source-id: 046ce5bc4e005f3d443794202352bfdc5c80895a
Summary: Youssef accidentally discovered that config hot reloading was not working for SCS server when working on an oncall task. He then went ahead and fixed that issue in D44206206. However, we did not get any notification or alert about this problem and it would have stayed hidden much longer had we not accidentally detected it. This diff adds a liveness metric for the `ConfigUpdater` which will keep publishing every 5 mins as long as the `MononokeConfigs` object is live.
Reviewed By: YousefSalama
Differential Revision: D44293235
fbshipit-source-id: 6f54831c8b1f2154a23c021a7d2d8f01615a7d70
Summary:
A gotham request consists of two parts. The first part is a future, in which the handler resolves the request and produces the response headers. The second part is the body which may optionally be a stream. We added collection of future stats for the first part, but the majority of work may actually be done when computing the stream for the body.
In this diff we add stats collection for that body stream, if it is used. Since this is separate from the future stats we already collected, we prefix the scuba columns with "stream_" to make it clear this is the stream.
Reviewed By: liubov-dmitrieva
Differential Revision: D44260233
fbshipit-source-id: 9820d3f30fb147740a95d8afc2fc1bf1fcc16aab
Summary: The counter should be has_overlay_file, not has_overlay_dir.
Reviewed By: fanzeyi
Differential Revision: D44274455
fbshipit-source-id: 55563a0da14ec0ef1a089e3073cc8fd290d2f35d
Summary: Add this so we know where events in edenfs_events came from (Python CLI, Rust CLI, or the daemon).
Reviewed By: mshroyer
Differential Revision: D44044474
fbshipit-source-id: e23af5e186121657dfabf25c5c50882cc9aec923
Summary:
Removes need for patch.
It had already been updated to 3.4 but this just formalizes it.
Reviewed By: jasonwhite
Differential Revision: D44274378
fbshipit-source-id: d17ecbbee06c7df30f689eb859fb31fdf07d8d44
Summary:
Generelizing the Hash structure and adding Hash32 support.
Also, added a few basic keyed blake3 methods for that.
This is mainly a preparation to start supporting blake3.
bypass-github-export-checks
Reviewed By: chadaustin
Differential Revision: D44173515
fbshipit-source-id: 87c55d47dabe50c7104f09ee0078f29513068862
