Summary:
This change move logic associated with mercurial changeset derivation to `mercurial_derived_data` crate.
NOTE: it is not converted to derived data infrastructure at this point, it is a preparation step to actually do this
Reviewed By: farnz
Differential Revision: D23573610
fbshipit-source-id: 6e8cbf7d53ab5dbd39d5bf5e06c3f0fc5a8305c8
Summary:
The Mac integration test workflow already installs a modern curl that fixes https://github.com/curl/curl/issues/4801, but it does so after "hg" is built, so "hg" uses the system curl libraries, which fails when used with a certificate not present in keychain.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/53
Reviewed By: krallin
Differential Revision: D23597285
Pulled By: lukaspiatkowski
fbshipit-source-id: a7b8b6ae55ce338bfb9946a852cbb6b929e73203
Summary:
There are blobs that fail to scrub and terminate the process early for a variety of reasons; when this is running as a background task, it'd be nice to get the remaining keys scrubbed, so that you don't have a large number of keys to fix up later.
Instead of simply outputting to stdout, write keys to one of three files in the format accepted on stdin:
1. Success; you can use `sort` and `comm -3` to remove these keys from the input dat, thus ensuring that you can continue scrubbing.
2. Missing; you can look at these keys to determine which blobs are genuinely lost from all blobstores, and fix up.
3. Error; these will need running through scrub again to determine what's broken.
Reviewed By: krallin
Differential Revision: D23574855
fbshipit-source-id: a613e93a38dc7c3465550963c3b1c757b7371a3b
Summary:
With three blobstores in play, we have issues working out exactly what's wrong during a manual scrub. Make the error handling better:
1. Manual scrub adds the key as context for the failure.
2. Scrub error groups blobstores by content, so that you can see which blobstore is most likely to be wrong.
Reviewed By: ahornby, krallin
Differential Revision: D23565906
fbshipit-source-id: a199e9f08c41b8e967d418bc4bc09cb586bbb94b
Summary:
Sorting bookmark names can be expensive for the MySQL server. As we
don't rely on the ordering of bookmark names when requesting all bookmarks,
remove the sorting.
I've not modified the `Select.*After` queries as they are used for pagination,
which does rely on the order of bookmark names. Further, any queries for
bookmarks that have a limit other than `std::u64::MAX` will remain sorted.
Reviewed By: ahornby
Differential Revision: D23574741
fbshipit-source-id: 79e07b64bb8bb34229c429bdf885c5144963f140
Summary:
The test-blobimport.t creates few files that are conflicting in a case insensitive file system, so make them differ by changing number of underscores in one of the files.
test-pushrebase-block-casefolding.t is directly testing a feature of case sensitive file system, so it cannot be really tested on MacOS
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/49
Reviewed By: farnz
Differential Revision: D23573165
Pulled By: lukaspiatkowski
fbshipit-source-id: fc16092d307005b6f0c8764c1ce80c81912c603b
Summary:
Previously we were not logging a redacted access if previous access was logged
less < MIN_REPORT_TIME_DIFFERENCE_NS ago. That doesn't work well with our
tests.
Let's instead add a sampling tunable.
Reviewed By: krallin
Differential Revision: D23595067
fbshipit-source-id: 47f6152945d9fdc2796fd1e74804e8bcf7f34940
Summary: Moving some of the functionality (which is required for mercurial changeset derivation) into a separate crate. This is required to convert mercurial changeset to derived data to avoid circular dependency it would create otherwise.
Reviewed By: StanislavGlebik
Differential Revision: D23566293
fbshipit-source-id: 9d30b4b3b7d8a922f72551aa5118c43104ef382c
Summary:
In some very rare scenario, we're seeing bind fail due to WSAEADDRINUSE, and
I'm suspecting that a connect race is going on, where connect creates the file
on disk while a bind is ongoing. Since at this point there is no way for
another EdenFS process to be alive due to the fs lock being held, there is no
way for the socket to be already bound to another EdenFS process. Thus, we can
remove the special case.
Reviewed By: genevievehelsel
Differential Revision: D23591009
fbshipit-source-id: 094a7926aa9ac45ea9dea5646000aa287a9b5b7a
Summary: No more tokio-core! More `async/await`.
Reviewed By: kulshrax
Differential Revision: D23586509
fbshipit-source-id: b2e766ddb7575bc96963432f0c8582b4370b19aa
Summary: These are now not working on mac due to the Catalina upgrade, so fixing here to be multi platform compatible by just following our same steps for path resolution.
Reviewed By: wez
Differential Revision: D23582015
fbshipit-source-id: e813986f5523bb721a4f01679559380c2348b98c
Summary: This fixes the `The program issued a command but the command length is incorrect` error we have been seeing in EdenFS Windows.
Reviewed By: wez
Differential Revision: D23589264
fbshipit-source-id: 72478a653ff07fca81c163bab12c0fd33f6a0a67
Summary:
This diff adds a `SocketTransport` implementation that no longer uses legacy `tokio-core` based futures but `tokio-tower` and `tower-service` for processing Thrift requests.
The old implementation is renamed to `SocketTransportLegacy` for better transitioning.
Reviewed By: dtolnay
Differential Revision: D20019196
fbshipit-source-id: 3bee684e9254bf1a81669ef0d2c2262a55e75daa
Summary: This is needed in a later diff that requires "codec" feature from `future-util`.
Reviewed By: dtolnay
Differential Revision: D23575630
fbshipit-source-id: e9cdf11b6ec05e5f2744da6b6efd8cb7bf08b212
Summary:
One of the difference between linux/macOS and Windows is that `edenfsctl` needs
to be used while `eden` works on the other platforms. This forces both users to
change their habit, and all the scripts at FB to be changed to take edenfsctl
into consideration.
Reviewed By: chadaustin
Differential Revision: D23550567
fbshipit-source-id: de2b0853137409e595a0012ca9286c37208b98a1
Summary:
This method is a future of synced-commit-mapping: there can be multiple query
results and we should make a decision of whether it is acceptable for the
business logic in the business logic, rather than pick a random one.
In later diffs I will introduce the consumers for this method.
Reviewed By: mitrandir77
Differential Revision: D23574165
fbshipit-source-id: f256f82c9848f54e5096c6e50d42600bfd260081
Summary:
Another preparatory step for the actuall mapping model fix. This just renames
`get` method into a `get_one` to emphasize it's use-case and to ease the search later.
At the end of this change, I expect there to be no use-cases for `get_one` and expect is to be gone.
Reviewed By: mitrandir77
Differential Revision: D23574116
fbshipit-source-id: f5015329b15f3f08961006607d0f9bf10f499a88
Summary: This is just preparatory extraction to make further work more convenient.
Reviewed By: mitrandir77
Differential Revision: D23574077
fbshipit-source-id: 352ca8ac62bae4fd8fcb980da05c95ce477a414e
Summary:
In order to keep the hgcache size bounded we need to keep track of pack
file size even during normal operations and delete excess packs.
This has the negative side effect of deleting necessary data if the operation is
legitimately huge, but we'd rather have extra downloading time than fill up the
entire disk.
Reviewed By: quark-zju
Differential Revision: D23486922
fbshipit-source-id: d21be095a8671d2bfc794c85918f796358dc4834
Summary:
In a future diff we'll add logic to delete old pack files. We'll want
to use this pack iteration code, so let's move it to a function.
Reviewed By: quark-zju
Differential Revision: D23486920
fbshipit-source-id: 5f872e946ffe816289c925dd2e03c292e29da5af
Summary:
As the repository grows the opportunity for large downloads increases.
Today all writes to data packs get sent straight to disk, but we have no way to
prevent this from eating all the disk.
Let's automatically flush datapacks when they reach a certain size (default
4GB). In a future diff this will let us automatically garbage collect data packs
to bound the maximum size of packs.
Rotatelog already have this behavior.
Reviewed By: quark-zju
Differential Revision: D23478780
fbshipit-source-id: 14f9f707e8bffc59260c2d04c18b1e4f6bdb2f90
Summary:
Top of the stack, last process to implement for full implementation on the `edenfs_restarter` code path. Again, we don't have `/proc/pid/stat`, so instead we use datetimes to calculate the start time of the process since epoch in seconds.
Note here that none of these manual runs look at the versions installed/running, and that is because that kind of manual testing only works if I build and manually deploy an rpm.
Reviewed By: fanzeyi
Differential Revision: D23443268
fbshipit-source-id: 370426f2cc0d5209b96615f2c017ac08acf266fc
Summary: Implementation for making a getExportedValues() thrift call to the process if `get_build_info_from_pid()` is unavailable (which is the case on mac).
Reviewed By: fanzeyi
Differential Revision: D23442884
fbshipit-source-id: 011bcb63832226e2dabd5be60dd30e13f2481dcc
Summary: Since mac does not have `/proc/`, instead here we call `ps` commands and parse the output. This mirrors the same logic used up for linux.
Reviewed By: wez
Differential Revision: D23442710
fbshipit-source-id: ed5160e4dd52884e5752949a4fb2077690906ac4
Summary:
See D23538897 for context. This adds a killswitch so we can rollout client
certs gradually through dynamicconfig.
Reviewed By: StanislavGlebik
Differential Revision: D23563905
fbshipit-source-id: 52141365d89c3892ad749800db36af08b79c3d0c
Summary:
Like it says in the title, this updates remotefilelog to present client
certificates when connecting to LFS (this was historically the case in the
previous LFs extension). This has a few upsides:
- It lets us understand who is connecting, which makes debugging easier;
- It lets us enforce ACLs.
- It lets us apply different rate limits to different use cases.
Config-wise, those certs were historically set up for Ovrsource, and the auth
mechanism will ignore them if not found, so this should be safe. That said, I'd
like to a killswitch for this nonetheless. I'll reach out to Durham to see if I
can use dynamic config for that
Also, while I was in there, I cleaned up few functions that were taking
ownership of things but didn't need it.
Reviewed By: DurhamG
Differential Revision: D23538897
fbshipit-source-id: 5658e7ae9f74d385fb134b88d40add0531b6fd10
Summary:
We recently switched fbsource from using `.eden-redirections`
to manage some common buck-out redirections to relying on buck's
`eden redirect add PATH bind` invocation.
As part of this, a few users have run into an issue where buck hasn't
realized that the buck-out directory was unmounted and proceeds to
try to build. It assumes the `eden redirect add PATH bind` will
restore a missing mount, but that command skips doing any real work
in the case that the mount is configured even if it is unmounted.
This commit aims to improve the UX around this situation by:
* When removing a redirection, test to see if buckd is running for
the containing path and stop it.
* When running `eden redirect add PATH bind`, if the path is configured
but not mounted then fixup the mount. (Previously we'd just print
that we're skipping it)
Reviewed By: genevievehelsel
Differential Revision: D23502306
fbshipit-source-id: 56e823f0b59981c19d0c44723948bd84d6d9008a
Summary:
I ran into a couple of problems with this as part of
looking at some improvements to `eden redirect` later in this
stack:
* There's currently an issue with the BUCKVERSION=last handling
in our internal version of buck on macos, so sidestep it for now.
* We have a bad interaction between the environment set up to
run edenfsctl.par and a different par file used by the FB internal
buck wrapper script that causes it to fail on startup.
This commit cleans up the environment to compensate for these issues.
Reviewed By: genevievehelsel
Differential Revision: D23502307
fbshipit-source-id: 34b5099529dcc5f2b2d638bcb333e4dd00211766
Summary:
Generated by formatting with rustfmt 2.0.0-rc.2 and then a second time with fbsource's current rustfmt (1.4.14).
This results in formatting for which rustfmt 1.4 is idempotent but is closer to the style of rustfmt 2.0, reducing the amount of code that will need to change atomically in that upgrade.
---
*Why now?* **:** The 1.x branch is no longer being developed and fixes like https://github.com/rust-lang/rustfmt/issues/4159 (which we need in fbcode) only land to the 2.0 branch.
---
Reviewed By: StanislavGlebik
Differential Revision: D23568780
fbshipit-source-id: b4b4a0aa683d236e2fdeb5b96d723ac2d84b9faf
Summary:
This test fail on sandcastle because the last two lines are not showing up.
I have a hunch that the last two lines just weren't flushed, and this diff
attempts to fix it.
Reviewed By: krallin
Differential Revision: D23570321
fbshipit-source-id: fd7a3315582c313a05e9f46b404e811384bd2a50
Summary: When we imported a repo (T71717570), we received a network connect error after querying a lot from graphql. I am not sure, if it's because of the frequent amount of queries, but just to be on the safe side, I increased the default sleep time between queries.
Reviewed By: krallin
Differential Revision: D23538886
fbshipit-source-id: 6a84f509e5e19f86880d3f8c6413f2f47e4a469b
Summary:
Generated by formatting with rustfmt 2.0.0-rc.2 and then a second time with fbsource's current rustfmt (1.4.14).
This results in formatting for which rustfmt 1.4 is idempotent but is closer to the style of rustfmt 2.0, reducing the amount of code that will need to change atomically in that upgrade.
---
*Why now?* **:** The 1.x branch is no longer being developed and fixes like https://github.com/rust-lang/rustfmt/issues/4159 (which we need in fbcode) only land to the 2.0 branch.
---
Reviewed By: zertosh
Differential Revision: D23568779
fbshipit-source-id: 477200f35b280a4f6471d8e574e37e5f57917baf
Summary: Add Scuba logging using `ScubaMiddleware` from `gotham_ext`. Each request will be logged to the Scuba dataset specified by the `--scuba-dataset` flag, as well as optionally to the log file specified by `--scuba-log-file`.
Reviewed By: sfilipco
Differential Revision: D23547668
fbshipit-source-id: e6cd88ad729a40cf45b63538f7481ee098ea12dc
Summary: Import middleware directly from `gotham_ext` rather than relying on reexports in the `middleware` module.
Reviewed By: farnz
Differential Revision: D23547320
fbshipit-source-id: e64a8acff55445a646b0a1b3b1e71cf6606c3d02
Summary:
Move `ScubaMiddleware` out of the LFS server and into `gotham_ext`.
This change required splitting up the `ScubaKey` enum to separate generally useful column names (e.g., HTTP columns that would be applicable to any HTTP service) from LFS-specific columns. `ScubaMiddlwareState` has been modified to accept any type that implements `Into<String>` as a key, and the `ScubaKey` enum has been split up into `HttpScubaKey` (in `gotham_ext`) and `LfsScubaKey` (in `lfs_server`).
The middleware now takes a type parameter to specify a "handler" (implementing the new `ScubaHandler` trait) which allows the application to add application-specific Scuba columns in addition to the default columns. The application-specific columns will be added immediately prior to the sample being logged.
Reviewed By: krallin
Differential Revision: D23458748
fbshipit-source-id: 3e99f3e0b5d3475a4f5ac9eaefade2eeff12c2fa
Summary:
This makes it easy for `metaedit` to be used by automation. Provided
with a simple JSON file with hash->{user, message} mapping metaedit will
do all of its work without any prompts.
Reviewed By: quark-zju
Differential Revision: D23545527
fbshipit-source-id: 18763ecacff9143b9ad492faf654b176b0f86d1f
Summary:
This diff is more complex than I wished for it as originally I didn't take into
account direcotries when designing `commit_compare` method.
Reviewed By: StanislavGlebik
Differential Revision: D23541892
fbshipit-source-id: 0e2b2abf7b3c541529d9881e48a575239374040f
Summary: We need that to replace similar feature in SCMQuery
Reviewed By: StanislavGlebik
Differential Revision: D23541893
fbshipit-source-id: 3dd6357ea834337a81216e24cb132e23b01bc77d
Summary:
Hooks have been recently made public. Remove from list of excluded tests the ones that were blocked by missing hooks and fix them up.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/48
Reviewed By: farnz
Differential Revision: D23564883
Pulled By: lukaspiatkowski
fbshipit-source-id: 101dd093eb11003b8a4b4aa4c5ce242d9a9b9462
Summary:
The "meaningfulparents" concept is coupled with rev numbers.
Remove it. This changes default templates to not show parents, and `{parents}`
template to show parents.
Reviewed By: DurhamG
Differential Revision: D23408970
fbshipit-source-id: f1a8060122ee6655d9f64147b35a321af839266e
Summary:
Those are new tests that use functionality not compatible yet with OSS.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/47
Reviewed By: chadaustin
Differential Revision: D23538921
Pulled By: lukaspiatkowski
fbshipit-source-id: c512a1b2359f9ff772d0e66d2e6a66f91e00f95c
