Summary:
Will be used in tests to check what data is inserted into IdMap during clone or
pull.
Reviewed By: andll
Differential Revision: D30314054
fbshipit-source-id: 8a58473ef3d85263985c1be8a6c57a9a10b3fdb4
Summary:
In rare cases, the local IdMap might miss parents of draft roots.
This provides a way to emulate that in tests.
Reviewed By: andll
Differential Revision: D30314050
fbshipit-source-id: 09983604bf3259e62a41224a579774265ae0b272
Summary: This would help investigate some errors.
Reviewed By: andll
Differential Revision: D30314057
fbshipit-source-id: 070e75cf0a39180c544b49c9ed1292d8536040fa
Summary:
- This diff adds validation so that changesets that are not snapshots cannot have untracked or missing files.
- It removes the THIS IS A SNAPSHOT commit message.
- It makes the snapshot created by `hg snapshot createremote` be an actual snapshot.
Differential Revision: D30159184
fbshipit-source-id: 976968c0c2222f950a4a937aa805b25dc07c9207
Summary:
This diff adds some data to BonsaiChangeset that tells whether it is a snapshot or not.
For now, this marks every changeset as not being a snapshot. The next diff will add validation to snapshots, some tests, and mark the current `snapshot createremote` command as uploading snapshots.
Reviewed By: markbt
Differential Revision: D30158530
fbshipit-source-id: 9835450ac44e39ce8d653938f3a629f081247d2f
Summary:
This diff makes the snapshot command upload all types of files that were missing (added/untracked/missing/deleted), using the new types of file changes added on the previous diff.
Next steps:
- Add some indicator to Bonsai Changeset saying it is a snapshot. Verify only snapshots can have certain file changes (untracked/missing).
- Upload the files and the changeset inside an ephemeral bubble instead of in the main blobstore
- Start writing the `snapshot restore` command
Differential Revision: D30137673
fbshipit-source-id: 555238f1d64a5438cde35a843043884a939de4fe
Summary: I have my editor set up to format on save - let's unify this on the standard FB format for thrift files, so that I don't create junk.
Reviewed By: ahornby
Differential Revision: D30285082
fbshipit-source-id: 17b09635a2473174a92e29bb042432dbac44865a
Summary:
The current Mononoke Blobstore Trace scuba table is used with idea of having a record per blobstore and operation. This diff adds logging to the new scuba table of the combined multiplexed operations' outcome, like time spent on the `put` including sync-queue and blobstore write or tracking record of the "some failed others none" cases in `get/is_present`.
This helps to see the real time spent on writes and reads and to assess the impact of changes coming in `get` and `is_present`.
Reviewed By: ahornby
Differential Revision: D30248284
fbshipit-source-id: f79050ced32ba77bd2e220e242407bcd711a9b6d
Summary:
On Windows, it's hard to rename `.hg/store/segments/v1` away because the files
are (too easily) being currently used. Instead, let's write the new lazy
segments to `segments/v1next`, and try to rename it back later on follow-up
commands.
Reviewed By: DurhamG
Differential Revision: D30118952
fbshipit-source-id: e3edb588dccf1acb5f4ed106bbb979bcc8c6e67e
Summary:
OD was running `hg rage --preview` in the background periodically and that has caused edenfsctl creating paste on the user's behalf without them actually knowing.
This will make Mercurial to collect EdenFS rage in dry-run mode (i.e. do not create paste).
Reviewed By: quark-zju
Differential Revision: D30288390
fbshipit-source-id: 1e7f25648ca1b76f24264ee13ed98cf974148b0f
Summary:
Autocargo only allows 1 rust-library per Cargo.toml, but right now we have 3
per Thrift library so that doesn't work:
https://www.internalfb.com/intern/sandcastle/log/?instance_id=27021598231105145&step_id=27021602582167211&step_index=13&name=Run%20config
There's little benefit in Autocargo-ifying those rules anyway since they're of
use to Thrift servers and this doesn't work at all in our OSS builds, so let's
just see if we can just noop them. That'll make the crate not exist at all as a
dep, but even considering that it exists only to link to a C++ library that
Autocargo doesn'tk now how to build anyway, that seems OK?
drop-conflicts
Reviewed By: markbt
Differential Revision: D30304720
fbshipit-source-id: 047524985b2dadab8610267c05e3a1b3770e84e6
Summary:
This is basically a refactor.
Before this diff, `bubble.handle(main)` could be used to access things in bubble with fallback. With this diff, `bubble.wrap_repo_blobstore(main)` can be used for the same effect.
The difference is **the type**, which now is `RepoBlobstore` instead of `EphemeralHandle`. Both are blobstores and work the same way for fetching/putting, but on the following diffs I will want to replace some code (e.g. that creates a changeset) to use the ephemeral blobstore for snapshots, and in order to reuse the same code (which expects `RepoBlobstore`), we need the change of types.
This is part of BlobRepo refactoring as well, as what I'm gonna do is replace BlobRepo with a different facet container that has a RepoBlobstore inside.
Reviewed By: markbt
Differential Revision: D30282624
fbshipit-source-id: 4132797104ecd2596e7da91b1daacc1c6fc85934
Summary: Allow `manifest-tree::Entry` to be constructed directly by other crates, and introduce a `TryFrom` conversion to `manifest::List::Directory`. This will be used by the scmstore `BackingStore` implementation to avoid having to rely on the `TreeStore` trait, which does not support batching.
Reviewed By: kulshrax
Differential Revision: D30282738
fbshipit-source-id: 590350dd53217fa8a181e91b194abca753a8adbe
Summary: Add a new results adapter for `FileStore` (which will probably become the primary method of consuming batch errors in the future), which iterates over all keys, and explicitly encodes not found with `Option`, separate from the outer `Result`.
Reviewed By: kulshrax
Differential Revision: D30254000
fbshipit-source-id: 074c6ac4c279fdf72078dca17231e58d0c704956
Summary:
Add a `local` method to `TreeStore` like the one on `FileStore` which returns a `TreeStore` with only the local subset of backends.
Required for `BackingStore` implementation.
Reviewed By: kulshrax
Differential Revision: D30253980
fbshipit-source-id: 142f2d88454826ff9cb9c34f30b7d21bf62b297c
Summary: Add a new method, `edenapi_override`, to the `TreeStore` and `FileStore` builders for use in the `BackingStore` implementation.
Reviewed By: kulshrax
Differential Revision: D30253837
fbshipit-source-id: 4a42e83621fb2634024e4ee8529c26aeae0256a4
Summary:
Instrument all the fetching-related legacy API implementations (ie, not `get_logged_fetches` and `get_shared_mutable`) to track the number of calls, keys, and single-key calls.
This introduces an additional lock acquisition to each of these implementations (and it'd be awkward to merge it with the one in `fetch`), but I think that's probably fine.
For APIs which do not support a variable number of keys, I use `.call(0)` so we simply track the total number of API calls.
Reviewed By: DurhamG
Differential Revision: D30003444
fbshipit-source-id: 8756d2669ca038b3f6a08e211e44e8ccb9251312
Summary:
Adds tracking of FileStore write errors.
Write batches will still abort after the first failure as written.
Reviewed By: DurhamG
Differential Revision: D29997203
fbshipit-source-id: e1cc2ffc4a8d97ca935a7fc9aab30bde3dc548b2
Summary: Refactor the `FileStore::write_batch` method, extracting the various write cases into other methods.
Reviewed By: DurhamG
Differential Revision: D29997096
fbshipit-source-id: 8efac4b2efa2e2225d39583a5c6893efc5096fec
Summary:
Previously errors from `prefetch` weren't counted. With this change I move calculating the pending keys out of the `inner` method, making it easier to handle all batch error metrics from `?` calls right beside the existing `other_errors` call.
This is probably how I'll structure the other methods too, though it's still a little subtle - currently the batch errors only come from `prefetch`, but if batch errors were returned after the `drain` call (which always drains the vector, even if you don't consume the whole iterator), the second `pending.len()` call could incorrectly report 0. To address this, I could separate out the "single key fetching" part of the method, or just let it be understood that you should repeatedly call `pop` in such cases.
Reviewed By: DurhamG
Differential Revision: D29995789
fbshipit-source-id: 09753081c5e7076b83cb26fdddb949205ad7cd43
Summary: Previously, contentstore metrics were tracked as a special case. Now that I'm tracking metrics on each backend, I've standardized the contentstore metrics to match.
Reviewed By: DurhamG
Differential Revision: D29995500
fbshipit-source-id: 3133dca5912d9cd6c26bd88d9aeb6a572f6eca40
Summary:
Added try_continue_traversal_when_no_parents function. For now it continues
traversal only if we deleted file manifest found something, but it will be
extended in the next diff to also use mutable renames.
Differential Revision: D30279931
fbshipit-source-id: b2cdae62d7841cfa0834ac1dd280ffb8dafa43ef
Summary: This adds types to FileChange thrift and rust structs to deal with additional possible snapshot states, that is, untracked and missing files. Conflicted stuff not added yet.
Reviewed By: StanislavGlebik
Differential Revision: D30103162
fbshipit-source-id: 59faa9e4af8dca907b1ec410b8af74985d85b837
Summary:
Increase [CURLOPT_LOW_SPEED_TIME](`https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html`) because users were hitting low speed timeouts during normal usage.
2 minutes was arbitrarily chosen, but it seems unlikely that we'd have situations where connections legitimately sit idle for more than 2 minutes. (If it turns out we do, that would be a bit concerning.)
This diff also updates the config option to specify the time in seconds instead of milliseconds because libcurl only supports second-level precision when specifying `CURLOPT_LOW_SPEED_TIME`. The config option name has been updated to explicitly include the unit to avoid confusion.
Differential Revision: D30272017
fbshipit-source-id: 0ae5e912c0dafc897fe19de3ea8e5aeb4ed8459c
Summary:
We'd like to have support for mutable renames i.e. make it possible to mark a
file or directory as renamed from another file/directory.
This diff adds a mysql table which will store these renames. A few notes:
1) Note that table stores `src_unode_id` - this is an optimization to make
fastlog traversal after renames faster.
2) I've been considering whether we want to use "insert or ignore" or "insert
... on duplicate key update". I opted for the latter - with "insert or ignore"
we'd need to process additional case when insert was ignored, and that would
make implementation harder. Besides I think that overwriting should be fine given
that we can always change mutable renames later
Differential Revision: D30277300
fbshipit-source-id: 35e5cab79f0db5a1ecf28a8a9b9f9b86d0f42fb6
Summary:
The `async-compression` crate is currently only used by Mononoke for zstd and gzip compression. We'd like to use it in Mercurial too (primarily for zstd), but unlike Mononoke, Mercurial needs to build on all platforms.
One of `async-compression`'s dependencies, `lzma-sys`, has a complicated build script which required extensive fixups (see D21455819). Unfortunately, `lzma-sys` currently does not build on Windows with Buck because the [fixups.toml](https://www.internalfb.com/code/fbsource/[ba27fac3d5b5]/third-party/rust/fixups/lzma-sys/fixups.toml?lines=35) and [a fixup header file](https://www.internalfb.com/code/fbsource/[468048d6e50b]/third-party/rust/fixups/lzma-sys/include/config.h?lines=34) both enable pthreads, which causes conditional compilation to attempt to include POSIX dependencies. The exact error is:
```
third-party/rust/vendor/lzma-sys-0.1.16/xz-5.2/src/common\mythread.h:103:10: fatal error: 'sys/time.h' file not found
```
Given that this crate is currently only used by Mercurial and Mononoke, and only for the zstd algorithm in practice, as a quick workaround let's just disable LZMA and XZ support. This is unfortunate; it would be better to figure out how to make the fixup work correctly, but Buck on Windows is such a niche use case at the moment that I'm not really sure where to begin.
Reviewed By: dtolnay
Differential Revision: D30271553
fbshipit-source-id: 76560c39b6f2d8750fa34c30ccb3e7db734e92a7
Summary: This diff adds a CLI option to be able to override the manifold request priority for a particular job or command line run.
Reviewed By: HarveyHunt
Differential Revision: D30277209
fbshipit-source-id: 58217c11234133dfc68e11b230d99066dd783600
Summary: Clean Up: Once we migrated to Mononoke we can remove the infinitepush path because it creates extra complexity in remotenames, infinitepush and commitcloud extensions
Reviewed By: yancouto
Differential Revision: D30277402
fbshipit-source-id: 1708a71eb33dc9a9b509bdc1b03bed08073b6152
Summary:
- This diff makes EphemeralBlobstore use RepoBlobstore under the hood, which gives it redaction for free.
- It also refactors and simplifies the ephemeral blobstore code, removing unnecessary separation between repo-less and repo-full ephemeral blobstore.
- It will be useful later on to be able to have a RepoBlobstore so we can transparently replace code that deals with blobrepo to also work with ephemeral stuff.
Differential Revision: D30229056
fbshipit-source-id: 956f1e8fecc2b3fa518eb11268fbbbfd27c4f5dd
Summary:
support streaming for file uploads rather than buffering the whole content
This is a preferable way for big files. We are currently using LFS endpoint instead for big files but not in snapshots.
Also, it we enable streaming, we have an option not to use LFS endpoint for file uploads in the future.
**[Land after hg release with D30100887 (fe8ed9d28c) has been fully rolled out]**
Reviewed By: yancouto
Differential Revision: D30158390
fbshipit-source-id: b62c498b8bdf23a5f413f6e4b71d7433906e4611
Summary:
for now this changes:
```
struct FileChange {
...stuff
}
fn f(x: Option<FileChange>)
```
to
```
struct TrackedFileChange {
...stuff
}
enum FileChange {
TrackedChange(TrackedFileChange),
Deleted,
}
fn f(x: FileChange)
```
This makes it much clearer that `None` actually means the file was deleted. It will also be useful as in the next diff I will add more stuff inside FileChange (for untracked changes), and this refactor will make it easy.
(The refactor from using `Option` to putting it all inside the enum isn't really necessary, but IMO it looks much clearer, so I did it.)
Reviewed By: StanislavGlebik
Differential Revision: D30103454
fbshipit-source-id: afd2f29dc96baf9f3d069ad69bb3555387cff604
Summary:
The time between logging total_unbundles and the actual successes is the time of the unbundle operation, which may be long.
This makes the alarm on D30222804 much less accurate, as success and fail for the same operation might fall in different buckets.
This diff changes two things:
- total_unbundles are logged at the end of the unbundle operation, which should make tracking more accurate when compared against successes.
- resolver_error is now logged in more cases that would previously error but not be logged
I created a wrapper function in order to make sure it always logs, as before it could not log if there were some early errors.
Differential Revision: D30248117
fbshipit-source-id: 4ec0c148dd7aa818b6d204fafecacacf4d267be7
Summary:
RepoBlobstore already has all the information necessary for rebuilding itself, we don't need to pass in `repo_id` again.
This is easier to use and less error-prone.
Reviewed By: markbt
Differential Revision: D30227978
fbshipit-source-id: b73407d5f022ce5614ee2fa9734f5a8b0c860fe7
Summary:
fix remote lookup for every bookmark
Some users have lots of local bookmarks and remote bookmarks, so the current logic would perform tons of sequential remote lookups via edenapi (due to segmented changelog) just to maintain correct omission state for local and remote bookmarks even if nothing has been changed.
Reviewed By: markbt
Differential Revision: D30247185
fbshipit-source-id: 7c7fd8491ff0bd67827c2a01eb852322ba15f994
Summary:
This diff cleans up all remaining places that Thrift metadata is being created as a raw pointer, which had to be converted unsafely to UniquePtr in D30180770 (ff5931b944).
It also eliminates all the places that definitions of `MetadataFunc` and `RustThriftMetadata` were duplicated across the codebase. It would have been UB if any of these were to fall out of sync, as I discovered when trying to adjust the representation of RustThriftMetadata in D30180770 (ff5931b944).
Reviewed By: guswynn
Differential Revision: D30182979
fbshipit-source-id: 3313440313f28863ac378986c04522d358cb4fd5
Summary:
The port argument to the `ServiceFramework` constructor has been totally unused since D28431427. You can see all of the call sites in this codemod are *always* first passing the port to `ThriftServerBuilder::with_port`, then passing the resulting ThriftServer to a `ServiceFramework` constructor, so ServiceFramework can just obtain the correct port out of the given ThriftServer.
---
API before:
```
impl ServiceFramework {
pub fn from_server(name: &str, server: ThriftServer, port: u16) -> Result<Self>;
pub fn from_primary_server(name: &str, server: ThriftServer, port: u16) -> Result<Self>;
}
```
API after:
```
impl ServiceFramework {
pub fn from_server(name: &str, server: ThriftServer) -> Result<Self>;
pub fn from_primary_server(name: &str, server: ThriftServer) -> Result<Self>;
}
```
---
Call site before:
```
let server = runtime.spawn(async move {
let thrift: ThriftServer = ThriftServerBuilder::new(fb)
.with_port(args.port) //<----------
.with_factory(exec, move || service)
.build();
let mut svc_framework =
ServiceFramework::from_server("example_server", thrift, args.port)?; //<----------
svc_framework.add_module(BuildModule)?;
svc_framework.add_module(ThriftStatsModule)?;
svc_framework.add_module(Fb303Module)?;
svc_framework.serve().await
});
```
Call site after:
```
let server = runtime.spawn(async move {
let thrift: ThriftServer = ThriftServerBuilder::new(fb)
.with_port(args.port)
.with_factory(exec, move || service)
.build();
let mut svc_framework =
ServiceFramework::from_server("example_server", thrift)?;
svc_framework.add_module(BuildModule)?;
svc_framework.add_module(ThriftStatsModule)?;
svc_framework.add_module(Fb303Module)?;
svc_framework.serve().await
});
```
Differential Revision: D30180773
fbshipit-source-id: 16cf32b582161395eab5af3f8aaef6015e69cd9f
Summary:
The safe signature is possible as of {D30180770 (ff5931b944)} but has been separated out of that diff because it requires this codemod of a large number of downstream service implementations.
---
API before:
```
impl ThriftServerBuilder {
pub unsafe fn with_metadata(mut self, metadata: *mut RustThriftMetadata) -> Self;
}
```
API after:
```
impl ThriftServerBuilder {
pub fn with_metadata(mut self, metadata: UniquePtr<RustThriftMetadata>) -> Self;
}
```
---
Call site before:
```
let thrift = unsafe {
ThriftServerBuilder::new(fb)
.with_port(thrift_options.port)
.with_metadata(create_metadata())
.with_max_requests(thrift_options.max_requests)
.with_factory(exec, move || service)
.build()
};
```
Call site after:
```
let thrift = ThriftServerBuilder::new(fb)
.with_port(thrift_options.port)
.with_metadata(create_metadata())
.with_max_requests(thrift_options.max_requests)
.with_factory(exec, move || service)
.build();
```
Reviewed By: guswynn
Differential Revision: D30180772
fbshipit-source-id: f8137b9f91b7c7b5de5bdee9dfd0b7925399cee2
Summary:
This diff eliminates the unsafe code from the bindgen-based ThriftServer and ServiceFramework binding in favor of a simpler safe binding based on CXX.
Followup codemods:
- {D30180771}
- {D30180772}
- {D30180773}
Reviewed By: Imxset21
Differential Revision: D30180770
fbshipit-source-id: e80f0c36f5a816d85a4810e275a97d402b5db4e4
Summary: They don't have to be required anymore
Reviewed By: markbt
Differential Revision: D30245130
fbshipit-source-id: 6563026f648439e5cda5d0e72ae40c0feec43ad9
Summary:
Whenever a file is modified in an EdenFS mount and a watchman subscription is
active, watchman will be nodified and will issue a getFilesChangedSince Thrift
call. In order to do that, Watchman ends up always re-creating a new connection
to EdenFS, causing the .eden/socket or .eden/config to be re-read in order to
find EdenFS's socket.
For workloads with heavy write traffic to EdenFS, this readlink/read can add
up. On Windows, writing ~2.5GB worth of data lead Watchman to read over 650MB
worth of data from the .eden/config!
Reviewed By: kmancini
Differential Revision: D29508654
fbshipit-source-id: 60440d645340bc4fe16ea9618d7a5080740e4d87
Summary: The journal stream is disconnected at Watchman shutdown, which is the expected behavior. This changes the log level to INFO.
Reviewed By: chadaustin
Differential Revision: D30231657
fbshipit-source-id: 94909daeba786b1bed7497e4a21ffcfc52d6d9cb
