Summary:
We've had a few sevs where expensive derivation for a single repo overloaded
our backends. This diff adds a tunable that makes it possible to quickly cancel
all derivations for a given repo. And one important caveat - it doesn't only
cancel derivations for a given repo, it also cancels the initial commit
traversal stage which can be quite expensive.
Note that it cancells all derivations for a given repo, at the moment there's
no way to cancel deriving just a single derived data type. I'd like to add it,
but in the next diffs since it might require changing the tunables - they don't
support tuning a list of strings for a given repo. Besides, having a way to
quickly disable all derivations for a given repo might be useful anyway.
Reviewed By: farnz
Differential Revision: D29028444
fbshipit-source-id: 559917e0e25eb060fe5be77a412bfec4ac3f1779
Summary: It will be used in the next diff to disable only certain derived data types
Reviewed By: yancouto
Differential Revision: D29031082
fbshipit-source-id: 9621a5eec522e369fef2b78e94599a056eda5317
Summary:
Mercurial has a lot of complex logic to try and create the "best" possible filenode for a given merge commit.
However, it works fine with imperfect filenodes, as long as they are semantically correct. Generate imperfect filenodes to speed up data derivation.
We are importing git repos to use here, which have some significantly mergy history; this change both reduces the number of filenodes we end up creating in data derivation, and speeds up data derivation by an order of magnitude overall.
Reviewed By: StanislavGlebik
Differential Revision: D28903515
fbshipit-source-id: 436ca4f43158533b32cf0d41db72228dda2688c3
Summary: revert the zstd crates back to previous version
Reviewed By: johansglock
Differential Revision: D29038514
fbshipit-source-id: 3cbc31203052034bca428441d5514557311b86ae
Summary:
implement `known` method based on the existing `lookup` API that has been already implemented
this might be useful for eden api pull project / wireproto deprecation, as the pull algorithm queries the server for whether commits are known by the server
Differential Revision: D29030429
fbshipit-source-id: 4d78c8b1e7983d89de917951d9765b1bd1d90fb0
Summary: This will be used in fastpath pull
Reviewed By: quark-zju
Differential Revision: D29010304
fbshipit-source-id: 9a7464fbc42544f8c5cc52f6fa25db5f7b5d6bd9
Summary: idset_to_flat_segments allows to return intersection between flat segments and span set
Reviewed By: quark-zju
Differential Revision: D28980521
fbshipit-source-id: da90c66a021a1e8bb95edbbc2e06cd2a850d7dcd
Summary:
This diff introduces Subspan trait for objects that contains spans and allows to take a 'subspan' of an object
Implementing Subspan trait allows to intersect arbitrary objects and spans
Reviewed By: quark-zju
Differential Revision: D28980523
fbshipit-source-id: 29a2e58ec3c79a838db751d8a37227b4cdaaa372
Summary: Separate `content_metadata` into a new method so that it can be used for both file and tree aux data fetching.
Reviewed By: DurhamG
Differential Revision: D28898226
fbshipit-source-id: 2dbca95fd02384e434ea80cad78793c8d8b2f489
Summary:
It is only useful for Python revlog-based changelog, which was removed by the
previous diff.
Reviewed By: DurhamG
Differential Revision: D29020190
fbshipit-source-id: 3332f1532a510dc72f21d68801e093b5eba10145
Summary:
Modify the status command to prefer comparing content sha256 when comparing files over comparing content directly, using cached aux data when possible and writing newly computed hashes to cache.
A new config option, `scmstore.status`, gates this new behavior.
When `scmstore.status` is enabled, the `status` output should be more accurate, not assuming files with non-matching `HgId`s in an arbitrary pair of revisions are different. As written, this will cause fetching of data that wouldn't otherwise be fetched, but remote fetching of aux data is planned.
This change also doesn't introduce batch fetching, this is planned.
Reviewed By: DurhamG
Differential Revision: D28826219
fbshipit-source-id: dcd253f169723983738148559d61ba655dd453a4
Summary: Introduce a config for enabling separate indexedlogs for storing "aux data" in scmstore. The storage format is not yet finalized, and isn't used in production yet, so we gate the constructing it on a cofig.
Reviewed By: DurhamG
Differential Revision: D28826209
fbshipit-source-id: 1ed8c20329f21b3ced598eea16d9fac0f40112e6
Summary:
Make `doctor` aware of non-revlog changelog and do not hide commits
incorrectly.
Reviewed By: DurhamG
Differential Revision: D29020189
fbshipit-source-id: af7fab75a877997e3f11dd21e506c965da078f83
Summary: The doctor command assumes revlog changelog, which does not cover all setups.
Reviewed By: DurhamG
Differential Revision: D29020188
fbshipit-source-id: 68bb4e20efb49482df84a71ab7d86a20a6fa93ce
Summary: For all clients and tests we're using revlog changelog based on Rust.
Reviewed By: DurhamG
Differential Revision: D29020191
fbshipit-source-id: 44ab603ea07b8891831692285e94ea0e4df849f5
Summary:
Verify is tightly coupled with revlog details, and assumes a non-lazy repo.
Revlog is going away and our repo is lazy typically, so let's just make verify
a no-op.
Reviewed By: DurhamG
Differential Revision: D28974546
fbshipit-source-id: 487dc65ceb9a1e445b58d03a53353a8382aefc08
Summary:
Practically, our client-side revlog changelogs should be non-inline now for a
long time. There is no need to keep the migration logic.
The revlog is being deprecated too so its implementation details (inline) is
going to be irreverent. The related test is then removed.
Reviewed By: DurhamG
Differential Revision: D28974551
fbshipit-source-id: ea456c46dac11d6a8b225c269b49598ab34c2548
Summary: It was only useful server-side and will be incompatible with upcoming changes.
Reviewed By: DurhamG
Differential Revision: D28974549
fbshipit-source-id: 70a715ce170baa78adb8b1fcf7d29e3d1479c05e
Summary: hgsql is irrelevant. Remove more tests that will be broken by upcoming changes.
Reviewed By: DurhamG
Differential Revision: D29019287
fbshipit-source-id: 6fd04d2eb088a0ca9c975b25a4f28a5772f0e088
Summary:
This test uses bundle2 details that are hard to maintain.
Let's just remove the test.
Reviewed By: DurhamG
Differential Revision: D29019286
fbshipit-source-id: a64918736039331bf2fc3cd23e9c67dd77510c22
Summary:
This test is too tricky to maintain with modern setups. Namely, we don't
support strip and are dropping revlog usage. Let's just remove the test.
Reviewed By: DurhamG
Differential Revision: D28974550
fbshipit-source-id: e8d30e726735432820ceaf4ef27d1b83753122a4
Summary:
We no longer uses this code base for hg server logic.
The test requires non-Rust commit backend which will be removed.
Reviewed By: DurhamG
Differential Revision: D28974547
fbshipit-source-id: 433a6697f6cbf08450c43ce810490fcdb53cf718
Summary: It tests revlog details that are going to be irrelevant.
Reviewed By: DurhamG
Differential Revision: D28974552
fbshipit-source-id: 3ff08473be236849442c3c30d5cf1e1c2a1b628d
Summary: It tests revlog details that are going to be irrelevant.
Reviewed By: DurhamG
Differential Revision: D28974548
fbshipit-source-id: a44e97daa24aece446d899e7711a59cb4a133398
Summary: Revise the help text so it matches the latest implementation.
Reviewed By: DurhamG
Differential Revision: D28971683
fbshipit-source-id: 3f8fb7ccc42a71fdb65b87e2b99d06fb347983f3
Summary: This option allows to exclude ignored files, this way if user specifices wildcard patterns in --include, they will still not include ignored files
Reviewed By: mrkmndz
Differential Revision: D29007062
fbshipit-source-id: a8458811b4c16e11a91abdc31967b53c3cdf2ed7
Summary:
Previously, if someone had included a service definition like the following:
```
service Foo {
void match(1: string bar);
}
```
rustc could not compile the generated code because functions like `fn match(...)` would be emitted. This commit emits [raw identifiers](https://rust-lang.github.io/rfcs/2151-raw-identifiers.html) when possible (ie we still mangle `crate`, `self` `super`, and `Self` -- so in the previous example we now emit `fn r#match(...)`). The thrift compiler will also now throw an exception when trying to emit a keyword that participates in name resolution (ie `crate`, `super`, `self`, and `Self`).
This commit also implements the `rust.name` annotation for enums, enum values, fields, functions, types, and typedefs.
Reviewed By: yfeldblum, jsgf
Differential Revision: D28612277
fbshipit-source-id: a86ec1b3fd880c42a82d0e2239f616e3364edcba
Summary: I'm going to touch these files soon, found this easy improvement to do before.
Reviewed By: StanislavGlebik
Differential Revision: D28993265
fbshipit-source-id: 89fe9ac52f6d99e1b5ce7cb24d949e048226436a
Summary:
`df` doesn't exist on Windows, so this part of `rage` isn't populated.
The closest equivalent on Windows is `wmic LogicalDisk`. Use this to query the
free space and size of the disks.
Reviewed By: quark-zju
Differential Revision: D28997337
fbshipit-source-id: 08b3b74d70928f2e9801061f049359a58108f4bf
Summary:
Using a PathMap to hold the list of mounts gives us case sensitivity for free
without needing to have #ifdef _WIN32, we can also avoid copying the paths just
to normalize them.
Reviewed By: fanzeyi
Differential Revision: D28969795
fbshipit-source-id: 859cdc3881e36db120913099ce3f451940fa48a4
Summary:
We're planning on rolling out NFS to our users soon, let's make sure NFS is
enabled.
Reviewed By: singhsrb
Differential Revision: D29002395
fbshipit-source-id: 8351fa45c9bd5e68e162baa80e20bd34564b5ece
Summary:
The testpilot command line getdeps generated was not suitable for tpx
when filtering. tpx would consider the passed test filter an
environment variable.
Reorder a few things to at least make filtering work.
Reviewed By: fanzeyi
Differential Revision: D28976061
fbshipit-source-id: 21c45b3a4a59711a2a5da36a1bd11b7b4871ec5d
Summary:
Turned out we still don't process copyfile attribute correctly. copyfile
creates two overrides like
{
"copyfile source" -> default prefix + "/copyfile source"
"copyfile source" -> "destination"
}
However we also have a check that validates that there no paths (that includes
default path, overrides, linkfiles etc) that are prefixes of each other. And
this check fails in the copyfile case, even though the path maps to exactly the
same path as if default prefix was applied.
Let's fix it. Also note that we haven't seen this failure in the integration
test because we didn't run verify_paths in tests. This diff fixes it as well.
Reviewed By: mitrandir77
Differential Revision: D28992456
fbshipit-source-id: 5fd993914b189cf768ba03010194b1c26026f7a8
Summary:
This makes sure that there is no caching when fetching skiplist from blobstores, by creating a blobstore that bypasses caching and using that on the creating of skiplists.
Because of D28898839 (6f05b85534) and stack, this is the only place that creates skiplists, so all creations do not use cache.
The alternative solution is D28901116, which is incomplete as I went with this one as it's much smaller, but I'm open to suggestions, and can polish up that one if needed.
Here are some pros and cons for this solution vs D28901116:
- Pro: much much smaller and simpler. Only needed to touch repo factory to create a blobstore without caching and feed that into Skiplist.
- Con: The "bypass-cache" feature of the skiplist is not baked into the type system, so it's harder to enforce. For example, this diff by itself might seem like the full solution, but if I had not done D28898839 (6f05b85534) and stack, those places would still use caching blobstores, while D28901116's solution wouldn't have that problem as it could enforce that skiplists can only be created with special blobstores that allow cache bypass on read (BlobstoreGetOps).
Reviewed By: StanislavGlebik
Differential Revision: D28903524
fbshipit-source-id: 09a09651a8b4003b03ca55e04235a8a75ec1f7bd
Summary:
Extracting `raw_blobstore` that creates a blobstore without caches, and `raw_repo_blobstore` which allows creating a repo_blobstore given a blobstore.
This diff is refactor only, the behaviour is still the same.
Reviewed By: StanislavGlebik
Differential Revision: D28903525
fbshipit-source-id: 20c7a0df6d00a26b2a8ec17a104af9cf1fa88e82
Summary:
Same as previous diff, but for a different command.
Building stuff through RepoFactory makes it easier to change how the build works *everywhere* later.
See D28877887 for goal
**This is the last diff of this type**. With this, I am able to delete `fetch_skiplist_index`, and now all skiplist building goes through factories.
Reviewed By: StanislavGlebik
Differential Revision: D28898839
fbshipit-source-id: af17d87b5890e09dbce317d52798bf78b14bc95c
Summary:
Same as previous diff, but for a different command.
Building stuff through RepoFactory makes it easier to change how the build works *everywhere* later.
See D28877887 for goal
Reviewed By: StanislavGlebik
Differential Revision: D28898653
fbshipit-source-id: 3db00cd99e50044d1acfa3d65065e5305280ec02
Summary:
Same as previous diff, but for a different command.
Building stuff through RepoFactory makes it easier to change how the build works *everywhere* later.
See D28877887 for goal
Reviewed By: StanislavGlebik
Differential Revision: D28898240
fbshipit-source-id: ede29b190c7bcf3f0a9c53c80149764a23cc4ef6
Summary:
Same as previous diff, but for a different command.
Building stuff through RepoFactory makes it easier to change how the build works *everywhere* later.
I changed from passing down a `BlobRepo` and a `LeastCommonAncestorsHint` to simply passing down an `InnerRepo`.
See D28877887 for goal
Reviewed By: StanislavGlebik
Differential Revision: D28897045
fbshipit-source-id: af546ae4702813bac2a0f938e7dec9d570c8cbf7
Summary:
Same as previous diff, but for a different command.
Building stuff through RepoFactory makes it easier to change how the build works *everywhere* later.
Had to move some code around in order to only build `BlobRepo` or `InnerRepo` once, and conditionally depending on `generate_bundles`.
See D28877887 for goal
Reviewed By: StanislavGlebik
Differential Revision: D28896888
fbshipit-source-id: 420dad831bd1759086ced72ed6a8b5726ca956d7
Summary:
Using the previous diff, this diff will now build `InnerRepo` instead of `BlobRepo`, as a way to build `Skiplist` without having to do it "manually" by calling `fetch_skiplist_index`.
See D28877887 for goal
Reviewed By: StanislavGlebik
Differential Revision: D28880352
fbshipit-source-id: 73e05864a0c0ffaba454a27ea521b4d3dc6ee78b
Summary:
BlobRepo is already easily cloneable (actually holds an arc inside).
This should also make code a little prettier, where we needed to use `(*blob_repo).clone()` instead of just `blob_repo.clone()`.
Reviewed By: StanislavGlebik
Differential Revision: D28930384
fbshipit-source-id: 59f95d10576a3f71808d0d26d36358421673351e
Summary:
The important change on this diff is in this file: `eden/mononoke/cmdlib/src/args/mod.rs`
On this diff I change that file's repo-building functions to be able to build both `BlobRepo` and `InnerRepo` (added on D28748221 (e4b6fd3751)). In fact, they are now able to build any facet container that can be built by the `RepoFactory` factory, so each binary can specify their own subset of needed "attributes" and only build those ones.
For now, they're all still using BlobRepo, this diff is only a refactor that enables easily changing the repo attributes you need.
The rest of the diff is mostly giving hints to the compiler, as in several places it couldn't infer it should use `BlobRepo` directly, so I had to add type hints.
## High level goal
This is part of the blobrepo refactoring effort.
I am also doing this in order to:
1. Make sure every place that builds `SkiplistIndex` uses `RepoFactory` for that.
2. Then add a `BlobstoreGetOps` trait for blobstores, and use the factory to feed it to skiplist index, so it can query the blobstore while skipping cache. (see [this thread](https://www.internalfb.com/diff/D28681737 (850a1a41b7)?dst_version_fbid=283910610084973&transaction_fbid=106742464866346))
Reviewed By: StanislavGlebik
Differential Revision: D28877887
fbshipit-source-id: b5e0093449aac734591a19d915b6459b1779360a
