Summary:
Move optional line handling logic into a separate function and simplify.
This diff is intended to be a pure refactoring with no observable changes in behavior. In particular, all the code dealing with the "optional" list appears to be dead code because if the line is optional, linematch will return "retry", so that branch is never reachable.
Reviewed By: DurhamG
Differential Revision: D30849757
fbshipit-source-id: 17283f9217466b3f85d913da66222b9a6779abe4
Summary:
The new debugscmstorereplay command replays scmstore fetches given an activity log created previously via the scmstore.activity log config parameter.
Replaying activity logs may help to understand or reproduce performance issues related to file fetching. Currently the replay tool ignores all complications such as concurrent fetches or variable backends.
Differential Revision: D30288701
fbshipit-source-id: c6b24acdbd37b5a51ccba3e74e8f074062e880e5
Summary:
The new scmstore.activitylog config knob optionally specifies a file for scmstore to record fetch activity. Currently it only records file fetches, but it is intended to also record tree fetches once that is fully baked.
The purpose of the log is to record file access patterns to help debug command performance. The following commit will include a tool to replay scmstore activity from the log file.
Activity is stored in the log as newline delimited JSON objects. In addition to fetched keys, we also record the start time and duration of each fetch.
Differential Revision: D30288715
fbshipit-source-id: c40177e95b1f613ebed41e50a476cbf39e6d9364
Summary:
When using hashbinary with a removed/moved file, hg throws with `TypeError: object supporting the buffer API required` this is because we are trying to `sha1(None)`.
This diff falls back to the `Binary file %s has changed` message when we have a removed file.
Reviewed By: quark-zju
Differential Revision: D30845897
fbshipit-source-id: a3d2b7d11d9c1ca3855140c9abd7550cf7076bbc
Summary:
The content store repair binding ate the entire repair message, making
it hard to debug when it wasn't working.
Reviewed By: quark-zju
Differential Revision: D30824740
fbshipit-source-id: 52dbfe79f2dd1568285cda63fb54cacf532aa4a1
Summary:
Make `verify` check the lazy changelog properties:
- Universal id <-> name mappings are known locally.
- Segments are sane (ex. high-level segments are built from low-level and there
are no cycles)
With `--dag`, also check the graph with a newly cloned remote graph.
This just calls the verification logic added in Rust `dag` crate to do the
heavy lifting.
Differential Revision: D30820773
fbshipit-source-id: 8f62f41738c3c8e3fe88442860a83fdb4944f178
Summary:
Do not assume `changelog.parents` returns 2 items.
This changes the behavior for root commits. `parents()` used to return
`[repo[nullid]]`, now it returns `[]`.
Reviewed By: andll
Differential Revision: D30784684
fbshipit-source-id: 73f58c85457391fb74b96b88dc4dcb69a25e81ac
Summary: Integrate http hash prefix lookup into the pull operation. One unfortunate change here is that if the prefix is ambiguous, we're only able to output possible full hashes as suggestions. Previously we'd also print commit log information. To retain that we'd need to add an error option to the response and have the server send back an error message with the log information or send another request to download the extra information.
Reviewed By: andll
Differential Revision: D30716050
fbshipit-source-id: 33f8bc38b0bfe7fce4ec11cd8def7feda3b3d3da
Summary:
I'm not sure what this was for, but it doesn't seem necessary, and removing it simplifies the code a lot, enabling us to make other improvements later.
This is an alternate, less ambitious version of https://www.internalfb.com/diff/D30620443.
Reviewed By: DurhamG
Differential Revision: D30674016
fbshipit-source-id: 17dee50b82c78d31e45492dc23826d8c3fe838e5
Summary:
This test relies on Mononoke, so it fails for make local build/test,
which breaks hgbuild. Let's only enable it for buck tests.
Reviewed By: quark-zju
Differential Revision: D30782799
fbshipit-source-id: 4b543beeb248715702b9072d84cdb8211dcd4a9b
Summary:
This logic scans all the ancestors of the working copy that are not
ancestor of the graft source and checks their extras. With lazy changelog this
is extremely expensive. Let's just drop this logic.
Reviewed By: quark-zju
Differential Revision: D30734017
fbshipit-source-id: ca5606cea08efe10f29847970379d6bff4eb4aee
Summary:
modernise tests (removing disabling treemanifest)
treemanifest is the default now, so it shouldn't be disabled unless absolutely necessary.
Also, if we would like to switch some of the tests to Mononoke, it shouldn't be disabled.
Only two tests left with it in commitcloud. Those two a bit harder to fix.
```
devvm1006.cln0 {emoji:1f440} ~/fbsource/fbcode/eden/scm/tests
[139] → rg 'disable treemanifest' | grep cloud
test-commitcloud-backup-bundlestore-short-hash.t: $ disable treemanifest
test-commitcloud-backup.t: $ disable treemanifest
```
Reviewed By: kulshrax
Differential Revision: D30278754
fbshipit-source-id: cf450084669c2b6b361cd34952bf986e913de1a8
Summary:
When we remove a file from a sparse profile and commit the profile, it
should delete the file on disk. There's a bug where it doesn't actually delete
the file. This fixes it by passing the correct commit parents to the refresh
function.
Reviewed By: andll
Differential Revision: D30683677
fbshipit-source-id: 7f012faa99975d8270209f2962e7f9236890daed
Summary: This breaks all use of `hg sparse`, because `.hg*` cannot be matched.
Reviewed By: mitrandir77
Differential Revision: D30666349
fbshipit-source-id: c06d1b798a57490f2e5560f178a2839ae5425146
Summary:
Sparse profiles should be roughly scoped around the files needed to
work on a certain product. If an engineer needs to work on multiple products
they should be able to enable multiple profiles.
Previously, multiple v2 profiles would be combined into an ordered list of
include/exclude rules, which meant that profiles enabled later could exclude
files included by the earlier profiles.
To fix this, let's treat each profile separately and create a matcher for each.
We then combine these into a union matcher, which means we're guaranteed to have
all the files that each profile specifies.
Differential Revision: D30586161
fbshipit-source-id: 2e04cfdba670ffce381a7c041706f315775ad7b0
Summary:
The failpoint feature supports more complex injections, such as sleep, or fail
after a few times. There is no need to keep the adhoc faultinjection feature.
Differential Revision: D30495223
fbshipit-source-id: b5613811e489a5a52e9c0dd1ebf1096c848a402b
Summary: The result is too flakey now that we've introduced threading.
Reviewed By: andll
Differential Revision: D30607733
fbshipit-source-id: f8bfa2a57d427731fb4ac3011f4364190a83b771
Summary: Apparently OSX has different ls -R output than linux.
Reviewed By: singhsrb
Differential Revision: D30566611
fbshipit-source-id: 2f232b12d1971bea18c7131c1ec82244252527c7
Summary:
The diff algorithm kept track of left vs right stores, and applied the
left vs right tree lookups to the appropriate store. In reality we only ever
have one store, so we're just wasting cycles doing two sequential lookups. Let's
just get rid of the distinction.
Reviewed By: andll
Differential Revision: D30553726
fbshipit-source-id: 72de95d298dad71dc04728d5140b4d489073a33c
Summary: Generalise the hook slightly; instead of relying on a specified file, use a revset, which opens this out to more use cases (e.g. blocking revisions in fbsource that aren't on our `master` branch)
Reviewed By: StanislavGlebik
Differential Revision: D30511155
fbshipit-source-id: 6b73c7c3e6caf2d670632110619eacb7b6216355
Summary:
Adds a helper "hg debugsparseprofilev2 $PROFILE" command that prints
out the difference in files matched by the profile using v1 and v2. This is
useful when migrating a profile to v2 and checking you haven't broken anything.
Reviewed By: kulshrax
Differential Revision: D30303868
fbshipit-source-id: 1b1d8b197470692fa13457ecb4c9344104b6d9c0
Summary:
v1 sparse profiles had excludes always take precedence over includes.
This made it hard to use and resulted in sparse profiles that were too wide.
Let's add a v2 sparse profile that respects the order of include/exclude rules.
Reviewed By: andll
Differential Revision: D30284698
fbshipit-source-id: 6765b8487695ec6cc7f8027e4da98ca65957b5d0
Summary:
In a future diff we'll want to handle ordered lists of include/exclude
rules. To do so we need to use the treematcher. Note, this means we can't use
regular expressions for sparse matchers now, but there's only one occurrence of
that in production.
Reviewed By: andll
Differential Revision: D30284700
fbshipit-source-id: c379e1feb233baee8c5ba2e1f57017b72ecaa835
Summary:
Currently excludes always take precedence over includes in sparse
configs. In a later diff we want to make this more flexible. To start with
though, let's refactor config loading to use a more "rules" oriented structure.
Excludes will still take precedence over includes, but will do so by being later
in the rule list. This will allow us to start changing rule ordering in a later
diff.
Reviewed By: andll
Differential Revision: D30284699
fbshipit-source-id: 955c016d67e22fce6b26ba7b8e1ffacc931989c8
Summary:
This allows us to get more insights on race condition (ex. pull reverts part of
what cloud sync did) issues.
Reviewed By: andll
Differential Revision: D30415135
fbshipit-source-id: c99ce77d2748e503aea523e485be5b7a57ee8b98
Summary:
Many diffs include multiple tags as prefixes, however the current implementation only greedily takes prefix as the substring up to ']'.
## Issues
Current behaviour for
- **non-tag right bracket:** `My fake diff title with [link]()` -> `My fake diff title with [link]`
- **multiple tags:** `[hg][extensions] fake diff title` -> `[hg]`
## Solution
Use regex to capture all prefix tags:
```
(?:\[.*?\])+
```
## Explanation
- Non capturing group ( `(?: ... )` )
- matching the open bracket ( `\[` )
- and any character as few times as necessary (lazy) ( `.*?` )
- and the closing bracket ( `\]` )
- matching the group as many times as needed (greedy) ( `+` )
Also note `re.match()` matches from the beginning of the string (unlike `re.search()`)
Reviewed By: ronmrdechai
Differential Revision: D30415867
fbshipit-source-id: e09d4e6d2759d0106d41d1a5d4e607ec34eef3fa
Summary: The nth ancestor revset was crashing with multiple base revisions, e.g. "(. + .^)~". Fix variable shadowing issue in revset.ancestorspec.
Differential Revision: D30375233
fbshipit-source-id: 37a78bf1000a40872600e587733a84029f68343b
Summary:
For large megarepos that are result of merging multiple smaller repos
navigating to pre-merge history is usually not what user wants. The checkout
will be slow (in non-edenfs repos), the tooling that expects some repo
structure won't work etc.
Reviewed By: StanislavGlebik
Differential Revision: D30394205
fbshipit-source-id: 23fc4fc31bf01d4cc14f6e3baa1e1165a26a1896
Summary:
During flush, there could be newly added vertexes in the master group,
overlay_map_next_id needs to be updated accordingly so lazy lookup
can still work. This is especially needed when the non-master group
needs to be re-assigned.
Reviewed By: andll
Differential Revision: D30314051
fbshipit-source-id: cce7080f62aec2617b8f3d7194864df41dfff7e0
Summary:
The test is simplified from a user report. It requires a non-master commit to
have a lazy parent in the master group and is unknown locally.
Reviewed By: andll
Differential Revision: D30314055
fbshipit-source-id: 9e2bf04728bc6c524fa3e32b8d80935d5cd7e96f
Summary: Those env vars might affect tests. Reset them in tests.
Reviewed By: andll
Differential Revision: D30314056
fbshipit-source-id: 2cedbe41a332fea582e8ad7577b9d5eaae79c015
Summary:
On Windows, it's hard to rename `.hg/store/segments/v1` away because the files
are (too easily) being currently used. Instead, let's write the new lazy
segments to `segments/v1next`, and try to rename it back later on follow-up
commands.
Reviewed By: DurhamG
Differential Revision: D30118952
fbshipit-source-id: e3edb588dccf1acb5f4ed106bbb979bcc8c6e67e
Summary:
Instrument all the fetching-related legacy API implementations (ie, not `get_logged_fetches` and `get_shared_mutable`) to track the number of calls, keys, and single-key calls.
This introduces an additional lock acquisition to each of these implementations (and it'd be awkward to merge it with the one in `fetch`), but I think that's probably fine.
For APIs which do not support a variable number of keys, I use `.call(0)` so we simply track the total number of API calls.
Reviewed By: DurhamG
Differential Revision: D30003444
fbshipit-source-id: 8756d2669ca038b3f6a08e211e44e8ccb9251312
Summary: Previously, contentstore metrics were tracked as a special case. Now that I'm tracking metrics on each backend, I've standardized the contentstore metrics to match.
Reviewed By: DurhamG
Differential Revision: D29995500
fbshipit-source-id: 3133dca5912d9cd6c26bd88d9aeb6a572f6eca40
Summary: Clean Up: Once we migrated to Mononoke we can remove the infinitepush path because it creates extra complexity in remotenames, infinitepush and commitcloud extensions
Reviewed By: yancouto
Differential Revision: D30277402
fbshipit-source-id: 1708a71eb33dc9a9b509bdc1b03bed08073b6152
Summary: This allows us to remove older files to free disk space.
Reviewed By: DurhamG
Differential Revision: D30092445
fbshipit-source-id: 20770968cfaeba4fb6f6c45b247cc1cdafb79a08
Summary: This removes some redundant lookups when using the `*_batch` APIs.
Reviewed By: andll
Differential Revision: D30094155
fbshipit-source-id: f64702a09f67b67a10a2d51545f3b6940e80022f
Summary:
Prefetch vertex on checkout (which is more expected to require network to download
files) so committing on them does not trigger remote lookup.
Reviewed By: andll
Differential Revision: D30094156
fbshipit-source-id: 917df2bbad7cee51a06f2fa38172775423893c37
Summary:
See the added comments. In some cases we can avoid remote lookups. This would
help make commit/amend more offline friendly.
Reviewed By: DurhamG
Differential Revision: D30004908
fbshipit-source-id: 94fbc7934a1eb3ae1058d8c542211a885d5ad8e6
Summary:
On Windows, EPIPE's raw os error might be 232, different from libc::EPIPE (32).
That means the Python world cannot check the errno properly to hide the EPIPE
error message. Fix it by reading io::ErrorKind first, and fallback to
raw_os_error.
Reviewed By: yancouto
Differential Revision: D30118006
fbshipit-source-id: b34a2142b24714e68d11a03a0771a5f9c037ed0f
Summary:
The "local log" algorithm in the fastlog extension is poorly implemented.
It can be much slower than the non-fastlog algorithm. This means, if the
"remote" fastlog thread does not work in production, then enabling fastlog
can actually seriously regress log performance.
The fastlog test (already) cannot meaningfully test the remote behavior. So it
is removed for now.
Differential Revision: D30145651
fbshipit-source-id: 601a38e03920fbcae86e23262949ae3e3a16b9af
Summary:
The test shows the offline commit problem that will be addressed by upcoming
changes.
Reviewed By: andll
Differential Revision: D30094154
fbshipit-source-id: c5f38ff6f0d33d241501050a1f6257633b90b31e
