Summary:
By making the EdenDispatcher less Windows dependant, we can more easily move it
into a non-Windows specific location later.
Reviewed By: chadaustin
Differential Revision: D23298028
fbshipit-source-id: 21726677808a9b8ce3d3e211dd65d9e47caad569
Summary: We need this functionality for scmquery replacement.
Reviewed By: krallin
Differential Revision: D22999141
fbshipit-source-id: e2e4177e56db85f65930b67a9e927a5c93b652df
Summary: We need this functionality for scmquery replacement.
Reviewed By: krallin
Differential Revision: D22999142
fbshipit-source-id: 04cea361ea6270626e7ff77255e3dc75875ece97
Summary:
Rust doesn't have named arguments as with positional it's hard to keep track
of all of them if there're many. I'm planning to add one more so let's switc to
struct.
Reviewed By: krallin
Differential Revision: D22999143
fbshipit-source-id: 54dade05f860b41d18bebb52317586015a893919
Summary:
If the imported commit has manifest id with all zeros (empty commit). Blobimport job can't find it in blobstore and returns error D23266254.
Add an early return when the manifest_id is NULL_HASH.
Reviewed By: StanislavGlebik
Differential Revision: D23266254
fbshipit-source-id: b8a3c47edfdfdc9d8cc8ea032fb96e27a04ef911
Summary:
Based on [user report](https://fb.workplace.com/groups/scm/permalink/3128221090560823/).
Note that slices in rust behave differently and if index exceeds slice size this will always be panic. My fix was based on assumption that behavior should be similar to python.
Reviewed By: quark-zju
Differential Revision: D23263922
fbshipit-source-id: 3d2a1a1b59f14e43b1f1a2b7102982b11637c0b4
Summary:
Having the type of data fetched can help in debugging where these fetches are
comming from. In the currently logs figuring out if a data fetch is blob or
tree requires some manual work. When looking at a big bunch of fetches this is
not super practical.
So this includes this info in our logging.
Reviewed By: chadaustin
Differential Revision: D23243444
fbshipit-source-id: 9abe5180c5d2afc0d02b27ba6a6b76401e86556e
Summary:
This could help simplify the graph a lot for repos with lots of merges. For
example, logging tags on linux.git looks like:
o fb893de3 Yesterday at 17:28 master
├─┬─┬─┬─┬─┬─┬─┬─┬─┬─╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o bcf87687 Aug 02 at 14:21 v5.8
╷ ╷ ╷ ╭─────┬─┬───┬─╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o 92ed3019 Jul 26 at 14:14 v5.8-rc7
╷ ╷ ╷ ╭─────┬─┬─┬─╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ba47d845 Jul 19 at 15:41 v5.8-rc6
╷ ╷ ╷ ╭─┬─┬─┬─┬─╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ o 11ba4688 Jul 12 at 16:34 v5.8-rc5
╷ ╷ ╷ ╭─┬─┬─┬─╯
╷ ╷ ╷ ╷ ╷ ╷ o dcb7fd82 Jul 05 at 16:20 v5.8-rc4
╷ ╷ ╷ ╭─┬─┬─┤
╷ ╷ ╷ ╷ ╷ o ╷ 9ebcfadb Jun 28 at 15:00 v5.8-rc3
╷ ╷ ╭─┬─┬─╯ ╷
╷ ╷ ╷ ╷ o ╷ 48778464 Jun 21 at 15:45 v5.8-rc2
╷ ╷ ╷ ╭─╯ ╷
╷ ╷ ╷ o ╷ b3a9e3b9 Jun 14 at 12:45 v5.8-rc1
╭─┬─┬─┼─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─╮
╷ ╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 3d77e6a8 May 31 at 16:49 v5.7
╭─┬─┴───────┬───────────┬─┬───┬─╮
╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 9cb1fd0e May 24 at 15:32 v5.7-rc7
╷ ╰─────────┬─────────────┬─┬─┬─╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o b9bbe6ed May 17 at 16:48 v5.7-rc6
╭───────────┬─────────────┬─┬─┬─╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o 2ef96a5b May 10 at 15:16 v5.7-rc5
╭───────────┬─────────────┬─┬─╯
╷ ╷ ╷ ╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 0e698dfa May 03 at 14:56 v5.7-rc4
╭───────────┴───────────┬─┬─╮
o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 6a8b55ed Apr 26 at 13:51 v5.7-rc3
╰─────────────────┬───────┬─╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ae83d0b4 Apr 19 at 14:35 v5.7-rc2
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╭─┤
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ╷ 8f3d9f35 Apr 12 at 12:35 v5.7-rc1
╭─┬─┬───────┬─────┬─┬─┬─┬─┼─╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ╷ ╷ 7111951b Mar 29 at 15:25 v5.6
╷ ╭─────────┬─────┬─┬─┬─┴───╮
╷ ╷ ╷ ╷ ╷ ╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ 16fbf79b Mar 22 at 18:31 v5.6-rc7
╷ ╷ ╭───────┴─────┬─┬─┬─────╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o fb33c651 Mar 15 at 15:01 v5.6-rc6
╷ ╭─┬─────────────┬─┬─┬─────╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ╷ 2c523b34 Mar 08 at 17:44 v5.6-rc5
╷ ╭─┬─────────────┬─┬─╯ ╷
╷ ╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 98d54f81 Mar 01 at 14:38 v5.6-rc4
╷ ╭─┴─────────────┬─╮ ╷
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ╷ f8788d86 Feb 23 at 16:17 v5.6-rc3
....
And with simplification turned on, it looks like:
o fb893de3 Yesterday at 17:28 master
├─╮
o ╷ bcf87687 Aug 02 at 14:21 v5.8
╷ ╷
o ╷ 92ed3019 Jul 26 at 14:14 v5.8-rc7
╷ ╷
o ╷ ba47d845 Jul 19 at 15:41 v5.8-rc6
╷ ╷
o ╷ 11ba4688 Jul 12 at 16:34 v5.8-rc5
╷ ╷
o ╷ dcb7fd82 Jul 05 at 16:20 v5.8-rc4
╷ ╷
o ╷ 9ebcfadb Jun 28 at 15:00 v5.8-rc3
╷ ╷
o ╷ 48778464 Jun 21 at 15:45 v5.8-rc2
├─╯
o b3a9e3b9 Jun 14 at 12:45 v5.8-rc1
╷
o 3d77e6a8 May 31 at 16:49 v5.7
╷
o 9cb1fd0e May 24 at 15:32 v5.7-rc7
╷
o b9bbe6ed May 17 at 16:48 v5.7-rc6
╷
o 2ef96a5b May 10 at 15:16 v5.7-rc5
╷
o 0e698dfa May 03 at 14:56 v5.7-rc4
╷
o 6a8b55ed Apr 26 at 13:51 v5.7-rc3
╷
o ae83d0b4 Apr 19 at 14:35 v5.7-rc2
╷
o 8f3d9f35 Apr 12 at 12:35 v5.7-rc1
╷
o 7111951b Mar 29 at 15:25 v5.6
╷
o 16fbf79b Mar 22 at 18:31 v5.6-rc7
╷
o fb33c651 Mar 15 at 15:01 v5.6-rc6
╷
o 2c523b34 Mar 08 at 17:44 v5.6-rc5
╷
o 98d54f81 Mar 01 at 14:38 v5.6-rc4
╷
o f8788d86 Feb 23 at 16:17 v5.6-rc3
....
Under the hood, the difference is how `reachableroots` gets calculated.
See also D22657197 (a5c36fd0b1) and D22368827 (da42f2c17e).
Since the old behavior almost always seems confusing to human. The new
config is turned on by default (but only takes effect if the "segments"
backend is used).
Reviewed By: sfilipco
Differential Revision: D23095468
fbshipit-source-id: f0fc631d2d9a00e3b36744e4236b43d230d10687
Summary:
Previously pieces of the command line for a process were seperated by `\0`.
This makes them a bit hard to read and also makes running queries on them
harder. Converts these `\0` back to spaces to fix this.
see https://fb.workplace.com/groups/edenfs/permalink/1446711485499079/ for
more motivation.
Reviewed By: wez
Differential Revision: D23266909
fbshipit-source-id: e4a9284e04039fcd971bed0d6e21d220e946acdb
Summary:
The files in commit cloud `References` structures are optional. Handle them
not being present.
Reviewed By: quark-zju
Differential Revision: D23266786
fbshipit-source-id: ed7128bc7e6b762d3509d77b40a00b77885191b9
Summary: This makes it a bit easier to track down perf issues printed by RUST_LOGs.
Reviewed By: sfilipco
Differential Revision: D23095463
fbshipit-source-id: 78221a1992389f512fac6e6e633be6d19123e04a
Summary:
Use `git config core.autocrlf false` to silent warnings like:
```
$ git add alpha
+ warning: LF will be replaced by CRLF in alpha.
+ The file will have its original line endings in your working directory
```
Reviewed By: sfilipco
Differential Revision: D23270146
fbshipit-source-id: af3bf241edb9f615bcc285b51cc491385f208039
Summary: The command is needed to restore a deleted workspace
Reviewed By: markbt
Differential Revision: D23250376
fbshipit-source-id: e24a7cbc0aad004291853b4c34d7474789aa9c2b
Summary:
The fuzz tests need `TestContext::id_dag()`, which was removed by D20471712 (1fb5acf242).
Restore it so fuzz tests can run. This is mainly to check the new `range`
function.
The `range` fuzz test does find an issue caused by `>` written as `>=`
relatively quickly.
Reviewed By: sfilipco
Differential Revision: D23106176
fbshipit-source-id: e9540cc932503a9d54246d24c70bac829fcb13df
Summary: Ensure that the commit text is verified, but do not verify git hashes.
Reviewed By: DurhamG
Differential Revision: D23095464
fbshipit-source-id: e62341f6c7258c6f18b7cc75088c25dfc7040ab1
Summary:
The immediate goal is to run benchmarks on a commit graph provided by a git
repo without converting a whole (large) repo from git to hg. Note git repos can
be cloned in a shallow way so it only contains the commit graph. For example:
git clone https://github.com/torvalds/linux --filter=tree:0 -n
Note: The above command writes repositoryformat=1 in `.git/config`
which is not supported by libgit2. Manually editing it to repositoryformat=0
would enable libgit2 to read it for this crate's use-case.
In the longer term we might want to extend the support so refs/trees/files can
be read/written directly via the git repo based on this work. However that's
currently beyond scope.
Reviewed By: DurhamG
Differential Revision: D23095467
fbshipit-source-id: 868beb0c7de60453b47962639863eb8f7e3f5753
Summary: Migrate to concrete types so it can be typechecked.
Reviewed By: DurhamG
Differential Revision: D23095469
fbshipit-source-id: 27c6da30ca8a1329df544cd2ded7d9734593e48a
Summary:
Read git commit graph and migrate them to `dag::Dag`.
This allows using Rust dag abstractions on the git
commit graph.
Reviewed By: DurhamG
Differential Revision: D23095471
fbshipit-source-id: 2163701350ce82ce6e97074e56ad5877f3c9c158
Summary:
Add alternative paths will be faster if changelog2 is used, since they are
backed by native paths.
Add a config option to disable the fast paths if they cause issues.
Reviewed By: DurhamG
Differential Revision: D23036074
fbshipit-source-id: 489b6eac64148867c209d595623d0b9c21ad1d5a
Summary:
OSX doesn't support touch -d. Let's just skip that part of the test on
that platform. This fixes the OSX build.
Reviewed By: singhsrb
Differential Revision: D23253475
fbshipit-source-id: 0eccb884cbdd4bf0a4068fbf943ba7dac9df4e04
Summary:
Detect the "segments" backend and calculate the revset differently.
Practically, with collapse-obsolete disabled, the time of related revset
calculation drops from 0.14s to 0.03s in my fbsource repo.
The `obsolete()` set calculation is expensive (0.4-0.6s) and a bit more
expensive with the new DAG APIs, which will be addressed in upcoming
changes. EDIT: Addressed by D23036063.
Reviewed By: DurhamG
Differential Revision: D23036055
fbshipit-source-id: 71140a88599cc68bfa90d564c786da89b3ebd38b
Summary:
The `compact` template is rarely used and is coupled with rev numbers (ex. rev
number decides what "parents" to show). Use explicit templates. This makes the
test change easier to check.
Reviewed By: DurhamG
Differential Revision: D23036076
fbshipit-source-id: f2cc0f25191711fa7d846a8ad38aee8fb9171273
Summary:
The `notbackedup()` revset is used as part of `summary` that prints information
at the end of `smartlog`. It can take hundreds of milliseconds if there are
many heads. Detect segmented changelog and use a fast path for it.
Practically this reduces `summary` from 594ms to 91ms for me:
With segmented changelog (doublewrite backend) and new code path:
91 \ summary status.py:23
2 \ currentworkspace workspace.py:121
3 | _get (2 times) workspace.py:110
3 | read (2 times) config.py:195
3 | parse (2 times) config.py:116
2 | compile (14 times) util.py:1464
3 \ __init__ syncstate.py:44
82 \ revs localrepo.py:1203
With revlog and old code path:
594 \ summary status.py:23
2 \ currentworkspace workspace.py:121
4 | _get (2 times) workspace.py:110
3 | read (2 times) config.py:195
3 | parse (2 times) config.py:116
3 | compile (14 times) util.py:1464
3 \ __init__ syncstate.py:44
46 \ revs localrepo.py:1203
539 \ _iterfilter smartset.py:647
538 | <lambda> (1565 times) commitcloud/__init__.py:371
537 | __contains__ (1565 times) smartset.py:1039
533 | _consumegen (17355 times) smartset.py:1122
Reviewed By: markbt
Differential Revision: D23036075
fbshipit-source-id: 09dcc34f34a42814c6526e558d40b4d75ba9d75f
Summary: Expose the Rust API so `getdag` can choose to skip successors or predecessors.
Reviewed By: markbt
Differential Revision: D23036056
fbshipit-source-id: 30cd437c5420d2d10176e33ef9de98814046f4ce
Summary:
The new path does not calculate the complicated `successorssets`, and is
known to make wez's repo operations significantly faster (which, I suspect is
slowed by a very long chain).
The new code is about 3x faster on my repo too:
# before
In [1]: list(repo.nodes('draft()'))
In [2]: %time len(m.mutation.obsoletenodes(repo))
CPU times: user 246 ms, sys: 42.2 ms, total: 288 ms
Wall time: 316 ms
Out[2]: 1127
# after
In [1]: list(repo.nodes('draft()'))
In [2]: %time len(m.mutation.obsoletenodes(repo))
CPU times: user 74.3 ms, sys: 7.92 ms, total: 82.3 ms
Wall time: 82.3 ms
Out[2]: 1127
Reviewed By: markbt
Differential Revision: D23036063
fbshipit-source-id: afd6ac122bb5d8d513b5cdc033e04d2c377286eb
Summary:
Optimize get_dag:
- Avoid parsing mutation entries once they are parsed, by keeping an in-memory
`parent_map`.
- Pass `heads` to `add_heads` so the segments are less fragmented, cycle break
helper is more efficient.
The `heads` optimization is effective. Practically this makes `get_dag` about 2x faster.
This has a subtle change on cycle handling - full cycle without any non-cycle heads will
be ignored. Practically cycles are rare so it might be okay.
Together with improvements on the `dag` side, `get_dag` is about 4x faster.
Reviewed By: markbt
Differential Revision: D23036062
fbshipit-source-id: 3dc407b562f7ebf2543a87c5cd651ad6a2339d67
Summary:
If there is no new master segments, it's still possible to have new non-master
segments. Fix the loop condition so we don't skip building non-master segments.
Reviewed By: sfilipco
Differential Revision: D23095465
fbshipit-source-id: 46eb9d5b5f2b04241981558646e0bc090652abce
Summary:
I noticed that high-level segments are somehow not built for non-master vertexes.
Add a test to demonstrate the issue.
Reviewed By: DurhamG, sfilipco
Differential Revision: D23095466
fbshipit-source-id: c5a6da14bdfabcf7c432f6c6dfe096c71cc10ee9
Summary: This is useful to investigate internals of dag calculations.
Reviewed By: sfilipco
Differential Revision: D23095473
fbshipit-source-id: 4750c1b4ffad32b1317051d17db9659aaaed59c4
Summary:
Follow up of the previous change by actually using the flat segments to build
segments. This significantly improved the perf. `cargo bench --bench dag_ops`
shows:
building segments (old) 774.109 ms
building segments (new) 143.879 ms
Besides, a `O(N^2)` update to `head_ids` is changed. It improves performance
when the graph has many heads (ex. the mutation graph).
Reviewed By: sfilipco
Differential Revision: D23036080
fbshipit-source-id: 033565700f253c6f20e30a00adb6b579921d6679
Summary:
While testing the `obsolete()` set, I found an in-memory segmented DAG takes
10x time to build than a HashMap DAG.
Part of the inefficiency is to use a translated "parent_func" that round-trips
through Id and Vertex, used by segment building logic. This diff makes
`IdMap::assign_head` return flat segments, so we don't need a translated
"parent_func" to build flat segments.
This diff only adds checks to make sure the parent_func (Id version) matches
the segments. The next diff switches the segment building to not use the
translated parent_func.
Reviewed By: sfilipco
Differential Revision: D23036060
fbshipit-source-id: 99137f4b5be455cdf43218ba23eb3954b6d9e05a
Summary:
This affects the `tonodes` API in the Python world. Practically this will bind
the main commit graph to sets like draft, public.
The `ToSet` requirement on `DagAlgorithm` has to be removed to avoid stack
overflow of rustc resolving constraints.
Reviewed By: sfilipco
Differential Revision: D23036077
fbshipit-source-id: 912b924e29611680ab6b2ee4dbcd7ab39824409a
Summary: This will be useful for the `obsolete()` set.
Reviewed By: sfilipco
Differential Revision: D23036072
fbshipit-source-id: 2f944ef31cf19f902622d90545fa02b7dda89221
Summary:
If two sets have different IdMap, their Ids cannot be compared directly
for correctness.
Reviewed By: sfilipco
Differential Revision: D23036068
fbshipit-source-id: e800e8273b95c1f8174236e0f30445db7fd44556
Summary: This is similar to the previous change. This allows "binding" IdMaps to sets.
Reviewed By: sfilipco
Differential Revision: D23036058
fbshipit-source-id: ec1b1ec73e949ad4865aecf17bfcc5c1ca723e0d
