Commit Graph

149 Commits

Author SHA1 Message Date
Stefan Filip
7502ce31ca dag: add in process stored IdMap constructor
Summary: The interesting observation is that InProcessStore is not public.

Reviewed By: quark-zju

Differential Revision: D20635578

fbshipit-source-id: a0149929c8059ff77f047fd385bf3b26dc738dfd
2020-03-27 13:48:51 -07:00
Stefan Filip
c400809eba dag: rename child index iteration to iter_master_flat_segments_with_parent
Summary:
`iter_segments_with_parent` has a few more conditions attached to it than the
name would imply. We are renaming it to give a better sense of its true
behavior.

Reviewed By: quark-zju

Differential Revision: D20547631

fbshipit-source-id: 406f46b9de5efc9e8e6a8c4bc22ab18fa5bc54bb
2020-03-24 13:58:07 -07:00
Stefan Filip
59ff2a8571 dag: remove_non_master implementation for
Summary: Also adding better tests for non master entries.

Reviewed By: quark-zju

Differential Revision: D20504483

fbshipit-source-id: 60d4a20aecb00f7750db2fff5d3832aac99d00e2
2020-03-24 13:58:06 -07:00
Stefan Filip
03c1e1cac5 dag: iterator implementations for InProcessStore
Summary:
The main question I had while writing the tests was whether we expect a
specific order for Segments for `iter_segments_with_parent`. `InProcessStore`
will return the segments in the order that they were inserted.

Reviewed By: quark-zju

Differential Revision: D20501401

fbshipit-source-id: 48ceb78f3191c7425c1488a3392cf3167f7e7268
2020-03-24 13:58:06 -07:00
Stefan Filip
5f4e706f81 dag: Add InProcessStore as iddagstore
Summary:
First 6 methods implemented from the IdDagStore trait for the InProcessStore.

Any suggestions welcome.

Reviewed By: quark-zju

Differential Revision: D20499228

fbshipit-source-id: cb536a3a0136077ada78934d82a25d079a5bc809
2020-03-24 13:58:06 -07:00
Stefan Filip
3dcb56535e dag: add descriptions to IdDagStore methods
Summary: Documentation.

Reviewed By: quark-zju

Differential Revision: D20499926

fbshipit-source-id: ebbb7a1249109bd56ff459a659e0c628c2974179
2020-03-24 13:58:05 -07:00
Jun Wu
8cc30ac302 dag: add Segment::new API
Summary:
Now Segment has no lifetime we can create it directly and return the ownership.

Performance of "building segments" does not seem to change:

  # before
  building segments                                 750.129 ms

  # after
  building segments                                 712.177 ms

Reviewed By: sfilipco

Differential Revision: D20505200

fbshipit-source-id: 2448814751ad1a754b90267e43262da072bf4a16
2020-03-18 15:05:58 -07:00
Jun Wu
1bd54a5971 dag: drop lifetime on Segment<'a>
Summary:
This allows structures like BTreeMap to own and store Segment.

It was not possible until D19818714, which adds minibytes::Bytes interface for
indexedlog.

In theory this hurts performance a little bit. But the perf difference does not
seem visible by `cargo bench --bench dag_ops`:

  # before
  building segments                                 714.420 ms
  ancestors                                          54.045 ms
  children                                          490.386 ms
  common_ancestors (spans)                            2.579 s
  descendants (small subset)                        406.374 ms
  gca_one (2 ids)                                   161.260 ms
  gca_one (spans)                                     2.731 s
  gca_all (2 ids)                                   287.857 ms
  gca_all (spans)                                     2.799 s
  heads                                             234.130 ms
  heads_ancestors                                    39.383 ms
  is_ancestor                                       113.847 ms
  parents                                           251.604 ms
  parent_ids                                         11.412 ms
  range (2 ids)                                     117.037 ms
  range (spans)                                     241.156 ms
  roots                                             507.328 ms

  # after
  building segments                                 750.129 ms
  ancestors                                          53.341 ms
  children                                          515.607 ms
  common_ancestors (spans)                            2.664 s
  descendants (small subset)                        411.556 ms
  gca_one (2 ids)                                   164.466 ms
  gca_one (spans)                                     2.701 s
  gca_all (2 ids)                                   290.516 ms
  gca_all (spans)                                     2.801 s
  heads                                             240.548 ms
  heads_ancestors                                    39.625 ms
  is_ancestor                                       115.735 ms
  parents                                           239.353 ms
  parent_ids                                         11.172 ms
  range (2 ids)                                     115.483 ms
  range (spans)                                     235.694 ms
  roots                                             506.861 ms

Reviewed By: sfilipco

Differential Revision: D20505201

fbshipit-source-id: c34d48f0216fc5b20a1d348a75ace89ace7c080b
2020-03-18 15:05:57 -07:00
Stefan Filip
1fb5acf242 dag: use IdDagStore in IdDag with type parameter
Summary: Make IdDag storage generic by depending on IdDagStore.

Reviewed By: quark-zju

Differential Revision: D20471712

fbshipit-source-id: 3a2668f301758a3c880db35c9f0db6887ef1dd38
2020-03-16 14:41:41 -07:00
Stefan Filip
236292c0fd dag: add the GetLock trait
Summary: Used to generalize `get_lock` functionality.

Reviewed By: quark-zju

Differential Revision: D20471710

fbshipit-source-id: e44d5b22ecacdb653170ef83914354f521f82dfc
2020-03-16 14:41:40 -07:00
Stefan Filip
66436b4a3c dag: add the IdDagStore trait
Summary: Abstract the storage functionality required by IdDag.

Reviewed By: quark-zju

Differential Revision: D20449122

fbshipit-source-id: fc3c7d7b88d74f7a93670d310be2e680f35e8ce7
2020-03-16 14:41:40 -07:00
Stefan Filip
1239628ef8 dag: move IdDag storage details to the iddagstore module
Summary:
Right now the module has one implementation IndexedLogStore. The name could
be more specific in the context of the crate.

The goal will be to add a trait for storage requirements of IdDag and
make IndexedLogStorage one implementation of that trait.

Reviewed By: quark-zju

Differential Revision: D20446042

fbshipit-source-id: 7576e1cc4ad757c1a2c00322936cc884838ff710
2020-03-16 14:41:40 -07:00
Jun Wu
1f64b4ec50 nameset: fix LazySet iteration
Summary:
The `next` method forgot to increase the iteration index, causing infinite
iteration.

Reviewed By: ikostia

Differential Revision: D20473206

fbshipit-source-id: 82a95de1b1c12ac4e9e4d328a0adba7145d7b24c
2020-03-16 13:00:35 -07:00
Jun Wu
194b38385a nameset: add a way to convert between NameSet and SpanSet
Summary:
This will be used in the Python world for legacy reasons. It shouldn't be used
in new Rust node.

To use it, the name `LegacyCodeNeedIdAccess` has to be used so we can do a code
search to find all users of it.

Reviewed By: sfilipco

Differential Revision: D20367834

fbshipit-source-id: 9b93a29f1461ce24bba6f31a2bbb1f327e216c6d
2020-03-11 20:37:30 -07:00
Jun Wu
eef56d9c5b namedag: add a sort API
Summary: This will be useful to actually sort commits.

Reviewed By: sfilipco

Differential Revision: D20367835

fbshipit-source-id: 43bc7835277af3a14ef323ce34247e0c03878dc8
2020-03-11 20:37:29 -07:00
Jun Wu
2ecc0bb757 namedag: move "all" concept to DagSet
Summary:
The old "AllSet" implementation is not very practical - it does not support
iteration. Practically, the "all()" set comes from the DAG. Change the "all"
concept to a hint similar to "is_topo_sorted", and update the fast path
(intersection) accordingly.

Reviewed By: sfilipco

Differential Revision: D20367837

fbshipit-source-id: fdbf370897c93058bfcab0571c1f6fa4b99b0f6b
2020-03-11 20:37:29 -07:00
Jun Wu
ef1696b4db namedag: rename arc_map to snapshot_map
Summary: The word "snapshot" more accurately describes its purpose.

Reviewed By: sfilipco

Differential Revision: D20367836

fbshipit-source-id: c91a0bd402fa1718b5d805beedc0e062824c53d3
2020-03-11 20:37:29 -07:00
Jun Wu
4960709aa3 dag: do not depend on types
Summary:
The dag crate is designed to work with any kind of binary commit hashes (ex. bonsai,
git or hg). The only use of `types` is to convert from binary to hex. Since dag
already has its own `to_hex` logic in `VertexName`. Let's use that instead.

Reviewed By: sfilipco

Differential Revision: D20378447

fbshipit-source-id: 00ecb551ea927fdb60dd91e5e645064f23139bcd
2020-03-11 10:49:31 -07:00
Stefan Filip
d8b4ddcecf dag: split lock file acquisition to own function
Summary:
Spliting lock file acquisition from `IdDag::prepare_filesystem_sync` to its own
function.
Useful when looking ahead to split IdDag from IndexedLog.

Reviewed By: quark-zju

Differential Revision: D20316443

fbshipit-source-id: a0fd43439730376920706bb4349ce497f6624335
2020-03-09 10:18:07 -07:00
Stefan Filip
620cdd96f2 dag: add IdDag::iter_segments_with_parent
Summary:
This removes an inline use of the indexedlog indexes.
This is going to be useful when we try to separate IndexedLog specifics from
IdDag functionality.

Reviewed By: quark-zju

Differential Revision: D20316058

fbshipit-source-id: 942a0a71660bb327376c81fd3ac435d002ecca6e
2020-03-09 10:18:07 -07:00
Jun Wu
64ba669a51 nameset: add some tests for DagSet
Summary:
With the new crate-public interfaces and Debug implementations it's possible to
write tests for DagSet. So let's do it.

Reviewed By: sfilipco

Differential Revision: D20242561

fbshipit-source-id: 180e04d9535f79471c79c4307f6ab6e8e8815067
2020-03-05 11:46:18 -08:00
Jun Wu
bb1562604a dag: make some test APIs public in crate
Summary: Those will be reused by nameset::DagSet.

Reviewed By: sfilipco

Differential Revision: D20242563

fbshipit-source-id: 944e9a04aeb15439256ecea64355b67e326e5c89
2020-03-04 17:33:25 -08:00
Jun Wu
b8e1477401 nameset: impl Debug for other sets
Summary:
This is useful for `assert_eq!(format!("{:?}", set), "...")` tests.

It will be eventually exposed to Python as `__repr__`, similar to Python's
smartsets.

Reviewed By: sfilipco

Differential Revision: D20242562

fbshipit-source-id: 5373bb180db7cafebf273ace7cf2cb80fbfb8038
2020-03-04 17:33:25 -08:00
Jun Wu
fa069204e3 nameset: impl Debug for StaticSet
Summary:
In the Python world all smartsets have some kind of "debug" information. Let's
do something similar in Rust.

Related code is updated so the test is more readable.

Reviewed By: sfilipco

Differential Revision: D20242564

fbshipit-source-id: 7439c93d82d5d037c7167818f4e1125c5a1e513e
2020-03-04 17:33:24 -08:00
Jun Wu
10bb5a144e revset: replace some repo.revs with repo.nodes
Summary:
Migrate away from some uses of revision numbers.
Some dead code in discovery.py is removed.

I also fixed some test issues when I run tests locally.

Reviewed By: sfilipco

Differential Revision: D20155399

fbshipit-source-id: bfdcb57f06374f9f27be51b0980652ef50a2c8e0
2020-02-28 17:45:26 -08:00
Jun Wu
0220b4a0c3 nameset: make NameIter Send
Summary: This makes it possible to use NameIter in py_class.

Reviewed By: sfilipco

Differential Revision: D20020529

fbshipit-source-id: b9147b7dccb38d18d8361b420507fcbe97e01351
2020-02-28 16:35:25 -08:00
Jun Wu
782f2017aa dag: add hex prefix lookup
Summary: This will be used by commit hash prefix lookup.

Reviewed By: sfilipco

Differential Revision: D20020523

fbshipit-source-id: f2905ddf63098704b08dad8eb48272c3ffba7e25
2020-02-28 16:35:24 -08:00
Jun Wu
12441f48bf dag: re-export common types at top-level
Summary: Export common types at the top-level of the crate so it's easier to use.

Reviewed By: sfilipco

Differential Revision: D20020526

fbshipit-source-id: e9a0a8bc3cc91f81d0bc74e7530cd4613fc1dd61
2020-02-28 16:35:23 -08:00
Jun Wu
bc9f72ccf3 dag: implement DAG algorithms on NameDag
Summary: Those just delegate to IdDag for the actual calculation.

Reviewed By: sfilipco

Differential Revision: D20020522

fbshipit-source-id: 272828c520097c993ab50dac6ecc94dc370c8e8b
2020-02-28 16:35:23 -08:00
Jun Wu
b88da34fb0 dag: expose NameDag in tests
Summary: This allows tests to check NameDag APIs.

Reviewed By: sfilipco

Differential Revision: D20020525

fbshipit-source-id: 4ee8e4bcbd0731512ba17068e827b8045fc5d522
2020-02-28 16:35:23 -08:00
Jun Wu
194cd25f4f dag: add Arc<IdMap> to NameDag
Summary: This will be used to produce NameSet.

Reviewed By: sfilipco

Differential Revision: D20020519

fbshipit-source-id: abf6d73f2b985b74560d6b5db2800ff25450f02e
2020-02-28 16:35:22 -08:00
Jun Wu
7a343271b9 dag: rename NameDag::parents to NameDag::parent_names
Summary: This matches IdDag::parents (taking a set) and IdDag::parent_ids.

Reviewed By: sfilipco

Differential Revision: D20020524

fbshipit-source-id: 6e90727c355a7400f9a23e0b25e3392bdc032f49
2020-02-28 16:35:22 -08:00
Jun Wu
e3b28a683c nameset: add fast paths for DagSet
Summary: DagSet's SpanSet has fast paths for set operations. Use them.

Reviewed By: sfilipco

Differential Revision: D19912104

fbshipit-source-id: 24b55aa14d03be2f1be59c923e0b8e79d6bcbe6d
2020-02-28 16:35:22 -08:00
Jun Wu
587b06efee nameset: AllSet
Summary: This is similar to hg's fullreposet. It'll be useful as a dummy "subset".

Reviewed By: sfilipco

Differential Revision: D19912108

fbshipit-source-id: 33a95bcb3cf5931a431a1201d1a1f3c627cec7a1
2020-02-28 16:35:21 -08:00
Jun Wu
d41c55a13b nameset: SortedSet
Summary: SortedSet is a wrapper to other sets that marks it as topologically sorted.

Reviewed By: sfilipco

Differential Revision: D19912111

fbshipit-source-id: 2637e8fd29b97f6db0c5bae3f0decd7ac382eeb1
2020-02-28 16:35:21 -08:00
Jun Wu
51bea7aff7 nameset: LazySet
Summary: Similar to Mercurial's smartset.generatorset.

Reviewed By: sfilipco

Differential Revision: D19912110

fbshipit-source-id: 7d940b8578ec7090282e2addb1fde871cddb2b25
2020-02-28 16:35:20 -08:00
Jun Wu
5e451d07b1 nameset: DagSet
Summary:
Wraps SpanSet + IdMap so it only exposes commit names without ids.
There is no equivalent smartset in Mercurial.

Reviewed By: sfilipco

Differential Revision: D19912112

fbshipit-source-id: 0d257de11527dfa8836065ac94f652730a97a468
2020-02-28 16:35:20 -08:00
Jun Wu
e7e7a5b356 nameset: StaticSet
Summary: Similar to Mercurial's smartset.baseset. All names are statically known.

Reviewed By: sfilipco

Differential Revision: D19912105

fbshipit-source-id: e4fcf2d59291adb3ca01b3b90f1ac32c65ad7eaa
2020-02-28 16:35:20 -08:00
Jun Wu
349d1bc33e nameset: IntersectionSet
Summary: Similar to Mercurial's smartset.filterset.

Reviewed By: sfilipco

Differential Revision: D19912113

fbshipit-source-id: 7cf2101b2eb7ba34b542199293cdbfd3973ef72f
2020-02-28 16:35:19 -08:00
Jun Wu
c0a1a3ab22 nameset: DifferenceSet
Summary: Similar to Mercurial's smartset.filterset.

Reviewed By: sfilipco

Differential Revision: D19912107

fbshipit-source-id: a3187c94f8e0c64f6d92e924ba46e83ce74c3e19
2020-02-28 16:35:19 -08:00
Jun Wu
b6cea95ea5 dag: use Bytes to avoid some VertexName copies
Summary:
This is an example about how to use the new Bytes type. The performance change
is not obviously visible in benchmarks since the bottleneck is not at the bytes
copying.

Reviewed By: DurhamG

Differential Revision: D19818720

fbshipit-source-id: a431ae206cfa4fa08b2e162a48b3d7cbcd900f7f
2020-02-28 09:23:59 -08:00
Jun Wu
76ab726056 dag: switch from bytes to minibytes
Summary: The APIs are compatible so the switch is straightforward.

Reviewed By: DurhamG

Differential Revision: D19818713

fbshipit-source-id: 504e9149567c90eb661804e0dad20580a401aa76
2020-02-28 09:23:59 -08:00
Jun Wu
9e3920ca1c dag: fix benchmarks
Summary: D19559127 forgot those files.

Reviewed By: DurhamG

Differential Revision: D19818715

fbshipit-source-id: 92321492eae89ed9f748800b3bfcc306a54aab20
2020-02-28 09:23:59 -08:00
Jun Wu
bce29c9562 nameset: UnionSet
Summary: Similar to Mercurial's smartset.addset.

Reviewed By: sfilipco

Differential Revision: D19912106

fbshipit-source-id: 0d0c8d0b71d2757259d26295eb4a564fea807dea
2020-02-27 07:34:57 -08:00
Jun Wu
c906a21ce1 nameset: initial NameSet abstraction
Summary:
The NameSet is something similar to SpanSet and Mercurial's smartset but speaks
VertexNames instead of Ids. The idea is, NameSet will be part of NameDag APIs,
and potentially replace Mercurial's smartset layer (just smartset the container
types, not the revset language), in a way that revision numbers are completely
hidden behind the scenes.

This diff adds some basic abstraction around iteration-related operations.
Other operations will be added later.

Reviewed By: sfilipco

Differential Revision: D19912109

fbshipit-source-id: 504a26c074282ec51f260535ca63e943124f688e
2020-02-27 07:34:57 -08:00
Stefan Filip
d78982a6e8 dag: move iddag to own file
Summary:
Separate Segment and IdDag in two individual files. This is preparation for
refactoring IdDag to be more flexible in terms of storage. That will probably
involve moving stuff out of IdDag into a new file that deals with the storage
abstractions.

Reviewed By: quark-zju

Differential Revision: D19559127

fbshipit-source-id: b3b9b18e2653157e69148b1f29292a57b30016ec
2020-01-24 15:49:54 -08:00
Jun Wu
29c749ef7d dag: add fuzz tests on the octopus DAG
Summary: This gives us some confidence about octopus merge handling.

Reviewed By: DurhamG

Differential Revision: D19540726

fbshipit-source-id: e84de74aecae54429483edd185d39fd1bd858f87
2020-01-23 17:58:51 -08:00
Jun Wu
8ac97da54e bindag: make TestContext more flexible
Summary:
TestContext uses ParentRevs. That limits parents to at most 2.

Use a type parameter so we can opt-in Vec<usize> for octopus merge support,
at the cost of worse cache efficiency.

Reviewed By: DurhamG

Differential Revision: D19540727

fbshipit-source-id: f9e8de151b7b296fd6f0fd89be9de2b8996634c7
2020-01-23 17:58:51 -08:00
Jun Wu
df23791d08 bindag: add some octopus examples
Summary:
Our new algorithms support octopus merges. However there were no tests using
octopus merges. This diff adds a simple one.

Reviewed By: DurhamG

Differential Revision: D19540728

fbshipit-source-id: 8411024f0b7e27c2ebfabbe1935496124c25df7b
2020-01-23 17:58:51 -08:00
Jun Wu
494bdae7cc dag: add a fuzz test about range algorithm
Summary:
The test runs the old and new algorithm and compares their result.  This is more
interesting than using random numbers, since the fuzzing framework will try to
explore new code paths.

Reviewed By: sfilipco

Differential Revision: D19511576

fbshipit-source-id: e9a2066769b54a60bb92643e5715f91a6fccbcb5
2020-01-23 17:58:50 -08:00
Jun Wu
78ea96cb9d bindag: port range algorithm from hg
Summary:
The ported algorithm will work as a comparison to verify dag's range
implementation.

Reviewed By: sfilipco

Differential Revision: D19511574

fbshipit-source-id: 589353d6e6c91b8d6707c977eeb8558ac733b525
2020-01-23 17:58:50 -08:00
Jun Wu
fff2cb833f dag: add a fuzz test about gca algorithm
Summary:
The test runs the old and new algorithm and compares their result.  This is more
interesting than using random numbers, since the fuzzing framework will try to
explore new code paths.

This cannot run on stable Rust yet. I added a README for how to run it.

Reviewed By: sfilipco

Differential Revision: D19504096

fbshipit-source-id: 621da02c50a771dee9932f9d7a407cb1f412a543
2020-01-22 19:30:50 -08:00
Jun Wu
af85f4ff3b bindag: add a way to get a subdag of a parsed bindag
Summary: Sometimes the graph is too large. Provide a way to slice it.

Reviewed By: sfilipco

Differential Revision: D19511575

fbshipit-source-id: 504317d6894764043b23ea49dcf09c8cdea96961
2020-01-22 19:30:49 -08:00
Jun Wu
b5482f8976 bindag: add utilities for easier testing
Summary:
As we plan to test the dag crate with some other DAG implementation,
add a convenient structure that setups both DAG implementations.

Reviewed By: sfilipco

Differential Revision: D19503371

fbshipit-source-id: 3e9933ad37301bfac36eb1af6d82b4298af778b6
2020-01-22 19:30:49 -08:00
Jun Wu
7d11508dfa bindag: port GCA algorithm from hg
Summary:
The ported algorithm will work as a comparison to verify dag's gca
implementation.

Reviewed By: sfilipco

Differential Revision: D19503373

fbshipit-source-id: f5253db89fbcdc2fd02f3fdaa0796e24338b1fba
2020-01-22 19:30:49 -08:00
Jun Wu
a98d288938 bindag: apply smallvec optimization
Summary:
This is similar to D17581248. It will make the old linear-scan algorithm (which
will be added later) about 5x faster.

Reviewed By: sfilipco

Differential Revision: D19503372

fbshipit-source-id: c65d7217e7b144603dadd57f54a5e70f513c8e51
2020-01-22 19:30:48 -08:00
Jun Wu
64271f24ba dag: move bindag from benches to a separate crate
Summary: This allows bindag to be used outside benches.

Reviewed By: sfilipco

Differential Revision: D19503374

fbshipit-source-id: 131061f7d1d28125875a86afc330dbb9634249cf
2020-01-22 19:30:48 -08:00
Jun Wu
124e275377 dag: make NameDag use MultiLog for data consistency
Summary: This ensures IdMap and IdDag are guaranteed consistent in the storage layer.

Reviewed By: DurhamG

Differential Revision: D19432658

fbshipit-source-id: 00f1a9b4c747baa1f14d78c31d925682317463b4
2020-01-17 21:49:57 -08:00
Jun Wu
94c383327a dag: make NameDag support buffered changes
Summary:
Previously, NameDag only supports writing directly to disk. That is not easy
for hg to use, since hg knows commits before bookmark names during pull, while
the API requires both commits and bookmarks (to decide which commits belong to
the "master" group) at the same time.

The old API also does not match other storage layers like indexedlog and
metalog, where writes are buffered until explicitly flushed.

This diff adds the "write-in-memory" (named "add_heads"), and "flush" APIs to
make NameDag easier to use. Under the hood, it just copies the "added" portion
of the DAG, and re-use the old API ("build", renamed to "add_heads_and_flush")
to do the job.

Note: The buffered changes pattern cannot be used in IdDag, since Ids might
have to be re-assigned on flush. But NameDag does not expose the raw Ids in its
"normal" interface, so it's fine to reassign Ids on flush.

Reviewed By: markbt

Differential Revision: D19405474

fbshipit-source-id: 75e5e5815c78c3577a0138f48185f6c4b5a80891
2020-01-15 14:02:07 -08:00
Jun Wu
3a74b82a39 dag: rename NamedDag to NameDag
Summary:
This is more consistent with the name "IdDag", because both "Name" and "Id" are
noun.

Reviewed By: singhsrb

Differential Revision: D19405473

fbshipit-source-id: a3b7546dd0ddcae5d9ed562838bd6be10a47063c
2020-01-15 14:02:06 -08:00
Jun Wu
3c5766d59e dag: rename Dag to IdDag
Summary: This is more consistent with the name "NamedDag".

Reviewed By: singhsrb

Differential Revision: D19405472

fbshipit-source-id: f7023307acaf96bf77c9fa9704dcaf6fc59b56f2
2020-01-15 14:02:06 -08:00
Jun Wu
85a89c6f96 dag: add git commit graph as test fixture
Summary:
The git repo has a lot of merges. So it's interesting as a testing graph.
The file was generated by:

    git clone https://github.com/git/git --bare
    hg --config extensions.hggit= clone git.git git-hg
    hg --cwd git-hg debugbindag -r 'all()' -o git.bindag

Reviewed By: DurhamG

Differential Revision: D19411825

fbshipit-source-id: 0fec8a16786dc8e6790986bca5c62a76276aa942
2020-01-15 11:03:52 -08:00
Jun Wu
064068169b dag: rename slice to VertexName
Summary:
Per discussion, we decided to use "VertexName" as the struct name for things
like commit hashes, or the string names in tests (or the Mozilla DAG in tests).
Therefore, introduce a dedicated VertexName type and repalce all callsites to
use it.

`bytes::Bytes` is used so copying the `VertexName` is somewhat considered cheap.

This adds some overhead copying slices (and `Bytes` has some overhead). It
regresses the "building segments" benchmark from 673ms to 773ms, which seems
okay given the cleaner interface.

Reviewed By: markbt

Differential Revision: D19154905

fbshipit-source-id: 4c6d4eca67c11c10ed5f21999ccdc3f1b01695e8
2020-01-08 21:35:28 -08:00
Jun Wu
232d5047cc dag: fix incorrect gca calculation
Summary: `gca(X) = gca(roots(X))`, not `gca(heads(X))`.

Reviewed By: sfilipco

Differential Revision: D19154907

fbshipit-source-id: 8d54992fbf5b03ce7ab2fbcc689c79351bf75947
2020-01-08 18:46:08 -08:00
Jun Wu
c6c4baa07e dag: redefine Id::{MIN, MAX} using Group
Summary:
Redefine Id::{MIN, MAX} using Group so Id::MAX.group() is a valid group.
Previously Id::MAX has an invalid group.

Reviewed By: sfilipco

Differential Revision: D19154908

fbshipit-source-id: 6108f02da786eb228a7d1b772ca5134c1ebb6f6b
2020-01-08 18:46:07 -08:00
Jun Wu
c4cd7df3b2 dag: extract logic calculating universal commits to a method
Summary:
Address review feedback on D18670160.

This also fixes the definition of "universal" a bit - the new code no longer
assumes there are only one head in the master group.

Reviewed By: sfilipco

Differential Revision: D19154906

fbshipit-source-id: 4fbf080354318300ce09dc7fba2dcc5f942b6308
2020-01-08 18:46:07 -08:00
Jun Wu
bcaff9062c dag: rebuild non-master id and segments if needed
Summary:
Add utilities to rebuild non-master ids and segments if necessary.

The `NamedDag` structure ensures indexes have 1:1 mapping.

Reviewed By: sfilipco

Differential Revision: D18838995

fbshipit-source-id: 4a48b183c182bd5e6336a2ca4973c36091fbbfd8
2020-01-08 18:46:07 -08:00
Jun Wu
28380e0272 dag: fix logic about avoiding unnecessary high level segments
Summary:
The check of "is this high level segment necessary or not" should be done
before dropping the last segment. This fixes an issue discovered in D18838992.

Reviewed By: sfilipco

Differential Revision: D19154904

fbshipit-source-id: fb4c83c39d66215bae168ad98e5cf78de91cc5a3
2020-01-08 18:46:06 -08:00
Jun Wu
d4eddd6b11 dag: use NamedDag in tests
Summary:
NamedDag is the high-level interface that is more interesting to test.

High-level segments are changed subtly because NamedDag syncs them to disk
first.

Reviewed By: sfilipco

Differential Revision: D18838992

fbshipit-source-id: c6a557e0a44a1d24320ea4a9e4283262f6f30f67
2020-01-08 18:46:06 -08:00
Jun Wu
5c7acfc1ca dag: add a NamedDag that keeps Dag and IdMap consistent
Summary:
It turns out the interaction between dag and idmap can be complex, especially
when a non-master id gets re-assigned a master id.

A high-level structure is helpful here to handle the corner cases correctly.

This diff adds basic definitions, and one of the most complex function - build.
The build function is ported from pydag.

Reviewed By: sfilipco

Differential Revision: D18838994

fbshipit-source-id: a2f6db3b188d5fd04d8e1d6df40ba985e33281f6
2019-12-22 11:27:32 -08:00
Jun Wu
61e0f06417 dag: add ways to remove non-master ids in Dag and IdMap
Summary: First step to implement segment and idmap rewrites for non-master ids.

Reviewed By: sfilipco

Differential Revision: D18838996

fbshipit-source-id: 0380c0193d9078a2a2d7fde9c5f565e8dbc8e713
2019-12-22 11:27:32 -08:00
Jun Wu
05ba5a56d4 dag: rewrite Debug for Dag
Summary:
The current implementation iterates through all entries, which will include
deleted entries. Rewrite it to use index lookups so it does not print
deleted entries.

Reviewed By: sfilipco

Differential Revision: D18838993

fbshipit-source-id: 58d3f2da27cd3e91bc10cbb04bcdca3b7ff07dbb
2019-12-20 16:14:21 -08:00
Jun Wu
859b79de3b dag: replace Group::MAX with Group::COUNT
Summary:
Address review comment in D18640899. I went a bit further that makes COUNT,
ALL, and BITS consistent.

Reviewed By: sfilipco

Differential Revision: D18820721

fbshipit-source-id: e354a3aecf0d8cbe66c43d56198933cdc6420241
2019-12-20 16:14:21 -08:00
Jun Wu
710e2c9ae8 dag: replace Id(0) and Id::max_value() with Id::{MIN,MAX}
Summary:
Address review comment in D18640898.

Id::min_value and Id::max_value become unused and are removed.

Reviewed By: sfilipco

Differential Revision: D18820722

fbshipit-source-id: d9682bf5918307166571fa35aa15cc9eee1a7140
2019-12-20 16:14:21 -08:00
Jun Wu
a1d710fa58 dag: rename GroupId to Group
Summary:
Address review comment in D18640899. This makes the word `id` universally
refer to a same concept within the crate.

Reviewed By: sfilipco

Differential Revision: D18820723

fbshipit-source-id: 6803192db7e1304a72100568f8f29b90f25c7779
2019-12-20 16:14:21 -08:00
Jun Wu
d3543665eb dag: de-dup IdMap at insertion time
Summary: There is no need to insert duplicated entries to IdMap.

Reviewed By: sfilipco

Differential Revision: D18670158

fbshipit-source-id: 812ea421c9f36cd12b4aa2be5b2560abec627a52
2019-12-20 16:14:20 -08:00
Jun Wu
d60355d1ba dag: add a way to generate sparse IdMap
Summary: This emulates what IdMap the server will send to the client at initial clone.

Reviewed By: sfilipco

Differential Revision: D18670160

fbshipit-source-id: 496142c1e149df9accf816e3d0a1b8316ff64c67
2019-12-20 16:14:20 -08:00
Jun Wu
3125f6bbd5 dag: implement Debug for IdMap
Summary: This makes it easier to check what's stored in IdMap.

Reviewed By: sfilipco

Differential Revision: D18670159

fbshipit-source-id: 91387d01da7d52a0d349e87ffa0aa8983bbe2213
2019-12-20 16:14:20 -08:00
Jun Wu
0d8ae15299 dag: implement more protocols
Summary:
This diff implements the following part in a local-only environment:

  Id -> Slice: Id -> Request --> Response -> Slice
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Slice -> Id: Slice -> Request --> Response -> Id
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^

Reviewed By: sfilipco

Differential Revision: D18664210

fbshipit-source-id: 740b9cdfe533e7bcc9bbbf2a53c8bcc5bc3648bf
2019-12-20 16:14:20 -08:00
Jun Wu
7a35f20e72 dag: implement some protocols
Summary:
This diff implements the following part in a local-only environment:

  Id -> Slice: Id -> Request --> Response -> Slice
               ^^^^^^^^^^^^^
  Slice -> Id: Slice -> Request --> Response -> Id
               ^^^^^^^^^^^^^^^^

Reviewed By: sfilipco

Differential Revision: D18664205

fbshipit-source-id: 47a35c6f95a58ab29c014ac3c6799f16e6114da3
2019-12-20 16:14:20 -08:00
Jun Wu
2467448518 dag: abstractions around remote protocols
Summary:
In the future the client will have a sparse IdMap, and needs the server (who
has a complete IdMap)'s help to convert between Id and Slice (commit hash)
for commits in the MASTER group.

This diff defines basic abstractions for communications between a sparse IdMap
and a complete IdMap. The basic flows are:

  To convert Id to Slice: Id -> Request --> Response -> Slice
  To convert Slice to Id: Slice -> Request --> Response -> Id

(`->`: client-side; `-->`: server-side)

Reviewed By: sfilipco

Differential Revision: D18664206

fbshipit-source-id: 93a6b671be30ab1247c71d814817a2ea06b6bc66
2019-12-20 16:14:20 -08:00
Jun Wu
3d65779318 dag: add IdMapLike for minimal IdMap interface
Summary:
In the future we might have revlog or more complicated storage for maintaining
the IdMap. Define a trait for it.

Reviewed By: sfilipco

Differential Revision: D18664208

fbshipit-source-id: 242611d78852e8d82a52ab52192db87c13b848e7
2019-12-05 17:01:13 -08:00
Jun Wu
dbfc96ca15 dag: add to_first_ancestor_nth
Summary: This is the reverse of first_ancestor_nth. It will be used later.

Reviewed By: sfilipco

Differential Revision: D18664209

fbshipit-source-id: 8d3706cdc080086ec60da60870fde76c8933f6a8
2019-12-05 17:01:13 -08:00
Jun Wu
74e624297c dag: add first_ancestor_nth
Summary: This will be used in an upcoming change.

Reviewed By: sfilipco

Differential Revision: D18664207

fbshipit-source-id: 3030bb02e172b95c6dbabda74c318e7f16212f59
2019-12-05 17:01:12 -08:00
Jun Wu
f028832a62 dag: add a fast path to ancestors calculation
Summary:
With the ONLY_HEAD flag, ancestors can be sped up sometimes.  Since ancestors
is used widely in other DAG calculations, perf improvements are visible across
the benchmark. The trade-off is to spend a bit more time on building segments
to figure out ONLY_HEAD.

After this diff:

  building segments                                 576.625 ms
  ancestors                                          47.119 ms
  children                                          436.967 ms
  common_ancestors (spans)                          257.096 ms
  descendants (small subset)                        367.316 ms
  gca_one (2 ids)                                   146.315 ms
  gca_one (spans)                                   253.059 ms
  gca_all (2 ids)                                   236.075 ms
  gca_all (spans)                                   266.695 ms
  heads                                             220.766 ms
  heads_ancestors                                    36.258 ms
  is_ancestor                                        98.967 ms
  parents                                           219.173 ms
  parent_ids                                          9.282 ms
  range (2 ids)                                      99.859 ms
  range (spans)                                     205.154 ms
  roots                                             429.047 ms

Before this and previous diff:

  building segments                                 542.564 ms
  ancestors                                         316.075 ms
  children                                          428.612 ms
  common_ancestors (spans)                          452.850 ms
  descendants (small subset)                        537.483 ms
  gca_one (2 ids)                                   526.112 ms
  gca_one (spans)                                   446.726 ms
  gca_all (2 ids)                                   780.590 ms
  gca_all (spans)                                   490.906 ms
  heads                                             214.005 ms
  heads_ancestors                                   217.558 ms
  is_ancestor                                       256.769 ms
  parents                                           213.170 ms
  parent_ids                                          9.313 ms
  range (2 ids)                                     259.435 ms
  range (spans)                                     358.934 ms
  roots                                             423.005 ms

Reviewed By: sfilipco

Differential Revision: D18642797

fbshipit-source-id: 27f1ee5797a92142b2c19195eb26baef58769553
2019-12-05 17:01:12 -08:00
Jun Wu
94a09a890d dag: add a flag to indicate a segment is the only head so far
Summary: The pre-calculated flag can be used for some fast paths.

Reviewed By: sfilipco

Differential Revision: D18642796

fbshipit-source-id: bc16b3517cc29e23478f2027e83e2a2282be0d0a
2019-12-05 17:01:12 -08:00
Jun Wu
3e205e29b3 pydag: revise bindings
Summary:
The bindings APIs are out of date.

The latest idea is to write non-master commits to disk, in a different group.
There are no "segments in memory". Therefore:
- Rename `builddisk` to `build`.
- Make sure `build` updates internal states by calling `reload`.
- Drop `buildmem`.
- In case `builddisk` gets called very frequently, reduce its overhead by
  doing nothing if all commits are built already.

The constructor is revised to match the latest interface (no
"max segment level" and "set_new_segment_size" needs to be called).

Reviewed By: sfilipco

Differential Revision: D18640897

fbshipit-source-id: 76a5d0db363713dc5b763503c238c1bfa61ea02e
2019-12-05 17:01:11 -08:00
Jun Wu
744f17df8b dag: add group support for segments
Summary: Make it possible to build non-master segments.

Reviewed By: sfilipco

Differential Revision: D18641507

fbshipit-source-id: 4d0c46cdea240849af37d28219b6023578514323
2019-12-05 17:01:11 -08:00
Jun Wu
cdcbf836eb dag: fix a correctness issue that flat segment building might be incomplete
Summary:
`build_segments_volatile(id)` should build flat segments so it includes `id`.

Right now it can fail that in a corner case. Fix it.

Reviewed By: sfilipco

Differential Revision: D18641509

fbshipit-source-id: ad1de694e4d31c51f29e9bc5800291066bf16dbb
2019-12-05 17:01:11 -08:00
Jun Wu
5bac1b4598 dag: re-assign non-master ids to master ids
Summary:
This is needed to keep Ids topologically sorted.

This is reflected in the test added by D18641507, when assigning
master ids to `I`:

```
0---1---2---3---4---5---6--------H---I
     \               \          /
      N0-N1-N2-N3     N4-N5-N6-o
                \            \
                 -------------N7--N8
Lv0: R0-1[] 2-3[1] 4-6[3] N0-N1[1] N2-N3[N1] N4-N6[5] N7-N8[N3, N6]
Lv1: R0-3[] 4-6[3] N0-N1[1] N2-N3[N1] N4-N8[5, N3]
Lv2: R0-6[] N0-N3[1] N4-N8[5, N3]
```

Note: 0, 1, 2, ... are ids in the master group. N0, N1, ... are ids in the non-master group.

```
0---1---2---3---4---5---6--------11--12
     \               \          /
      N0-N1-N2-N3     7--8--9--10
                \            \
                 -------------N7--N8
Lv0: R0-1[] 2-3[1] 4-6[3] 7-10[5] 11-12[6, 10] N0-N1[1] N2-N3[N1] N4-N6[5] N7-N8[N3, N6]
Lv1: R0-3[] 4-6[3] 7-12[5, 6] N0-N1[1] N2-N3[N1] N4-N8[5, N3]
Lv2: R0-6[] 7-12[5, 6] N0-N3[1] N4-N8[5, N3]
Lv3: R0-12[] N0-N8[1, 5]
```

Notice that 7, 8, 9 replace the old non-master ids N4, N5, N6. Without this change,
N4, N5, N6 will be kept and break the topo-sorted property.

An `ensure!` was added to make sure Ids are topo-sorted.

Reviewed By: sfilipco

Differential Revision: D18641510

fbshipit-source-id: bbe642df3615c65db7f2e7c889624e00e53ef7ba
2019-12-05 17:01:10 -08:00
Jun Wu
fb718fe106 dag: allow re-assigning a non-master Id to a master Id
Summary:
The IdMap has assertions about 1:1 mapping between Id and Slice.

Practically, a non-master commit can be merged into a master commit.
That means a commit hash might map to 2 Ids: [master-id, non-master-id].
Allow that to happen (temporarily).

Note: The non 1:1 mapping causes more headache than benefit and
D18838995 restores 1:1 mapping. But it still requires temporary
non-1:1 mapping to work.

Reviewed By: sfilipco

Differential Revision: D18641508

fbshipit-source-id: bc4bc831d5ab5ebfc2588bafd48114ae49d31e0f
2019-12-05 17:01:10 -08:00
Jun Wu
81b7593249 dag: avoid panic on drop
Summary: panic on drop is a terrible idea. It can cause SIGILL kill without meaningful messages or meaningful backtrace. Remove it.

Reviewed By: xavierd

Differential Revision: D18640894

fbshipit-source-id: 3ab52ded9875e17e891acd8f9b45d82964615f01
2019-12-04 22:43:24 -08:00
Jun Wu
de9a7fecb1 dag: replace some panic! with bail!
Summary: This makes the code harder to crash in abnormal cases.

Reviewed By: sfilipco

Differential Revision: D18640895

fbshipit-source-id: ee81dd7739739d3d15052bdb7bb2ba80ed48cd0a
2019-12-04 22:43:24 -08:00
Jun Wu
75aff36c7f dag: add group support for IdMap
Summary:
This allows us to allocate Ids for non-master commits in a separate range
so they wouldn't cause fragmentation for commits in the master group.

Reviewed By: sfilipco

Differential Revision: D18640896

fbshipit-source-id: c19c53342583ded8795849584cf5f31bc86b2818
2019-12-04 22:43:23 -08:00
Jun Wu
05157a174a dag: replace u64 with the new Id type
Summary: Migrate the codebase to use the new Id type instead of an alias of u64.

Reviewed By: sfilipco

Differential Revision: D18640898

fbshipit-source-id: 415db9afeda1f8960e18c69e70f53a9eeb63cbbe
2019-12-04 22:43:23 -08:00
Jun Wu
c016535cb6 dag: add a dedicated Id type
Summary:
Background:
- Tests and Nuclide haven't removed usage of revision numbers.
- If we build segments for non-master commits on demand, there are performance
  concerns about very long release branches.
- It's very desirable to keep `ancestors(master)` continuous. In other words,
  `ancestors(master)` is a single integer span. This affects performance for
  DAG algorithms a lot.

To solve the problems, I'm going to use separate ranges for master and
non-master commit Ids (so assigning Ids to non-master commits wouldn't affect
what numbers can be used for the master group). To make the code easier to
read, I'm changing `type Id = u64` to `struct Id` where the struct is
group-aware.

This diff introduces related types and basic operations.

Reviewed By: sfilipco

Differential Revision: D18640899

fbshipit-source-id: 8f7ecbee6e0d1703d1225f18e92742981dbf3757
2019-12-04 22:43:22 -08:00
David Tolnay
d1d8fb939a Switch from failure::Fail trait to std::error::Error for errors
Summary:
This diff replaces eden's dependencies on failure::Error with anyhow::Error.

Failure's error type requires all errors to have an implementation of failure's own failure::Fail trait in order for cause chains and backtraces to work. The necessary methods for this functionality have made their way into the standard library error trait, so modern error libraries build directly on std::error::Error rather than something like failure::Fail. Once we are no longer tied to failure 0.1's Fail trait, different parts of the codebase will be free to use any std::error::Error-based libraries they like while still working nicely together.

Reviewed By: xavierd

Differential Revision: D18576093

fbshipit-source-id: e2d862b659450f2969520d9b74877913fabb2e5d
2019-11-22 08:53:31 -08:00
David Tolnay
b1793a4416 rust: Rename Fallible<T> to Result<T>
Summary:
This diff is preparation for migrating off of failure::Fail / failure::Error for errors in favor of errors that implement std::error::Error. The Fallible terminology is unique to failure and in non-failure code we should be using Result<T>. To minimize the size of the eventual diff that removes failure, this codemod replaces all use of Fallible with Result by:

- In modules that do not use Result<T, E>, we import `failure::Fallible as Result`;
- In modules that use a mix of Result<T, E> and Fallible<T> (only 5) we define `type Result<T, E = failure::Error> = std::result::Result<T, E>` to allow both Result<T> and Result<T, E> to work simultaneously.

Reviewed By: Imxset21

Differential Revision: D18499758

fbshipit-source-id: 9f5a54c47f81fdeedbc6003cef42a1194eee55bf
2019-11-14 14:11:01 -08:00
Adam Simpkins
ab3a7cb21f Move fb-mercurial sources into an eden/scm subdirectory.
Summary:
In preparation for merging fb-mercurial sources to the Eden repository,
move everything from the top-level directory into an `eden/scm`
subdirectory.
2019-11-13 16:04:48 -08:00