Summary:
The server is expected to provide head (of all segs), parents (of each seg),
roots (of all segs). We checked roots and parents but only check head in debug
build. Let's check head in release build too.
Reviewed By: andll
Differential Revision: D29405816
fbshipit-source-id: 1a97eb52a9a0d1d444ae5dabd1a01f0786be9fa9
Summary: This was used to narrow down issues.
Reviewed By: andll
Differential Revision: D29404054
fbshipit-source-id: 3bfdac332d63bdb13f40d5cf23dacec242b46d52
Summary: The returned value now includes roots. Rename the function to clarify.
Reviewed By: kulshrax
Differential Revision: D29383072
fbshipit-source-id: 02a255ce20d9797f482f6fe1c716f2d79a12d4e0
Summary: There is a regression in 1.7.0 (which we're on at the moment) so we might as well update.
Reviewed By: zertosh, farnz
Differential Revision: D29358047
fbshipit-source-id: 226393d79c165455d27f7a09b14b40c6a30d96d3
Summary:
An alternative to D29363808 (e396cab669). The benefit is that parents_and_head is used by
both the client and the server. So we don't need to duplicate D29363808 (e396cab669) in
Mononoke code.
Reviewed By: andll
Differential Revision: D29365079
fbshipit-source-id: bca60ba2b3df477929d8e72b2363e5a0f744b35d
Summary:
Pull fast path uses `reload` which drops pending changes.
To avoid misuse, raise an error if pending changes are present.
Reviewed By: andll
Differential Revision: D29363799
fbshipit-source-id: 8f520d2c5553432abc452bc7b2b59d7af80e0a99
Summary:
The import pull data logic used low-level locking, persisting APIs, it does not
write cached idmap to disk. So we need to manually insert the idmap remote
lookup result to the actual local idmap explicitly.
This addressed an issue that verify_missing fails in the pull fast path.
Reviewed By: andll
Differential Revision: D29363813
fbshipit-source-id: 2749855a6c8c924bd1b772173de066d400f73764
Summary:
For a NameDag, `IdConvert` on `self.map` cannot resolve names remotely, but
`IdConvert` on `self` can. Use the latter. This is similar to D27547584 (af3c3b3fd0) where
some `self.map` are updated to `self`.
This addressed an issue found in the pull fast path test. Note there is another
issues to solve.
Reviewed By: andll
Differential Revision: D29363810
fbshipit-source-id: 28ba583ed14bbc5d52af81d4128d965f24eef011
Summary: The test pulls when the client has a lazy graph, and the server has a few merges.
Reviewed By: andll
Differential Revision: D29363806
fbshipit-source-id: 09bc3c4c3d21924f500ca86e8d86f58a15159169
Summary:
`fmt::Debug` for a NameDag is too verbose. Separate part of it so we can debug
print segments for a given (group, level). This will be used by upcoming
changes.
Reviewed By: andll
Differential Revision: D29363805
fbshipit-source-id: e1c6713be10b8b64fc7a42178117e724e0d691d0
Summary:
The client TestDag might have outdated server Dag as the remote protocol,
because it is a static "snapshot". Ensure the remote Dag is updated when
using the pull API.
This is an issue solved by tracking down issues in tests added in upcoming
diffs.
Reviewed By: andll
Differential Revision: D29363807
fbshipit-source-id: a560b2e91999873338604907a6d83cc7d2ff5c58
Summary: It will be used by the next change.
Reviewed By: andll
Differential Revision: D29363802
fbshipit-source-id: 842735ac05ea5fea4ea0c3625a68d06d27bc37d5
Summary:
It is useful when drawdag itself triggers remote fetches.
This was used but is not used after some refactoring. I think it might be useful
in the future so kept it.
Reviewed By: andll
Differential Revision: D29363803
fbshipit-source-id: fa178ac9783d1dc1b73525eeb8cd3d766cf46a0f
Summary: The test will be used to verify upcoming changes.
Reviewed By: andll
Differential Revision: D29363809
fbshipit-source-id: d34d13123914cfabb5c82dee3873b6e0c4979ee2
Summary: Make it easier to write more tests around pull.
Reviewed By: andll
Differential Revision: D29363804
fbshipit-source-id: 5b2cf8675343898fabc1d8845228e240e463edf8
Summary:
The roots data will be useful for the client to check if the pulled commits are
going to overlap with its existed DAG.
Reviewed By: andll
Differential Revision: D29363808
fbshipit-source-id: e09d924d65537f59fd4ea209b568265d07a80e46
Summary: Minor change to make the code a little bit more straightforward.
Reviewed By: andll
Differential Revision: D29363801
fbshipit-source-id: 2c4bd6ece07282f044622227a3c077cb31db6d17
Summary: Make the docstring a bit more consistent.
Reviewed By: andll
Differential Revision: D29363798
fbshipit-source-id: 1b4e2a7a1af4c4cffe3693e437a831bab1b43fd7
Summary: Update versions for several of the crates we depend on.
Reviewed By: danobi
Differential Revision: D29165283
fbshipit-source-id: baaa9fa106b7dad000f93d2eefa95867ac46e5a1
Summary: They will be reused in import_pull_data
Reviewed By: quark-zju
Differential Revision: D29147950
fbshipit-source-id: 192bf33c30067f43c4fcaaf3054741b39efb4e25
Summary: This is an interface for importing pull data into dag
Reviewed By: quark-zju
Differential Revision: D29142979
fbshipit-source-id: b40b94403a044c0b74d1574528aa374ec309a0cf
Summary: This will be used in eager repo integration tests
Reviewed By: quark-zju
Differential Revision: D29113218
fbshipit-source-id: a24232bd6c19010d8ac90d1305f57f1094b06323
Summary: This method will be used to get all Ids that needs to be included into CloneData::id_map during fast forward pull.
Reviewed By: quark-zju
Differential Revision: D29045538
fbshipit-source-id: f6eb04f537aa5365f7588391ea4c60b3dc010a53
Summary: idset_to_flat_segments allows to return intersection between flat segments and span set
Reviewed By: quark-zju
Differential Revision: D28980521
fbshipit-source-id: da90c66a021a1e8bb95edbbc2e06cd2a850d7dcd
Summary:
This diff introduces Subspan trait for objects that contains spans and allows to take a 'subspan' of an object
Implementing Subspan trait allows to intersect arbitrary objects and spans
Reviewed By: quark-zju
Differential Revision: D28980523
fbshipit-source-id: 29a2e58ec3c79a838db751d8a37227b4cdaaa372
Summary:
The server1 was not used after D27629318 (ba7e1c6952) while the test intentionally wants to
exercise graph isomorphism. So let's revive server1 in the test.
Reviewed By: andll
Differential Revision: D28269926
fbshipit-source-id: 0a04031415f559f8a6eb81f1e2f2530329a2a3bc
Summary:
This makes it easier to filter logs related to remote fetching.
The `DEBUG dag::protocol: resolve ids [0] remotely` means the lazy hash resolution is working.
Reviewed By: kulshrax
Differential Revision: D27971117
fbshipit-source-id: f2492204c70d793997d0c3865e500bbad56b1953
Summary: This makes CloneData possible to represent an empty repo.
Reviewed By: sfilipco
Differential Revision: D27926246
fbshipit-source-id: 0bcead224ef5b89c66d07a34d8217edaef62177f
Summary: It's been showing up while building mononoke. Let's fix it
Reviewed By: sfilipco
Differential Revision: D27789928
fbshipit-source-id: a15912f66b9ad3370545aed88405dbeb800e63de
Summary:
The `add_heads_and_flush` method might add new nodes in the master group,
and it should update `overlay_map_next_id` accordingly. Without it, it
might error out like:
RustError: ProgrammingError: Server returned x~n (x = 9ebc9ebc49f1819767b40f9ceb22c37547a10c37 8459584, n = 1411).
But x exceeds the head in the local master group {}. This is not expected and indicates some logic error on the server side.
Full error: P387088806
Reviewed By: sfilipco
Differential Revision: D27637278
fbshipit-source-id: b45370db0561dec52cd513a12e2fd0110f18e0e5
Summary:
While it is in theory "correct" going through the remote resolution even if the
protocol is "local". The overhead turns out to be something. And the tracing
message "resolve .. remotely" can be quite noisy. Let's just skip remote
resolutions early in IdConvert implementations.
Reviewed By: sfilipco
Differential Revision: D27630094
fbshipit-source-id: 7d87079876f040cf8f764f7362021dddba0d4723
Summary:
Currently the "contains vertex" check can trigger excessive
fetches for add_heads (and add_heads_and_flush used by flush).
Add a test to demonstrate the problem.
Reviewed By: sfilipco
Differential Revision: D27630091
fbshipit-source-id: ce3639c2a13226ba5681b4e8696edd7acbcb57f9
Summary:
Otherwise it can cause a lazy dag to think vertexes as "missing", insert
vertexes unnecessarily, and potentially break key graph properties (a
vertex should only have one Id).
Reviewed By: sfilipco
Differential Revision: D27629315
fbshipit-source-id: 1688d13cb94015bbc675613ecf9225556ff48372
Summary:
Also move related functions.
Similar to D27547584 (af3c3b3fd0), this allows `add_heads_and_flush` use `IdConvert`
on the `NameDag`, instead of the `IdMap` to trigger remote fetches properly.
This diff is easier to view with whitespace changes ignored.
Reviewed By: sfilipco
Differential Revision: D27629314
fbshipit-source-id: 8f79223c5d324aabfc5ab9813a9f65400fc533ec
Summary:
See the previous diff for context. Drop Locked and related APIs (prepare_filesystem_sync).
This makes it easier to use operate on a mut NameDag on flush because it does not need
to use separate types (Locked) for writing which has issues of not having the remote protocol
involved.
Reviewed By: sfilipco
Differential Revision: D27629306
fbshipit-source-id: 301445b242321ad5f424741ea91ebf6c075bff1c
Summary:
See the previous diff for context. Drop SyncableIdMap so we are one step
closer to using mut NameDag directly on add_heads, which knows when and how to
do remote fetching properly.
Reviewed By: sfilipco
Differential Revision: D27629310
fbshipit-source-id: 883606e40bb83907dfa6142ddd2c3030de61698f
Summary:
By using SyncableIdDag and SyncableIdMap, it's harder to use extra features
around them (ex. remote fetching). Drop SyncableIdDag so we are one step
closer to using mut NameDag directly on add_heads, which knows when and how to
do remote fetching properly.
Reviewed By: sfilipco
Differential Revision: D27629307
fbshipit-source-id: 8e9a5a4348a42b9751752b82feb3f3d2d7c4ba45
Summary:
The `Parents` trait is used for input of adding (non-lazy) vertexes to the
graph. The API will be used to provide extra hints to optimize network
fetches.
With the current logic, `assign_head` will ask the server to resolve the heads
first, to check if it is already assigned, then to resolve the parents, etc. to
the roots (in the "to assign" set). Ideally the `assign_head` logic can ask
the server to resolve the roots first, and if that's unassigned, then just mark
all descendants of the roots as unassigned, do not send more requests.
Note: the current pull logic has all the hashes ready (hashes are known).
But whether the hashes have Id assigned are unknown. It is more tricky
taking the "lock" and "reload" into consideration - hashes without Ids might
"become" having ids assigned after we obtain the lock to write data to disk.
Practically, `pull` using the current logic would take a very long time because
it tries to resolve things remotely for every "to assign" commits.
Reviewed By: sfilipco
Differential Revision: D27629317
fbshipit-source-id: e02f54f43ef65228ce6e3a8a8723dd9ae0a07008
Summary: This just simplifies the test code a bit.
Reviewed By: sfilipco
Differential Revision: D27629308
fbshipit-source-id: 04eac5cd045c71123e7fc410af74609dbadb8fb7
Summary: This avoids triggering remote lookups if an unknown name was looked up multiple times.
Reviewed By: sfilipco
Differential Revision: D27629316
fbshipit-source-id: 64c1ce5e872a5ce4f14c650a946ae8396f4cc74c
Summary:
When translating RequestNameToLocation to ResponseIdNamePair. If "heads" are
known, but some "names" aren't. Do not treat it as an error. This will be
used by the client-side to properly handle the "contains" check.
Reviewed By: sfilipco
Differential Revision: D27629309
fbshipit-source-id: 206ec5df956b33f4e816ab8d6a67ce776fd7bd74
